selectolax.parser module

HtmlParser

class selectolax.parser.HTMLParser(html, detect_encoding=True, use_meta_tags=True, decode_errors=u'ignore')

The HTML parser.

Use this class to parse raw HTML.

Parameters:
html : str (unicode) or bytes
detect_encoding : bool, default True

If True and html type is bytes then encoding will be detected automatically.

use_meta_tags : bool, default True

Whether to use meta tags in encoding detection process.

decode_errors : str, default ‘ignore’

Same as in builtin’s str.decode, i.e ‘strict’, ‘ignore’ or ‘replace’.

any_css_matches(self, tuple selectors)

Returns True if any of the specified CSS selectors matches a node.

body

Returns document body.

clone(self)

Clone the current tree.

css(self, unicode query)

A CSS selector.

Matches pattern query against HTML tree. CSS selectors reference.

Parameters:
query : str

CSS selector (e.g. “div > :nth-child(2n+1):not(:has(a))”).

Returns:
selector : list of Node objects
css_first(self, unicode query, default=None, strict=False)

Same as css but returns only the first match.

Parameters:
query : str
default : bool, default None

Default value to return if there is no match.

strict: bool, default True

Set to True if you want to check if there is strictly only one match in the document.

Returns:
selector : Node object
css_matches(self, unicode selector)
decode_errors

decode_errors: unicode

detect_encoding

detect_encoding: ‘bool’

head

Returns head node.

html

Return HTML representation of the page.

input_encoding

Return encoding of the HTML document.

Returns unknown in case the encoding is not determined.

merge_text_nodes(self)

Iterates over all text nodes and merges all text nodes that are close to each other.

This is useful for text extraction. Use it when you need to strip HTML tags and merge “dangling” text.

Examples

>>> tree = HTMLParser("<div><p><strong>J</strong>ohn</p><p>Doe</p></div>")
>>> node = tree.css_first('div')
>>> tree.unwrap_tags(["strong"])
>>> tree.text(deep=True, separator=" ", strip=True)
"J ohn Doe" # Text extraction produces an extra space because the strong tag was removed.
>>> node.merge_text_nodes()
>>> tree.text(deep=True, separator=" ", strip=True)
"John Doe"
raw_html

raw_html: bytes

root

Returns root node.

script_srcs_contain(self, tuple queries)

Returns True if any of the script SRCs attributes contain on of the specified text.

Caches values on the first call to improve performance.

Parameters:
queries : tuple of str
scripts_contain(self, unicode query)

Returns True if any of the script tags contain specified text.

Caches script tags on the first call to improve performance.

Parameters:
query : str

The query to check.

select(self, query=None)

Select nodes give a CSS selector.

Works similarly to the css method, but supports chained filtering and extra features.

Parameters:
query : str or None

The CSS selector to use when searching for nodes.

Returns:
selector : The Selector class.
strip_tags(self, list tags, bool recursive=False)

Remove specified tags from the node.

Parameters:
tags : list of str

List of tags to remove.

recursive : bool, default True

Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser('<html><head></head><body><script></script><div>Hello world!</div></body></html>')
>>> tags = ['head', 'style', 'script', 'xmp', 'iframe', 'noembed', 'noframes']
>>> tree.strip_tags(tags)
>>> tree.html
'<html><body><div>Hello world!</div></body></html>'
tags(self, unicode name)

Returns a list of tags that match specified name.

Parameters:
name : str (e.g. div)
text(self, bool deep=True, unicode separator=u'', bool strip=False)

Returns the text of the node including text of all its child nodes.

Parameters:
strip : bool, default False

If true, calls str.strip() on each text part to remove extra white spaces.

separator : str, default ‘’

The separator to use when joining text from different nodes.

deep : bool, default True

If True, includes text from all child nodes.

Returns:
text : str
unwrap_tags(self, list tags)

Unwraps specified tags from the HTML tree.

Works the same as th unwrap method, but applied to a list of tags.

Parameters:
tags : list

List of tags to remove.

Examples

>>> tree = HTMLParser("<div><a href="">Hello</a> <i>world</i>!</div>")
>>> tree.head.unwrap_tags(['i','a'])
>>> tree.head.html
'<body><div>Hello world!</div></body>'
use_meta_tags

use_meta_tags: ‘bool’

Node

class selectolax.parser.Node

A class that represents HTML node (element).

any_css_matches(self, tuple selectors)

Returns True if any of CSS selectors matches a node

attributes

Get all attributes that belong to the current node.

The value of empty attributes is None.

Returns:
attributes : dictionary of all attributes.

Examples

>>> tree = HTMLParser("<div data id='my_id'></div>")
>>> node = tree.css_first('div')
>>> node.attributes
{'data': None, 'id': 'my_id'}
attrs

A dict-like object that is similar to the attributes property, but operates directly on the Node data.

Warning

Use attributes instead, if you don’t want to modify Node attributes.

Returns:
attributes : Attributes mapping object.

Examples

>>> tree = HTMLParser("<div id='a'></div>")
>>> node = tree.css_first('div')
>>> node.attrs
<div attributes, 1 items>
>>> node.attrs['id']
'a'
>>> node.attrs['foo'] = 'bar'
>>> del node.attrs['id']
>>> node.attributes
{'foo': 'bar'}
>>> node.attrs['id'] = 'new_id'
>>> node.html
'<div foo="bar" id="new_id"></div>'
child

Return the child node.

css(self, unicode query)

Evaluate CSS selector against current node and its child nodes.

css_first(self, unicode query, default=None, bool strict=False)

Evaluate CSS selector against current node and its child nodes.

css_matches(self, unicode selector)

Returns True if CSS selector matches a node.

decompose(self, bool recursive=True)

Remove a Node from the tree.

Parameters:
recursive : bool, default True

Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser(html)
>>> for tag in tree.css('script'):
>>>     tag.decompose()
html

Return HTML representation of the current node including all its child nodes.

Returns:
text : str
id

Get the id attribute of the node.

Returns None if id does not set.

Returns:
text : str
insert_after(signatures, args, kwargs, defaults)

Insert a node after the current Node.

Parameters:
value : str, bytes or Node

The text or Node instance to insert after the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.insert_after(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get <img src="" alt="Laptop">Laptop</div>'
>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.insert_after(html_parser2.body.child)
<div>Get <span alt="Laptop"><img src="/jpg"><div>Test</div> <div></div></span></div>'
insert_before(signatures, args, kwargs, defaults)

Insert a node before the current Node.

Parameters:
value : str, bytes or Node

The text or Node instance to insert before the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.insert_before(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get Laptop<img src="" alt="Laptop"></div>'
>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.insert_before(html_parser2.body.child)
<div>Get <span alt="Laptop"><div>Test</div><img src="/jpg"> <div></div></span></div>'
iter(self, include_text=False)

Iterate over nodes on the current level.

Parameters:
include_text : bool

If True, includes text nodes as well.

Yields:
node
last_child

Return last child node.

merge_text_nodes(self)

Iterates over all text nodes and merges all text nodes that are close to each other.

This is useful for text extraction. Use it when you need to strip HTML tags and merge “dangling” text.

Examples

>>> tree = HTMLParser("<div><p><strong>J</strong>ohn</p><p>Doe</p></div>")
>>> node = tree.css_first('div')
>>> tree.unwrap_tags(["strong"])
>>> tree.text(deep=True, separator=" ", strip=True)
"J ohn Doe" # Text extraction produces an extra space because the strong tag was removed.
>>> node.merge_text_nodes()
>>> tree.text(deep=True, separator=" ", strip=True)
"John Doe"
next

Return next node.

parent

Return the parent node.

parser

parser: selectolax.parser.HTMLParser

prev

Return previous node.

raw_value

Return the raw (unparsed, original) value of a node.

Currently, works on text nodes only.

Returns:
raw_value : bytes

Examples

>>> html_parser = HTMLParser('<div>&#x3C;test&#x3E;</div>')
>>> selector = html_parser.css_first('div')
>>> selector.child.html
'&lt;test&gt;'
>>> selector.child.raw_value
b'&#x3C;test&#x3E;'
remove(self, bool recursive=True)

An alias for the decompose method.

replace_with

Replace current Node with specified value.

Parameters:
value : str, bytes or Node

The text or Node instance to replace the Node with. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.replace_with(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get Laptop</div>'
>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.replace_with(html_parser2.body.child)
'<div>Get <span alt="Laptop"><div>Test</div> <div></div></span></div>'
script_srcs_contain(self, tuple queries)

Returns True if any of the script SRCs attributes contain on of the specified text.

Caches values on the first call to improve performance.

Parameters:
queries : tuple of str
scripts_contain(self, unicode query)

Returns True if any of the script tags contain specified text.

Caches script tags on the first call to improve performance.

Parameters:
query : str

The query to check.

select(self, query=None)

Select nodes given a CSS selector.

Works similarly to the css method, but supports chained filtering and extra features.

Parameters:
query : str or None

The CSS selector to use when searching for nodes.

Returns:
selector : The Selector class.
strip_tags(self, list tags, bool recursive=False)

Remove specified tags from the HTML tree.

Parameters:
tags : list

List of tags to remove.

recursive : bool, default True

Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser('<html><head></head><body><script></script><div>Hello world!</div></body></html>')
>>> tags = ['head', 'style', 'script', 'xmp', 'iframe', 'noembed', 'noframes']
>>> tree.strip_tags(tags)
>>> tree.html
'<html><body><div>Hello world!</div></body></html>'
tag

Return the name of the current tag (e.g. div, p, img).

Returns:
text : str
text(self, bool deep=True, unicode separator=u'', bool strip=False)

Returns the text of the node including text of all its child nodes.

Parameters:
strip : bool, default False

If true, calls str.strip() on each text part to remove extra white spaces.

separator : str, default ‘’

The separator to use when joining text from different nodes.

deep : bool, default True

If True, includes text from all child nodes.

Returns:
text : str
text_content

Returns the text of the node if it is a text node.

Returns None for other nodes. Unlike the text method, does not include child nodes.

Returns:
text : str or None.
traverse(self, include_text=False)

Iterate over all child and next nodes starting from the current level.

Parameters:
include_text : bool

If True, includes text nodes as well.

Yields:
node
unwrap(self)

Replace node with whatever is inside this node.

Examples

>>>  tree = HTMLParser("<div>Hello <i>world</i>!</div>")
>>>  tree.css_first('i').unwrap()
>>>  tree.html
'<html><head></head><body><div>Hello world!</div></body></html>'
unwrap_tags(self, list tags)

Unwraps specified tags from the HTML tree.

Works the same as the unwrap method, but applied to a list of tags.

Parameters:
tags : list

List of tags to remove.

Examples

>>> tree = HTMLParser("<div><a href="">Hello</a> <i>world</i>!</div>")
>>> tree.body.unwrap_tags(['i','a'])
>>> tree.body.html
'<body><div>Hello world!</div></body>'

Selector

class selectolax.parser.Selector(Node node, query)

An advanced CSS selector that supports additional operations.

Think of it as a toolkit that mimics some of the features of XPath.

Please note, this is an experimental feature that can change in the future.

any_attribute_longer_than(self, unicode attribute, int length, unicode start=None)

Returns True any href attribute longer than a specified length.

Similar to string-length in XPath.

any_matches

Returns True if there are any matches

any_text_contains(self, unicode text, bool deep=True, unicode separator=u'', bool strip=False)

Returns True if any node in the current search scope contains specified text

attribute_longer_than(self, unicode attribute, int length, unicode start=None)

Filter all current matches by attribute length.

Similar to string-length in XPath.

css(self, unicode query)

Evaluate CSS selector against current scope.

matches

Returns all possible matches

text_contains(self, unicode text, bool deep=True, unicode separator=u'', bool strip=False)

Filter all current matches given text.