selectolax.parser module¶

HtmlParser¶

class selectolax.parser.HTMLParser(html, detect_encoding=True, use_meta_tags=True, decode_errors=u'ignore')¶

The HTML parser.

Use this class to parse raw HTML.

Parameters:	html : str (unicode) or bytes detect_encoding : bool, default True If True and html type is bytes then encoding will be detected automatically. use_meta_tags : bool, default True Whether to use meta tags in encoding detection process. decode_errors : str, default ‘ignore’ Same as in builtin’s str.decode, i.e ‘strict’, ‘ignore’ or ‘replace’.

any_css_matches(self, tuple selectors)¶: Returns True if any of the specified CSS selectors matches a node.

body¶: Returns document body.

clone(self)¶: Clone the current tree.

css(self, unicode query)¶

A CSS selector.

Matches pattern query against HTML tree. CSS selectors reference.

Parameters:	query : str CSS selector (e.g. “div > :nth-child(2n+1):not(:has(a))”).
Returns:	selector : list of Node objects

css_first(self, unicode query, default=None, strict=False)¶

Same as css but returns only the first match.

Parameters:	query : str default : bool, default None Default value to return if there is no match. strict: bool, default True Set to True if you want to check if there is strictly only one match in the document.
Returns:	selector : Node object

css_matches(self, unicode selector)¶

decode_errors¶: decode_errors: unicode

detect_encoding¶: detect_encoding: ‘bool’

head¶: Returns head node.

html¶: Return HTML representation of the page.

input_encoding¶

Return encoding of the HTML document.

Returns unknown in case the encoding is not determined.

merge_text_nodes(self)¶

Iterates over all text nodes and merges all text nodes that are close to each other.

This is useful for text extraction. Use it when you need to strip HTML tags and merge “dangling” text.

Examples

>>> tree = HTMLParser("<div><p><strong>J</strong>ohn</p><p>Doe</p></div>")
>>> node = tree.css_first('div')
>>> tree.unwrap_tags(["strong"])
>>> tree.text(deep=True, separator=" ", strip=True)
"J ohn Doe" # Text extraction produces an extra space because the strong tag was removed.
>>> node.merge_text_nodes()
>>> tree.text(deep=True, separator=" ", strip=True)
"John Doe"

raw_html¶: raw_html: bytes

root¶: Returns root node.

script_srcs_contain(self, tuple queries)¶

Returns True if any of the script SRCs attributes contain on of the specified text.

Caches values on the first call to improve performance.

Parameters:	queries : tuple of str

scripts_contain(self, unicode query)¶

Returns True if any of the script tags contain specified text.

Caches script tags on the first call to improve performance.

Parameters:	query : str The query to check.

select(self, query=None)¶

Select nodes given a CSS selector.

Works similarly to the css method, but supports chained filtering and extra features.

Parameters:	query : str or None The CSS selector to use when searching for nodes.
Returns:	selector : The Selector class.

strip_tags(self, list tags, bool recursive=False)¶

Remove specified tags from the node.

Parameters:	tags : list of str List of tags to remove. recursive : bool, default True Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser('<html><head></head><body><script></script><div>Hello world!</div></body></html>')
>>> tags = ['head', 'style', 'script', 'xmp', 'iframe', 'noembed', 'noframes']
>>> tree.strip_tags(tags)
>>> tree.html
'<html><body><div>Hello world!</div></body></html>'

tags(self, unicode name)¶

Returns a list of tags that match specified name.

Parameters:	name : str (e.g. div)

text(self, bool deep=True, unicode separator=u'', bool strip=False)¶

Returns the text of the node including text of all its child nodes.

Parameters:	strip : bool, default False If true, calls `str.strip()` on each text part to remove extra white spaces. separator : str, default ‘’ The separator to use when joining text from different nodes. deep : bool, default True If True, includes text from all child nodes.
Returns:	text : str

unwrap_tags(self, list tags)¶

Unwraps specified tags from the HTML tree.

Works the same as th unwrap method, but applied to a list of tags.

Parameters:	tags : list List of tags to remove.

Examples

>>> tree = HTMLParser("<div><a href="">Hello</a> <i>world</i>!</div>")
>>> tree.head.unwrap_tags(['i','a'])
>>> tree.head.html
'<body><div>Hello world!</div></body>'

use_meta_tags¶: use_meta_tags: ‘bool’

Node¶

class selectolax.parser.Node¶

A class that represents HTML node (element).

any_css_matches(self, tuple selectors)¶: Returns True if any of CSS selectors matches a node

attributes¶

Get all attributes that belong to the current node.

The value of empty attributes is None.

Returns:	attributes : dictionary of all attributes.

Examples

>>> tree = HTMLParser("<div data id='my_id'></div>")
>>> node = tree.css_first('div')
>>> node.attributes
{'data': None, 'id': 'my_id'}

attrs¶

A dict-like object that is similar to the attributes property, but operates directly on the Node data.

Warning

Use attributes instead, if you don’t want to modify Node attributes.

Returns:	attributes : Attributes mapping object.

Examples

>>> tree = HTMLParser("<div id='a'></div>")
>>> node = tree.css_first('div')
>>> node.attrs
<div attributes, 1 items>
>>> node.attrs['id']
'a'
>>> node.attrs['foo'] = 'bar'
>>> del node.attrs['id']
>>> node.attributes
{'foo': 'bar'}
>>> node.attrs['id'] = 'new_id'
>>> node.html
'<div foo="bar" id="new_id"></div>'

child¶: Return the child node.

css(self, unicode query)¶: Evaluate CSS selector against current node and its child nodes.

css_first(self, unicode query, default=None, bool strict=False)¶: Evaluate CSS selector against current node and its child nodes.

css_matches(self, unicode selector)¶: Returns True if CSS selector matches a node.

decompose(self, bool recursive=True)¶

Remove a Node from the tree.

Parameters:	recursive : bool, default True Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser(html)
>>> for tag in tree.css('script'):
>>>     tag.decompose()

html¶

Return HTML representation of the current node including all its child nodes.

Returns:	text : str

id¶

Get the id attribute of the node.

Returns None if id does not set.

Returns:	text : str

insert_after(signatures, args, kwargs, defaults)¶

Insert a node after the current Node.

Parameters:	value : str, bytes or Node The text or Node instance to insert after the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the `Node` object when you want to work with HTML. Does not clone the `Node` object. All future changes to the passed `Node` object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.insert_after(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get <img src="" alt="Laptop">Laptop</div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.insert_after(html_parser2.body.child)
<div>Get <span alt="Laptop"><img src="/jpg"><div>Test</div> <div></div></span></div>'

insert_before(signatures, args, kwargs, defaults)¶

Insert a node before the current Node.

Parameters:	value : str, bytes or Node The text or Node instance to insert before the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the `Node` object when you want to work with HTML. Does not clone the `Node` object. All future changes to the passed `Node` object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.insert_before(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get Laptop<img src="" alt="Laptop"></div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.insert_before(html_parser2.body.child)
<div>Get <span alt="Laptop"><div>Test</div><img src="/jpg"> <div></div></span></div>'

iter(self, include_text=False)¶

Iterate over nodes on the current level.

Parameters:	include_text : bool If True, includes text nodes as well.
Yields:	node

last_child¶: Return last child node.

mem_id¶

Get the mem_id attribute of the node.

Returns:	text : int

merge_text_nodes(self)¶

Iterates over all text nodes and merges all text nodes that are close to each other.

This is useful for text extraction. Use it when you need to strip HTML tags and merge “dangling” text.

Examples

>>> tree = HTMLParser("<div><p><strong>J</strong>ohn</p><p>Doe</p></div>")
>>> node = tree.css_first('div')
>>> tree.unwrap_tags(["strong"])
>>> tree.text(deep=True, separator=" ", strip=True)
"J ohn Doe" # Text extraction produces an extra space because the strong tag was removed.
>>> node.merge_text_nodes()
>>> tree.text(deep=True, separator=" ", strip=True)
"John Doe"

next¶: Return next node.

parent¶: Return the parent node.

parser¶: parser: selectolax.parser.HTMLParser

prev¶: Return previous node.

raw_value¶

Return the raw (unparsed, original) value of a node.

Currently, works on text nodes only.

Returns:	raw_value : bytes

Examples

>>> html_parser = HTMLParser('<div>&#x3C;test&#x3E;</div>')
>>> selector = html_parser.css_first('div')
>>> selector.child.html
'&lt;test&gt;'
>>> selector.child.raw_value
b'&#x3C;test&#x3E;'

remove(self, bool recursive=True)¶: An alias for the decompose method.

replace_with¶

Replace current Node with specified value.

Parameters:	value : str, bytes or Node The text or Node instance to replace the Node with. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the `Node` object when you want to work with HTML. Does not clone the `Node` object. All future changes to the passed `Node` object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.replace_with(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get Laptop</div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.replace_with(html_parser2.body.child)
'<div>Get <span alt="Laptop"><div>Test</div> <div></div></span></div>'

script_srcs_contain(self, tuple queries)¶

Returns True if any of the script SRCs attributes contain on of the specified text.

Caches values on the first call to improve performance.

Parameters:	queries : tuple of str

scripts_contain(self, unicode query)¶

Returns True if any of the script tags contain specified text.

Caches script tags on the first call to improve performance.

Parameters:	query : str The query to check.

select(self, query=None)¶

Select nodes given a CSS selector.

Works similarly to the css method, but supports chained filtering and extra features.

Parameters:	query : str or None The CSS selector to use when searching for nodes.
Returns:	selector : The Selector class.

strip_tags(self, list tags, bool recursive=False)¶

Remove specified tags from the HTML tree.

Parameters:	tags : list List of tags to remove. recursive : bool, default True Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser('<html><head></head><body><script></script><div>Hello world!</div></body></html>')
>>> tags = ['head', 'style', 'script', 'xmp', 'iframe', 'noembed', 'noframes']
>>> tree.strip_tags(tags)
>>> tree.html
'<html><body><div>Hello world!</div></body></html>'

tag¶

Return the name of the current tag (e.g. div, p, img).

Returns:	text : str

text(self, bool deep=True, unicode separator=u'', bool strip=False)¶

Returns the text of the node including text of all its child nodes.

Parameters:	strip : bool, default False If true, calls `str.strip()` on each text part to remove extra white spaces. separator : str, default ‘’ The separator to use when joining text from different nodes. deep : bool, default True If True, includes text from all child nodes.
Returns:	text : str

text_content¶

Returns the text of the node if it is a text node.

Returns None for other nodes. Unlike the text method, does not include child nodes.

Returns:	text : str or None.

traverse(self, include_text=False)¶

Iterate over all child and next nodes starting from the current level.

Parameters:	include_text : bool If True, includes text nodes as well.
Yields:	node

unwrap(self)¶

Replace node with whatever is inside this node.

Examples

>>>  tree = HTMLParser("<div>Hello <i>world</i>!</div>")
>>>  tree.css_first('i').unwrap()
>>>  tree.html
'<html><head></head><body><div>Hello world!</div></body></html>'

unwrap_tags(self, list tags)¶

Unwraps specified tags from the HTML tree.

Works the same as the unwrap method, but applied to a list of tags.

Parameters:	tags : list List of tags to remove.

Examples

>>> tree = HTMLParser("<div><a href="">Hello</a> <i>world</i>!</div>")
>>> tree.body.unwrap_tags(['i','a'])
>>> tree.body.html
'<body><div>Hello world!</div></body>'

Selector¶

class selectolax.parser.Selector(Node node, query)¶

An advanced CSS selector that supports additional operations.

Think of it as a toolkit that mimics some of the features of XPath.

Please note, this is an experimental feature that can change in the future.

any_attribute_longer_than(self, unicode attribute, int length, unicode start=None)¶

Returns True any href attribute longer than a specified length.

Similar to string-length in XPath.

any_matches¶: Returns True if there are any matches

any_text_contains(self, unicode text, bool deep=True, unicode separator=u'', bool strip=False)¶: Returns True if any node in the current search scope contains specified text

attribute_longer_than(self, unicode attribute, int length, unicode start=None)¶

Filter all current matches by attribute length.

Similar to string-length in XPath.

css(self, unicode query)¶: Evaluate CSS selector against current scope.

matches¶: Returns all possible matches

text_contains(self, unicode text, bool deep=True, unicode separator=u'', bool strip=False)¶: Filter all current matches given text.