selectolax.parser module¶

HtmlParser¶

class selectolax.parser.HTMLParser(html, detect_encoding=True, use_meta_tags=True, decode_errors='ignore')¶

The HTML parser using modest backend.

This backend is deprecated. Please use lexbor backend instead.

Use this class to parse raw HTML.

Parameters:

htmlstr (unicode) or bytes
detect_encodingbool, default True: If True and html type is bytes then encoding will be detected automatically.
use_meta_tagsbool, default True: Whether to use meta tags in encoding detection process.
decode_errorsstr, default ‘ignore’: Same as in builtin’s str.decode, i.e ‘strict’, ‘ignore’ or ‘replace’.

any_css_matches(self, tuple selectors)¶: Returns True if any of the specified CSS selectors matches a node.

body¶: Returns document body.

clone(self)¶: Clone the current tree.

css(self, str query)¶

A CSS selector.

Matches pattern query against HTML tree. CSS selectors reference.

Parameters:

querystr: CSS selector (e.g. “div > :nth-child(2n+1):not(:has(a))”).

Returns:

selectorlist of Node objects

css_first(self, str query, default=None, strict=False)¶

Same as css but returns only the first match.

Parameters:

querystr
defaultAny, default None: Default value to return if there is no match.
strict: bool, default False: Set to True if you want to check if there is strictly only one match in the document.

Returns:

selectorNode object

css_matches(self, str selector)¶

decode_errors¶: decode_errors: str

detect_encoding¶: detect_encoding: ‘bool’

head¶: Returns head node.

html¶: Return HTML representation of the page.

input_encoding¶

Return encoding of the HTML document.

Returns unknown in case the encoding is not determined.

merge_text_nodes(self)¶

Iterates over all text nodes and merges all text nodes that are close to each other.

This is useful for text extraction. Use it when you need to strip HTML tags and merge “dangling” text.

Examples

>>> tree = HTMLParser("<div><p><strong>J</strong>ohn</p><p>Doe</p></div>")
>>> node = tree.css_first('div')
>>> tree.unwrap_tags(["strong"])
>>> tree.text(deep=True, separator=" ", strip=True)
"J ohn Doe" # Text extraction produces an extra space because the strong tag was removed.
>>> node.merge_text_nodes()
>>> tree.text(deep=True, separator=" ", strip=True)
"John Doe"

raw_html¶: raw_html: bytes

root¶: Returns root node.

script_srcs_contain(self, tuple queries)¶

Returns True if any of the script SRCs attributes contain on of the specified text.

Caches values on the first call to improve performance.

Parameters:

queriestuple of str

scripts_contain(self, str query)¶

Returns True if any of the script tags contain specified text.

Caches script tags on the first call to improve performance.

Parameters:

querystr: The query to check.

select(self, query=None)¶

Select nodes given a CSS selector.

Works similarly to the css method, but supports chained filtering and extra features.

Parameters:

querystr or None: The CSS selector to use when searching for nodes.

Returns:

selectorThe Selector class.

strip_tags(self, list tags, bool recursive=False)¶

Remove specified tags from the node.

Parameters:

tagslist of str: List of tags to remove.
recursivebool, default True: Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser('<html><head></head><body><script></script><div>Hello world!</div></body></html>')
>>> tags = ['head', 'style', 'script', 'xmp', 'iframe', 'noembed', 'noframes']
>>> tree.strip_tags(tags)
>>> tree.html
'<html><body><div>Hello world!</div></body></html>'

tags(self, str name)¶

Returns a list of tags that match specified name.

Parameters:

namestr (e.g. div)

text(self, bool deep=True, str separator='', bool strip=False)¶

Returns the text of the node including text of all its child nodes.

Parameters:

stripbool, default False: If true, calls str.strip() on each text part to remove extra white spaces.
separatorstr, default ‘’: The separator to use when joining text from different nodes.
deepbool, default True: If True, includes text from all child nodes.

Returns:

textstr

unwrap_tags(self, list tags, delete_empty: bool = False)¶

Unwraps specified tags from the HTML tree.

Works the same as th unwrap method, but applied to a list of tags.

Parameters:

tagslist: List of tags to remove.
delete_emptybool, default False: If True, removes empty tags.

Examples

>>> tree = HTMLParser("<div><a href="">Hello</a> <i>world</i>!</div>")
>>> tree.head.unwrap_tags(['i','a'])
>>> tree.head.html
'<body><div>Hello world!</div></body>'

use_meta_tags¶: use_meta_tags: ‘bool’

Node¶

class selectolax.parser.Node¶

A class that represents HTML node (element).

any_css_matches(self, tuple selectors)¶: Returns True if any of CSS selectors matches a node

attributes¶

Get all attributes that belong to the current node.

The value of empty attributes is None.

Returns:

attributesdictionary of all attributes.

Examples

>>> tree = HTMLParser("<div data id='my_id'></div>")
>>> node = tree.css_first('div')
>>> node.attributes
{'data': None, 'id': 'my_id'}

attrs¶

A dict-like object that is similar to the attributes property, but operates directly on the Node data.

Warning

Use attributes instead, if you don’t want to modify Node attributes.

Returns:

attributesAttributes mapping object.

Examples

>>> tree = HTMLParser("<div id='a'></div>")
>>> node = tree.css_first('div')
>>> node.attrs
<div attributes, 1 items>
>>> node.attrs['id']
'a'
>>> node.attrs['foo'] = 'bar'
>>> del node.attrs['id']
>>> node.attributes
{'foo': 'bar'}
>>> node.attrs['id'] = 'new_id'
>>> node.html
'<div foo="bar" id="new_id"></div>'

child¶

Alias for the first_child property.

Deprecated. Please use first_child instead.

css(self, str query)¶: Evaluate CSS selector against current node and its child nodes.

css_first(self, str query, default=None, bool strict=False)¶: Evaluate CSS selector against current node and its child nodes.

css_matches(self, str selector)¶: Returns True if CSS selector matches a node.

decompose(self, bool recursive=True)¶

Remove a Node from the tree.

Parameters:

recursivebool, default True: Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser(html)
>>> for tag in tree.css('script'):
>>>     tag.decompose()

html¶

Return HTML representation of the current node including all its child nodes.

Returns:

textstr

id¶

Get the id attribute of the node.

Returns None if id does not set.

Returns:

textstr

insert_after(signatures, args, kwargs, defaults, _fused_sigindex={})¶

Insert a node after the current Node.

Parameters:

valuestr, bytes or Node: The text or Node instance to insert after the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.insert_after(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get <img src="" alt="Laptop">Laptop</div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.insert_after(html_parser2.body.child)
<div>Get <span alt="Laptop"><img src="/jpg"><div>Test</div> <div></div></span></div>'

insert_before(signatures, args, kwargs, defaults, _fused_sigindex={})¶

Insert a node before the current Node.

Parameters:

valuestr, bytes or Node: The text or Node instance to insert before the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.insert_before(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get Laptop<img src="" alt="Laptop"></div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.insert_before(html_parser2.body.child)
<div>Get <span alt="Laptop"><div>Test</div><img src="/jpg"> <div></div></span></div>'

insert_child(signatures, args, kwargs, defaults, _fused_sigindex={})¶

Insert a node inside (at the end of) the current Node.

Parameters:

valuestr, bytes or Node: The text or Node instance to insert inside the Node. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src=""></div>')
>>> div = tree.css_first('div')
>>> div.insert_child('Laptop')
>>> tree.body.child.html
'<div>Get <img src="">Laptop</div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"> <div>Laptop</div> </span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> span_node = html_parser.css_first('span')
>>> span_node.insert_child(html_parser2.body.child)
<div>Get <span alt="Laptop"> <div>Laptop</div> <div>Test</div> </span></div>'

iter(self, include_text=False)¶

Iterate over nodes on the current level.

Parameters:

include_textbool: If True, includes text nodes as well.

Yields:

node

last_child¶: Return last child node.

mem_id¶

Get the mem_id attribute of the node.

Returns:

textint

merge_text_nodes(self)¶

Iterates over all text nodes and merges all text nodes that are close to each other.

This is useful for text extraction. Use it when you need to strip HTML tags and merge “dangling” text.

Examples

>>> tree = HTMLParser("<div><p><strong>J</strong>ohn</p><p>Doe</p></div>")
>>> node = tree.css_first('div')
>>> tree.unwrap_tags(["strong"])
>>> tree.text(deep=True, separator=" ", strip=True)
"J ohn Doe" # Text extraction produces an extra space because the strong tag was removed.
>>> node.merge_text_nodes()
>>> tree.text(deep=True, separator=" ", strip=True)
"John Doe"

next¶: Return next node.

parent¶: Return the parent node.

parser¶: parser: selectolax.parser.HTMLParser

prev¶: Return previous node.

raw_value¶

Return the raw (unparsed, original) value of a node.

Currently, works on text nodes only.

Returns:

raw_valuebytes

Examples

>>> html_parser = HTMLParser('<div>&#x3C;test&#x3E;</div>')
>>> selector = html_parser.css_first('div')
>>> selector.child.html
'&lt;test&gt;'
>>> selector.child.raw_value
b'&#x3C;test&#x3E;'

remove(self, bool recursive=True)¶: An alias for the decompose method.

replace_with(signatures, args, kwargs, defaults, _fused_sigindex={})¶

Replace current Node with specified value.

Parameters:

valuestr, bytes or Node: The text or Node instance to replace the Node with. When a text string is passed, it’s treated as text. All HTML tags will be escaped. Convert and pass the Node object when you want to work with HTML. Does not clone the Node object. All future changes to the passed Node object will also be taken into account.

Examples

>>> tree = HTMLParser('<div>Get <img src="" alt="Laptop"></div>')
>>> img = tree.css_first('img')
>>> img.replace_with(img.attributes.get('alt', ''))
>>> tree.body.child.html
'<div>Get Laptop</div>'

>>> html_parser = HTMLParser('<div>Get <span alt="Laptop"><img src="/jpg"> <div></div></span></div>')
>>> html_parser2 = HTMLParser('<div>Test</div>')
>>> img_node = html_parser.css_first('img')
>>> img_node.replace_with(html_parser2.body.child)
'<div>Get <span alt="Laptop"><div>Test</div> <div></div></span></div>'

script_srcs_contain(self, tuple queries)¶

Returns True if any of the script SRCs attributes contain on of the specified text.

Caches values on the first call to improve performance.

Parameters:

queriestuple of str

scripts_contain(self, str query)¶

Returns True if any of the script tags contain specified text.

Caches script tags on the first call to improve performance.

Parameters:

querystr: The query to check.

select(self, query=None)¶

Select nodes given a CSS selector.

Works similarly to the css method, but supports chained filtering and extra features.

Parameters:

querystr or None: The CSS selector to use when searching for nodes.

Returns:

selectorThe Selector class.

strip_tags(self, list tags, bool recursive=False)¶

Remove specified tags from the HTML tree.

Parameters:

tagslist: List of tags to remove.
recursivebool, default True: Whenever to delete all its child nodes

Examples

>>> tree = HTMLParser('<html><head></head><body><script></script><div>Hello world!</div></body></html>')
>>> tags = ['head', 'style', 'script', 'xmp', 'iframe', 'noembed', 'noframes']
>>> tree.strip_tags(tags)
>>> tree.html
'<html><body><div>Hello world!</div></body></html>'

tag¶

Return the name of the current tag (e.g. div, p, img).

Returns:

textstr

text(self, bool deep=True, str separator='', bool strip=False)¶

Returns the text of the node including text of all its child nodes.

Parameters:

stripbool, default False: If true, calls str.strip() on each text part to remove extra white spaces.
separatorstr, default ‘’: The separator to use when joining text from different nodes.
deepbool, default True: If True, includes text from all child nodes.

Returns:

textstr

text_content¶

Returns the text of the node if it is a text node.

Returns None for other nodes. Unlike the text method, does not include child nodes.

Returns:

textstr or None.

traverse(self, include_text=False)¶

Iterate over all child and next nodes starting from the current level.

Parameters:

include_textbool: If True, includes text nodes as well.

Yields:

node

unwrap(self, delete_empty=False)¶

Replace node with whatever is inside this node.

Parameters:

delete_emptybool, default False: Whenever to delete empty tags.

Examples

>>>  tree = HTMLParser("<div>Hello <i>world</i>!</div>")
>>>  tree.css_first('i').unwrap()
>>>  tree.html
'<html><head></head><body><div>Hello world!</div></body></html>'

Note: by default, empty tags are ignored, set “delete_empty” to “True” to change this.

unwrap_tags(self, list tags, delete_empty=False)¶

Unwraps specified tags from the HTML tree.

Works the same as th unwrap method, but applied to a list of tags.

Parameters:

tagslist: List of tags to remove.
delete_emptybool, default False: Whenever to delete empty tags.

Examples

>>> tree = HTMLParser("<div><a href="">Hello</a> <i>world</i>!</div>")
>>> tree.body.unwrap_tags(['i','a'])
>>> tree.body.html
'<body><div>Hello world!</div></body>'

Note: by default, empty tags are ignored, set “delete_empty” to “True” to change this.

Selector¶

class selectolax.parser.Selector(Node node, str query)¶

An advanced CSS selector that supports additional operations.

Think of it as a toolkit that mimics some of the features of XPath.

Please note, this is an experimental feature that can change in the future.

any_attribute_longer_than(self, str attribute, int length, str start=None)¶

Returns True any href attribute longer than a specified length.

Similar to string-length in XPath.

any_matches¶: Returns True if there are any matches

any_text_contains(self, str text, bool deep=True, str separator='', bool strip=False)¶: Returns True if any node in the current search scope contains specified text

attribute_longer_than(self, str attribute, int length, str start=None)¶

Filter all current matches by attribute length.

Similar to string-length in XPath.

css(self, str query)¶: Evaluate CSS selector against current scope.

matches¶: Returns all possible matches

text_contains(self, str text, bool deep=True, str separator='', bool strip=False)¶: Filter all current matches given text.