{"id":9715,"date":"2023-10-08T18:28:05","date_gmt":"2023-10-08T16:28:05","guid":{"rendered":"https:\/\/via-internet.de\/blog\/?p=9715"},"modified":"2023-10-14T17:36:41","modified_gmt":"2023-10-14T15:36:41","slug":"the-complete-beautifulsoup-cheatsheet-with-examples","status":"publish","type":"post","link":"https:\/\/via-internet.de\/blog\/2023\/10\/08\/the-complete-beautifulsoup-cheatsheet-with-examples\/","title":{"rendered":"BeautifulSoup | Complete Cheatsheet with Examples"},"content":{"rendered":"\n

Installation<\/h2>\n\n\n\n
pip install beautifulsoup4\n<\/pre>\n\n\n\n
from bs4 import BeautifulSoup\n<\/pre>\n\n\n\n
Creating a BeautifulSoup Object<\/h2>\n\n\n\n
Parse HTML string:<\/p>\n\n\n\n
html = \"<p>Example paragraph<\/p>\"\nsoup = BeautifulSoup(html, 'html.parser')<\/pre>\n\n\n\n
Parse from file:<\/p>\n\n\n\n
with open(\"index.html\") as file:\n  soup = BeautifulSoup(file, 'html.parser')<\/pre>\n\n\n\n
BeautifulSoup Object Types<\/h2>\n\n\n\n
When parsing documents and navigating the parse trees, you will encounter the following main object types:<\/p>\n\n\n\n
Tag<\/h3>\n\n\n\n
A Tag corresponds to an HTML or XML tag in the original document:<\/p>\n\n\n\n
soup = BeautifulSoup('<p>Hello World<\/p>')\np_tag = soup.p\n\np_tag.name # 'p'\np_tag.string # 'Hello World'<\/pre>\n\n\n\n
Tags contain nested Tags and NavigableStrings.<\/p>\n\n\n\n
NavigableString<\/h3>\n\n\n\n
A NavigableString represents text content without tags:<\/p>\n\n\n\n
soup = BeautifulSoup('Hello World')\ntext = soup.string\n\ntext # 'Hello World'\ntype(text) # bs4.element.NavigableString<\/pre>\n\n\n\n
BeautifulSoup<\/h3>\n\n\n\n
The BeautifulSoup object represents the parsed document as a whole. It is the root of the tree:<\/p>\n\n\n\n
soup = BeautifulSoup('<html>...<\/html>')\n\nsoup.name # '[document]'\nsoup.head # <head> Tag element\n<\/pre>\n\n\n\n
Comment<\/h3>\n\n\n\n
Comments in HTML are also available as Comment objects:<\/p>\n\n\n\n
<!-- This is a comment -->\n<\/pre>\n\n\n\n
Copy<\/p>\n\n\n\n
comment = soup.find(text=re.compile('This is'))\ntype(comment) # bs4.element.Comment\n<\/pre>\n\n\n\n
Knowing these core object types helps when analyzing, searching, and navigating parsed documents.<\/p>\n\n\n\n
Searching the Parse Tree<\/h2>\n\n\n\n
By Name<\/h3>\n\n\n\n
HTML:<\/p>\n\n\n\n
<div>\n  <p>Paragraph 1<\/p>\n  <p>Paragraph 2<\/p>\n<\/div>\n<\/pre>\n\n\n\n
Python:<\/p>\n\n\n\n
paragraphs = soup.find_all('p')\n# <p>Paragraph 1<\/p>, <p>Paragraph 2<\/p>\n<\/pre>\n\n\n\n
By Attributes<\/h3>\n\n\n\n
HTML:<\/p>\n\n\n\n
<div id=\"content\">\n  <p>Paragraph 1<\/p>\n<\/div>\n<\/pre>\n\n\n\n
Python:Copy<\/p>\n\n\n\n
div = soup.find(id=\"content\")\n# <div id=\"content\">...<\/div>\n<\/pre>\n\n\n\n
By Text<\/h3>\n\n\n\n
HTML:<\/p>\n\n\n\n
<p>This is some text<\/p>\n<\/pre>\n\n\n\n
Python:<\/p>\n\n\n\n
p = soup.find(text=\"This is some text\")\n# <p>This is some text<\/p>\n<\/pre>\n\n\n\n
Searching with CSS Selectors<\/h2>\n\n\n\n
CSS selectors provide a very powerful way to search for elements within a parsed document.<\/p>\n\n\n\n
Some examples of CSS selector syntax:<\/p>\n\n\n\n
By Tag Name<\/h3>\n\n\n\n
Select all<\/p>\n\n\n\n
tags:<\/p>\n\n\n\n
soup.select(\"p\")\n<\/pre>\n\n\n\n
By ID<\/h3>\n\n\n\n
Select element with ID “main”:<\/p>\n\n\n\n
soup.select(\"#main\")\n<\/pre>\n\n\n\n
By Class Name<\/h3>\n\n\n\n
Select elements with class “article”:<\/p>\n\n\n\n
soup.select(\".article\")\n<\/pre>\n\n\n\n
By Attribute<\/h3>\n\n\n\n
Select tags with a “data-category” attribute:<\/p>\n\n\n\n
soup.select(\"[data-category]\")\n<\/pre>\n\n\n\n
Descendant Combinator<\/h3>\n\n\n\n
Select paragraphs inside divs:<\/p>\n\n\n\n
soup.select(\"div p\")\n<\/pre>\n\n\n\n
Child Combinator<\/h3>\n\n\n\n
Select direct children paragraphs:<\/p>\n\n\n\n
soup.select(\"div > p\")\n<\/pre>\n\n\n\n
Adjacent Sibling<\/h3>\n\n\n\n
Select h2 after h1:<\/p>\n\n\n\n
soup.select(\"h1 + h2\")\n<\/pre>\n\n\n\n
General Sibling<\/h3>\n\n\n\n
Select h2 after any h1:<\/p>\n\n\n\n
soup.select(\"h1 ~ h2\")\n<\/pre>\n\n\n\n
By Text<\/h3>\n\n\n\n
Select elements containing text:<\/p>\n\n\n\n
soup.select(\":contains('Some text')\")\n<\/pre>\n\n\n\n
By Attribute Value<\/h3>\n\n\n\n
Select input with type submit:<\/p>\n\n\n\n
soup.select(\"input[type='submit']\")\n<\/pre>\n\n\n\n
Pseudo-classes<\/h3>\n\n\n\n
Select first paragraph:<\/p>\n\n\n\n
soup.select(\"p:first-of-type\")\n<\/pre>\n\n\n\n
Chaining<\/h3>\n\n\n\n
Select first article paragraph:<\/p>\n\n\n\n
soup.select(\"article > p:nth-of-type(1)\")\n<\/pre>\n\n\n\n
Accessing Data<\/h2>\n\n\n\n
HTML:<\/p>\n\n\n\n
<p class=\"content\">Some text<\/p>\n<\/pre>\n\n\n\n
Python:<\/p>\n\n\n\n
p = soup.find('p')\np.name # \"p\"\np.attrs # {\"class\": \"content\"}\np.string # \"Some text\"\n<\/pre>\n\n\n\n
The Power of find_all()<\/h2>\n\n\n\n
The find_all() method is one of the most useful and versatile searching methods in BeautifulSoup.<\/p>\n\n\n\n
Returns All Matches<\/h3>\n\n\n\n
find_all() will find and return a list of all matching elements:<\/p>\n\n\n\n
all_paras = soup.find_all('p')\n<\/pre>\n\n\n\n
This gives you all paragraphs on a page.<\/p>\n\n\n\n
Flexible Queries<\/h3>\n\n\n\n
You can pass a wide range of queries to find_all():Name – find_all(‘p’)Attributes – find_all(‘a’, class_=’external’)Text – find_all(text=re.compile(‘summary’))Limit – find_all(‘p’, limit=2)And more!<\/p>\n\n\n\n
Useful Features<\/h3>\n\n\n\n
Some useful things you can do with find_all():Get a count – len(soup.find_all(‘p’))Iterate through results – for p in soup.find_all(‘p’):Convert to text – [p.get_text() for p in soup.find_all(‘p’)]Extract attributes – [a[‘href’] for a in soup.find_all(‘a’)]<\/p>\n\n\n\n
Why It’s Useful<\/h3>\n\n\n\n
In summary, find_all() is useful because:It returns all matching elementsIt supports diverse and powerful queriesIt enables easily extracting and processing result data<\/p>\n\n\n\n
Whenever you need to get a collection of elements from a parsed document, find_all() will likely be your go-to tool.<\/p>\n\n\n\n
Navigating Trees<\/h2>\n\n\n\n
Traverse up and sideways through related elements.<\/p>\n\n\n\n
Modifying the Parse Tree<\/h2>\n\n\n\n
BeautifulSoup provides several methods for editing and modifying the parsed document tree.<\/p>\n\n\n\n
HTML:<\/p>\n\n\n\n
<p>Original text<\/p>\n<\/pre>\n\n\n\n
Python:<\/p>\n\n\n\n
p = soup.find('p')\np.string = \"New text\"\n<\/pre>\n\n\n\n
Edit Tag Names<\/h3>\n\n\n\n
Change an existing tag name:<\/p>\n\n\n\n
tag = soup.find('span')\ntag.name = 'div'\n<\/pre>\n\n\n\n
Edit Attributes<\/h3>\n\n\n\n
Add, modify or delete attributes of a tag:<\/p>\n\n\n\n
tag['class'] = 'header' # set attribute\ntag['id'] = 'main'\n\ndel tag['class'] # delete attribute\n<\/pre>\n\n\n\n
Edit Text<\/h3>\n\n\n\n
Change text of a tag:<\/p>\n\n\n\n
tag.string = \"New text\"\n<\/pre>\n\n\n\n
Append text to a tag:<\/p>\n\n\n\n
tag.append(\"Additional text\")\n<\/pre>\n\n\n\n
Insert Tags<\/h3>\n\n\n\n
Insert a new tag:<\/p>\n\n\n\n
new_tag = soup.new_tag(\"h1\")\ntag.insert_before(new_tag)\n<\/pre>\n\n\n\n
Delete Tags<\/h3>\n\n\n\n
Remove a tag entirely:<\/p>\n\n\n\n
tag.extract()\n<\/pre>\n\n\n\n
Wrap\/Unwrap Tags<\/h3>\n\n\n\n
Wrap another tag around:<\/p>\n\n\n\n
tag.wrap(soup.new_tag('div))\n<\/pre>\n\n\n\n
Unwrap its contents:<\/p>\n\n\n\n
tag.unwrap()\n<\/pre>\n\n\n\n
Modifying the parse tree is very useful for cleaning up scraped data or extracting the parts you need.<\/p>\n\n\n\n
Outputting HTML<\/h2>\n\n\n\n
Input HTML:<\/p>\n\n\n\n
<p>Hello World<\/p>\n<\/pre>\n\n\n\n
Python:<\/p>\n\n\n\n
print(soup.prettify())\n\n# <p>\n#  Hello World\n# <\/p>\n<\/pre>\n\n\n\n
Integrating with Requests<\/h2>\n\n\n\n
Fetch a page:<\/p>\n\n\n\n
import requests\n\nres = requests.get(\"<https:\/\/example.com>\")\nsoup = BeautifulSoup(res.text, 'html.parser')\n<\/pre>\n\n\n\n
Parsing Only Parts of a Document<\/h2>\n\n\n\n
When dealing with large documents, you may want to parse only a fragment rather than the whole thing. BeautifulSoup allows for this using SoupStrainers.<\/p>\n\n\n\n
There are a few ways to parse only parts of a document:<\/p>\n\n\n\n
By CSS Selector<\/h3>\n\n\n\n
Parse just a selection matching a CSS selector:<\/p>\n\n\n\n
from bs4 import SoupStrainer\n\nonly_tables = SoupStrainer(\"table\")\nsoup = BeautifulSoup(doc, parse_only=only_tables)\n<\/pre>\n\n\n\n
This will parse only the tags from the document.<\/p>\n\n\n\n
By Tag Name<\/h3>\n\n\n\n
Parse only specific tags:<\/p>\n\n\n\n
only_divs = SoupStrainer(\"div\")\nsoup = BeautifulSoup(doc, parse_only=only_divs)\n<\/pre>\n\n\n\n
By Function<\/h3>\n\n\n\n
Pass a function to test if a tag should be parsed:<\/p>\n\n\n\n
def is_short_string(string):\n  return len(string) < 20\n\nonly_short_strings = SoupStrainer(string=is_short_string)\nsoup = BeautifulSoup(doc, parse_only=only_short_strings)\n<\/pre>\n\n\n\n
This parses tags based on their text content.<\/p>\n\n\n\n
By Attributes<\/h3>\n\n\n\n
Parse tags that contain specific attributes:<\/p>\n\n\n\n
has_data_attr = SoupStrainer(attrs={\"data-category\": True})\nsoup = BeautifulSoup(doc, parse_only=has_data_attr)\n<\/pre>\n\n\n\n
Multiple Conditions<\/h3>\n\n\n\n
You can combine multiple strainers:<\/p>\n\n\n\n
strainer = SoupStrainer(\"div\", id=\"main\")\nsoup = BeautifulSoup(doc, parse_only=strainer)\n<\/pre>\n\n\n\n
This will parse only<\/p>\n\n\n\n
.<\/p>\n\n\n\n
Parsing only parts you need can help reduce memory usage and improve performance when scraping large documents.<\/p>\n\n\n\n
Dealing with Encoding<\/h2>\n\n\n\n
When parsing documents, you may encounter encoding issues. Here are some ways to handle encoding:<\/p>\n\n\n\n
Specify at Parse Time<\/h3>\n\n\n\n
Pass the from_encoding parameter when creating the BeautifulSoup object:<\/p>\n\n\n\n
soup = BeautifulSoup(doc, from_encoding='utf-8')\n<\/pre>\n\n\n\n
This handles any decoding needed when initially parsing the document.<\/p>\n\n\n\n
Encode Tag Contents<\/h3>\n\n\n\n
You can encode the contents of a tag:<\/p>\n\n\n\n
tag.string.encode(\"utf-8\")\n<\/pre>\n\n\n\n
Use this when outputting tag strings.<\/p>\n\n\n\n
Encode Entire Document<\/h3>\n\n\n\n
To encode the entire BeautifulSoup document:<\/p>\n\n\n\n
soup.encode(\"utf-8\")\n<\/pre>\n\n\n\n
This returns a byte string with the encoded document.<\/p>\n\n\n\n
Pretty Print with Encoding<\/h3>\n\n\n\n
Specify encoding when pretty printing<\/p>\n\n\n\n
print(soup.prettify(encoder=\"utf-8\"))\n<\/pre>\n\n\n\n
Unicode Dammit<\/h3>\n\n\n\n
BeautifulSoup’s UnicodeDammit <\/em>class can detect and convert incoming documents to Unicode:<\/p>\n\n\n\n
from bs4 import UnicodeDammit\n\ndammit = UnicodeDammit(doc)\nsoup = dammit.unicode_markup\n<\/pre>\n\n\n\n
This converts even poorly encoded documents to Unicode.<\/p>\n\n\n\n
Properly handling encoding ensures your scraped data is decoded and output correctly when using BeautifulSoup.<\/p>\n","protected":false},"excerpt":{"rendered":"
Installation Creating a BeautifulSoup Object Parse HTML string: Parse from file: BeautifulSoup Object Types When parsing documents and navigating the parse trees, you will encounter the following main object types: Tag A Tag corresponds to an HTML or XML tag in the original document: Tags contain nested Tags and NavigableStrings. NavigableString A NavigableString represents text content without tags: BeautifulSoup The BeautifulSoup object represents the parsed document as a whole. It is the root of the tree: Comment Comments in HTML are also available as Comment objects: Copy Knowing these core object types helps when analyzing, searching, and navigating parsed documents. Searching […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[158,65],"tags":[],"class_list":["post-9715","post","type-post","status-publish","format-standard","hentry","category-beautifulsoup","category-python"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/9715","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/comments?post=9715"}],"version-history":[{"count":6,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/9715\/revisions"}],"predecessor-version":[{"id":9724,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/posts\/9715\/revisions\/9724"}],"wp:attachment":[{"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/media?parent=9715"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/categories?post=9715"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/via-internet.de\/blog\/wp-json\/wp\/v2\/tags?post=9715"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}