Python Web Scraping with BeautifulSoup
BeautifulSoup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree.
Installation
To use BeautifulSoup, you need to install it along with a parser (like lxml or html5lib) and a library to make HTTP requests (like requests).
pip install beautifulsoup4 requests lxmlBasic Usage
First, you need to fetch the HTML content of the page you want to scrape.
import requests
from bs4 import BeautifulSoup
url = 'https://example.com'
response = requests.get(url)
# Create a BeautifulSoup object
soup = BeautifulSoup(response.text, 'lxml')
# Print the formatted HTML
print(soup.prettify())Navigating the Tree
You can navigate the parse tree using tag names.
print(soup.title)
# <title>Example Domain</title>
print(soup.title.string)
# Example Domain
print(soup.h1)
# <h1>Example Domain</h1>Searching the Tree
find() and find_all()
find(): Returns the first occurrence of a tag.find_all(): Returns a list of all occurrences.
# Find the first 'p' tag
p_tag = soup.find('p')
print(p_tag.get_text())
# Find all 'a' tags
links = soup.find_all('a')
for link in links:
print(link.get('href'))Searching by Attributes
You can search for tags with specific attributes (id, class, etc.).
# Find element by ID
element = soup.find(id="main-content")
# Find elements by Class (use class_ because class is a reserved keyword)
items = soup.find_all(class_="item")CSS Selectors
You can use the .select() method to find tags using CSS selectors.
# Select all 'a' tags inside a 'div' with class 'content'
links = soup.select('div.content a')