Python Web Crawler with Scrapy
This guide demonstrates how to build a web crawler using Scrapy. While Scrapy is typically used to generate full projects with multiple files, this guide shows how to run a spider from a single script for simplicity. It crawls a quote website, extracts data, and follows pagination links automatically.
Modules Used:
scrapy: A fast high-level web crawling and web scraping framework.
Installation
pip install scrapyThe Code
Save this as crawler.py.
import scrapy
from scrapy.crawler import CrawlerProcess
class QuoteSpider(scrapy.Spider):
"""
A simple Spider that navigates to a URL, extracts data,
and follows pagination links.
"""
name = "quotes"
# The starting URL(s)
start_urls = [
'http://quotes.toscrape.com/page/1/',
]
def parse(self, response):
"""
This method handles the response downloaded for each of the
requests made.
"""
# 1. Extract Data using CSS Selectors
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('small.author::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
# 2. Follow Pagination
# Find the link to the next page
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
# Schedule the next request
# response.follow automatically handles relative URLs
yield response.follow(next_page, callback=self.parse)
if __name__ == "__main__":
# 3. Configure and Run the Crawler
process = CrawlerProcess(settings={
# Save output to a JSON file
'FEEDS': {
'quotes_data.json': {'format': 'json', 'overwrite': True},
},
# Reduce console noise
'LOG_LEVEL': 'INFO',
# Identify yourself (good practice)
'USER_AGENT': 'Mozilla/5.0 (compatible; MyScraper/1.0)',
})
process.crawl(QuoteSpider)
process.start() # the script will block here until the crawling is finishedUsage
python crawler.py