Scrapy Master Guide - Complete Web Scraping Tutorial

Introduction to Scrapy

Scrapy is a fast, high-level web crawling and web scraping framework used to crawl websites and extract structured data from their pages. It's the most powerful and flexible Python framework for web scraping.

⚡ Fast & Powerful

Built on Twisted, an asynchronous networking library, Scrapy can handle thousands of requests simultaneously.

🔧 Extensible

Designed with plugins and middlewares in mind, making it easy to extend functionality.

📊 Built-in Export

Export scraped data in multiple formats: JSON, CSV, XML, and more.

🛡️ Robust

Handles cookies, redirects, retries, and various edge cases automatically.

Installation & Setup

Requirements: Python 3.8+ and pip package manager

Install via pip

pip install scrapy

Install via conda

conda install -c conda-forge scrapy

Verify Installation

scrapy version

Quick Start Guide

1. Create a New Project

scrapy startproject myproject
cd myproject

2. Generate a Spider

scrapy genspider quotes quotes.toscrape.com

3. Create Your First Spider

import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = ['https://quotes.toscrape.com/']
    
    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }
        
        # Follow pagination
        next_page = response.css('li.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

4. Run the Spider

scrapy crawl quotes -o quotes.json

Success! You've just created and run your first Scrapy spider. The scraped data is saved in quotes.json.

Understanding Spiders

Spiders are classes that define how a website should be scraped, including how to crawl through the site and extract data from pages.

Basic Spider

class BasicSpider(scrapy.Spider):
    name = 'basic'
    allowed_domains = ['example.com']
    start_urls = ['https://example.com/']
    
    def parse(self, response):
        # Extract data
        title = response.css('h1::text').get()
        yield {'title': title}

CrawlSpider for Following Links

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MyCrawlSpider(CrawlSpider):
    name = 'crawl_example'
    allowed_domains = ['example.com']
    start_urls = ['https://example.com/']
    
    rules = (
        Rule(LinkExtractor(allow=r'/page/\d+'), callback='parse_item', follow=True),
        Rule(LinkExtractor(allow=r'/category/'), follow=True),
    )
    
    def parse_item(self, response):
        yield {
            'url': response.url,
            'title': response.css('h1::text').get(),
        }

Spider Methods

Method	Description	When Called
`start_requests()`	Generate initial requests	Spider startup
`parse()`	Default callback for requests	For each response
`closed()`	Called when spider closes	Spider shutdown

Selectors & Data Extraction

Scrapy provides powerful selectors for extracting data from HTML/XML using CSS selectors and XPath expressions.

CSS Selectors

# Get first matching element
title = response.css('h1::text').get()
title = response.css('h1::text').get(default='No title')

# Get all matching elements
links = response.css('a::attr(href)').getall()

# Nested selectors
for article in response.css('article'):
    yield {
        'title': article.css('h2::text').get(),
        'author': article.css('.author::text').get(),
    }

XPath Selectors

# Basic XPath
title = response.xpath('//h1/text()').get()

# With conditions
price = response.xpath('//span[@class="price"]/text()').get()

# Text operations
normalized = response.xpath('normalize-space(//p/text())').get()

Tip: Use Scrapy Shell to test your selectors interactively: scrapy shell 'https://example.com'

Items & Item Loaders

Items provide a structured way to define the data you want to scrape, while Item Loaders offer a convenient mechanism for populating items.

Defining Items

# items.py
import scrapy

class ProductItem(scrapy.Item):
    name = scrapy.Field()
    price = scrapy.Field()
    stock = scrapy.Field()
    description = scrapy.Field()

Using Item Loaders

from scrapy.loader import ItemLoader
from itemloaders.processors import TakeFirst, MapCompose

class ProductLoader(ItemLoader):
    default_item_class = ProductItem
    default_output_processor = TakeFirst()
    
    price_in = MapCompose(lambda x: x.replace('$', ''), float)

# In your spider
def parse_product(self, response):
    loader = ProductLoader(response=response)
    loader.add_css('name', 'h1::text')
    loader.add_css('price', '.price::text')
    yield loader.load_item()

Item Pipelines

Pipelines process items after they've been scraped. Use them for validation, cleaning, and storage.

Basic Pipeline

# pipelines.py
class ValidationPipeline:
    def process_item(self, item, spider):
        if not item.get('price'):
            raise DropItem(f"Missing price in {item}")
        
        if item.get('price') < 0:
            item['price'] = 0
        
        return item

Database Pipeline

import sqlite3

class SQLitePipeline:
    def __init__(self):
        self.connection = sqlite3.connect('data.db')
        self.cursor = self.connection.cursor()
        
    def process_item(self, item, spider):
        self.cursor.execute('''
            INSERT INTO products (name, price)
            VALUES (?, ?)
        ''', (item['name'], item['price']))
        self.connection.commit()
        return item
    
    def close_spider(self, spider):
        self.connection.close()

Enable Pipelines in Settings

# settings.py
ITEM_PIPELINES = {
    'myproject.pipelines.ValidationPipeline': 300,
    'myproject.pipelines.SQLitePipeline': 400,
}

Settings & Configuration

Scrapy settings allow you to customize the behavior of all Scrapy components.

Common Settings

# settings.py

# Basic settings
BOT_NAME = 'mybot'
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS = 16
DOWNLOAD_DELAY = 0.5

# User agent
USER_AGENT = 'mybot (+http://www.example.com)'

# AutoThrottle
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

# Cache
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 3600

# Retry settings
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

Important: Always set ROBOTSTXT_OBEY = True in production to respect website policies.

Handling JavaScript

Many modern websites use JavaScript to load content dynamically. Scrapy can handle these sites using various tools.

Using Scrapy-Playwright

# Install
pip install scrapy-playwright

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

# Spider
class PlaywrightSpider(scrapy.Spider):
    def start_requests(self):
        yield scrapy.Request(
            url='https://example.com',
            meta={
                'playwright': True,
                'playwright_page_methods': [
                    {'method': 'wait_for_selector', 'selector': '.dynamic-content'}
                ]
            }
        )

Using Scrapy-Splash

# Install
pip install scrapy-splash

# Spider
from scrapy_splash import SplashRequest

class JsSpider(scrapy.Spider):
    def start_requests(self):
        yield SplashRequest(
            url='https://example.com',
            callback=self.parse,
            args={'wait': 2, 'html': 1}
        )

Common Scraping Patterns

Pagination

def parse(self, response):
    # Extract items
    for item in response.css('.item'):
        yield {...}
    
    # Follow next page
    next_page = response.css('.next::attr(href)').get()
    if next_page:
        yield response.follow(next_page, self.parse)

Login & Forms

def parse(self, response):
    return scrapy.FormRequest.from_response(
        response,
        formdata={'username': 'user', 'password': 'pass'},
        callback=self.after_login
    )

def after_login(self, response):
    if "Welcome" in response.text:
        # Continue scraping
        yield response.follow('/profile', callback=self.parse_profile)

API Crawling

import json

class APISpider(scrapy.Spider):
    def start_requests(self):
        headers = {'Authorization': 'Bearer TOKEN'}
        yield scrapy.Request(
            url='https://api.example.com/items',
            headers=headers,
            callback=self.parse
        )
    
    def parse(self, response):
        data = json.loads(response.text)
        for item in data['results']:
            yield item

Debugging & Testing

Scrapy Shell

# Start interactive shell
scrapy shell 'https://example.com'

# In the shell:
>>> response.css('h1::text').get()
>>> response.xpath('//title/text()').get()
>>> view(response)  # Opens in browser

Debug Settings

# settings.py for debugging
LOG_LEVEL = 'DEBUG'
DUPEFILTER_DEBUG = True
COOKIES_DEBUG = True

Testing Spiders

import unittest
from scrapy.http import TextResponse

class TestSpider(unittest.TestCase):
    def test_parse(self):
        response = TextResponse(
            url='https://example.com',
            body=b'Test'
        )
        spider = MySpider()
        results = list(spider.parse(response))
        self.assertEqual(results[0]['title'], 'Test')

Command Line Reference

Project Commands

Command	Description
`scrapy startproject NAME`	Create new project
`scrapy genspider NAME DOMAIN`	Generate new spider
`scrapy list`	List available spiders
`scrapy crawl SPIDER`	Run a spider
`scrapy shell URL`	Interactive debugging shell
`scrapy fetch URL`	Download page and show content
`scrapy view URL`	Open response in browser
`scrapy bench`	Run benchmark test

Output Options

# Save to different formats
scrapy crawl spider -o output.json
scrapy crawl spider -o output.csv
scrapy crawl spider -o output.xml

# Append to file
scrapy crawl spider -o items.jsonl:jsonlines

# Custom settings
scrapy crawl spider -s USER_AGENT='MyBot 1.0'

Deployment Options

Docker Deployment

# Dockerfile
FROM python:3.9-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

CMD ["scrapy", "crawl", "myspider"]

Scrapyd Deployment

# Install Scrapyd
pip install scrapyd scrapyd-client

# Start server
scrapyd

# Deploy project
scrapyd-deploy myproject

# Schedule spider
curl http://localhost:6800/schedule.json \
  -d project=myproject -d spider=myspider

Cron Job

# Add to crontab
0 */6 * * * cd /path/to/project && scrapy crawl spider

Troubleshooting

403 Forbidden Error

Add proper headers and user agent:

USER_AGENT = 'Mozilla/5.0...'

Timeout Issues

Increase timeout and retry settings:

DOWNLOAD_TIMEOUT = 60 RETRY_TIMES = 5

Memory Issues

Limit memory usage:

MEMUSAGE_LIMIT_MB = 512

Best Practice

Always use AutoThrottle in production to avoid overwhelming servers.

Resources & Community

Popular Extensions

Scrapy-Splash - JavaScript rendering
Scrapy-Playwright - Modern JS rendering
Scrapy-Redis - Distributed crawling
Scrapyd - Deploy and run spiders
Scrapy-Deltafetch - Incremental crawling

🕷️ Scrapy Master Guide