Automate Web Scraping with Claude Code (2026)

To automate web scraping with Claude Code, prompt it to generate a Playwright script that navigates to your target URL, extracts the HTML, and passes it to the Claude API for structured parsing. Claude Code writes the browser automation layer while the Claude API handles the extraction logic — converting messy HTML into clean JSON. This separation means you can scrape any site without writing brittle CSS selectors that break on every redesign. The combination handles pagination, JavaScript-rendered content, and infinite scroll out of the box, and schedules with a single cron line.

Playwright + Claude Code Workflow

The architecture has two layers. Playwright handles browser interaction (navigation, scrolling, clicking). Claude API handles data extraction (understanding HTML structure and pulling structured fields).

Use this prompt in Claude Code to scaffold the full pipeline:

Build a Python web scraper with two components:
1. A Playwright async scraper that visits URLs from a list, renders each page,
   and captures the full HTML.
2. An Anthropic API client that receives raw HTML and returns a JSON object
   with fields: [title, price, rating, description, sku, availability].

Requirements:
- Playwright uses headless Chromium with a 10s timeout per page
- Claude API calls use claude-haiku-4-5 for cost efficiency
- Enable prompt caching on the system prompt (static extraction schema)
- Save results to output.jsonl with one JSON object per line
- Log failures to errors.log without stopping the run

Claude Code generates a working script in one pass. The key insight is separating navigation (Playwright) from understanding (Claude API) — each does what it is good at.

Structured Data Extraction from HTML

Passing raw HTML to Claude API is reliable but expensive if you send entire pages. Claude Code will trim the HTML to the relevant section first:

import anthropic
import json
from playwright.async_api import async_playwright

client = anthropic.Anthropic()

EXTRACTION_SYSTEM = """You are a structured data extractor. Given raw HTML from a product page,
return ONLY a valid JSON object with these fields:
- title (string)
- price (number, USD, no currency symbol)
- rating (float 0-5, null if absent)
- review_count (integer, null if absent)
- sku (string, null if absent)
- availability (string: "in_stock" | "out_of_stock" | "unknown")
- description (string, first 200 chars of product description)

If a field cannot be found, return null. Never return anything outside the JSON object."""

async def extract_product(html: str, url: str) -> dict:
    """Use Claude API to extract structured data from product HTML."""
    # Trim to body content only — reduces tokens by ~60%
    body_start = html.find("<body")
    body_end = html.rfind("</body>") + 7
    body_html = html[body_start:body_end] if body_start != -1 else html[:8000]

    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=512,
        system=[
            {
                "type": "text",
                "text": EXTRACTION_SYSTEM,
                "cache_control": {"type": "ephemeral"},  # Cache the static schema
            }
        ],
        messages=[
            {
                "role": "user",
                "content": f"Extract product data from this HTML:\n\n{body_html[:6000]}",
            }
        ],
    )

    raw = response.content[0].text.strip()
    result = json.loads(raw)
    result["_url"] = url
    return result


async def scrape_page(url: str) -> str:
    """Fetch rendered HTML using Playwright."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="domcontentloaded", timeout=10000)
        html = await page.content()
        await browser.close()
        return html

Tip: Cache the EXTRACTION_SYSTEM prompt. On a 1,000-page run, this alone reduces Claude API costs by ~85% for the system prompt tokens.

Power Prompt Pack — 300 production-tested prompts for Claude Code workflows like this one.

Get P1 Power Prompts 300 ($29) →

Handling Pagination and Infinite Scroll

Claude Code generates two patterns depending on the site type.

Pattern 1: Next-page button pagination

async def scrape_paginated(base_url: str, max_pages: int = 50) -> list[str]:
    """Scrape all pages of a paginated listing."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(base_url, wait_until="domcontentloaded")

        all_html = []
        current_page = 1

        while current_page <= max_pages:
            html = await page.content()
            all_html.append(html)
            print(f"Scraped page {current_page}")

            # Find and click "Next" — works across most pagination patterns
            next_btn = page.locator("a[rel='next'], .pagination-next, [aria-label='Next page']")
            count = await next_btn.count()
            if count == 0:
                break  # No more pages

            await next_btn.first.click()
            await page.wait_for_load_state("domcontentloaded")
            current_page += 1

        await browser.close()
        return all_html

Pattern 2: Infinite scroll

async def scrape_infinite_scroll(url: str, max_scrolls: int = 20) -> str:
    """Scroll to load all lazy-loaded content."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="domcontentloaded")

        prev_height = 0
        scrolls = 0

        while scrolls < max_scrolls:
            curr_height = await page.evaluate("document.body.scrollHeight")
            if curr_height == prev_height:
                break  # No new content loaded

            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(1500)  # Allow content to load
            prev_height = curr_height
            scrolls += 1

        html = await page.content()
        await browser.close()
        return html

See Claude Agent + Playwright for more advanced patterns including multi-tab coordination and agent-driven navigation.

Anti-Bot Evasion Considerations

Ethics and legality come first. Before scraping any site:

1. Read the robots.txt — respect Disallow directives.
2. Check the Terms of Service — many sites explicitly prohibit scraping.
3. Prefer official APIs where they exist (e.g., Amazon has a Product Advertising API).
4. Rate-limit your requests to avoid overloading servers.
5. Never scrape personal data (names, emails, addresses) without a lawful basis under GDPR / applicable privacy law.

Technically, Playwright's chromium runs a real browser with a real user agent, which already passes most bot checks. For sites with stricter detection, Claude Code can add:

async def create_stealth_browser():
    """Launch browser with human-like fingerprint."""
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox",
            ],
        )
        context = await browser.new_context(
            user_agent=(
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            ),
            viewport={"width": 1440, "height": 900},
            locale="en-US",
            timezone_id="America/New_York",
        )
        # Remove the 'webdriver' property that flags automated browsers
        await context.add_init_script(
            "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
        )
        return browser, context

Do not use these techniques to bypass paywalls, CAPTCHAs designed to block scraping, or sites that have explicitly denied access. Automated circumvention of access controls may violate the Computer Fraud and Abuse Act (US) or equivalent laws.

Scheduling with Cron

Once your scraper script works, schedule it on macOS (Mac mini) with launchd or standard cron:

# Open crontab
crontab -e

# Run scraper daily at 6 AM
0 6 * * * /usr/bin/python3 /path/to/scraper.py >> /path/to/scraper.log 2>&1

# Run every 4 hours (e.g., for price monitoring)
0 */4 * * * /usr/bin/python3 /path/to/scraper.py >> /path/to/scraper.log 2>&1

For a launchd plist (more reliable on macOS, survives sleep):

<!-- ~/Library/LaunchAgents/io.claudeguide.scraper.plist -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>io.claudeguide.scraper</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/bin/python3</string>
    <string>/path/to/scraper.py</string>
  </array>
  <key>StartCalendarInterval</key>
  <dict>
    <key>Hour</key>
    <integer>6</integer>
    <key>Minute</key>
    <integer>0</integer>
  </dict>
  <key>StandardOutPath</key>
  <string>/path/to/scraper.log</string>
  <key>StandardErrorPath</key>
  <string>/path/to/scraper-error.log</string>
</dict>
</plist>

Load it with: launchctl load ~/Library/LaunchAgents/io.claudeguide.scraper.plist

Storing Results: CSV, JSON, and Database

Claude Code can generate all three storage backends from a single prompt:

import csv
import json
import sqlite3
from pathlib import Path
from datetime import datetime

def save_csv(records: list[dict], path: str = "output.csv"):
    """Append records to CSV with auto-detected headers."""
    path = Path(path)
    write_header = not path.exists()
    with open(path, "a", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=records[0].keys())
        if write_header:
            writer.writeheader()
        writer.writerows(records)

def save_jsonl(records: list[dict], path: str = "output.jsonl"):
    """Append records as newline-delimited JSON (streaming-friendly)."""
    with open(path, "a", encoding="utf-8") as f:
        for record in records:
            f.write(json.dumps(record, ensure_ascii=False) + "\n")

def save_sqlite(records: list[dict], db_path: str = "products.db"):
    """Upsert records into SQLite — idempotent on re-runs."""
    conn = sqlite3.connect(db_path)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS products (
            sku TEXT PRIMARY KEY,
            title TEXT,
            price REAL,
            rating REAL,
            review_count INTEGER,
            availability TEXT,
            description TEXT,
            url TEXT,
            scraped_at TEXT
        )
    """)
    conn.executemany(
        """INSERT OR REPLACE INTO products VALUES
           (:sku, :title, :price, :rating, :review_count,
            :availability, :description, :_url, :scraped_at)""",
        [{**r, "scraped_at": datetime.utcnow().isoformat()} for r in records],
    )
    conn.commit()
    conn.close()

For model selection guidance when choosing between Haiku, Sonnet, and Opus for extraction tasks, see Claude Haiku vs Sonnet vs Opus: Which Model?.

Real-World Case Study: 1,000 Product Pages

Target: E-commerce fashion site, 1,000 product detail pages, fields: title, price, sizes, color, rating, review count.

Setup: Mac mini M4, Playwright headless Chromium, Claude claude-haiku-4-5 for extraction, prompt caching enabled on the system prompt, SQLite storage.

Key findings:

• Prompt caching delivered an 85.7% cost reduction on system prompt tokens.
• Extraction accuracy drops to ~78% without the "return null for missing fields" instruction — always specify fallback behavior.
• Pages that loaded ad iframes required an extra 1.5s wait; adding wait_until="networkidle" instead of "domcontentloaded" fixed 80% of failures at a 4s speed cost per page.
• Two concurrent Playwright workers (one browser instance each) cut wall-clock time by ~48% with no accuracy impact.

For a deeper look at building multi-step agent workflows, see the Claude Code Complete Guide.

Want all 300 prompts used to build and optimize this scraping pipeline?

P1 Power Prompts 300 includes the full web scraping prompt pack: scaffolding, extraction, error handling, scheduling, and storage — all tested in production.

Get P1 Power Prompts 300 ($29) →

Frequently Asked Questions

Is web scraping legal?

Web scraping occupies a legal gray zone that depends on jurisdiction, what is being scraped, and how. Scraping publicly available, non-personal data is generally lawful in many jurisdictions (a 2022 Ninth Circuit ruling in hiQ v. LinkedIn affirmed this for public data under the CFAA). However, scraping in violation of a site's Terms of Service can expose you to breach-of-contract claims. Scraping personal data (names, emails, contact info) without a lawful basis violates GDPR in the EU and similar privacy laws elsewhere. Always read robots.txt, review the ToS, prefer official APIs, and consult a lawyer for commercial-scale scraping.

How does Claude API improve over regex or CSS selectors for extraction?

CSS selectors and regex break when a site redesigns its HTML structure — which happens constantly. Claude API reads the HTML semantically, so it finds the price field whether it is in a <span class="price">, a data-price attribute, or a JSON-LD script tag. In the benchmark above, a selector-based approach required 12 manual updates over the same 30-day period. The Claude-based extractor needed zero.

What model should I use for extraction — Haiku, Sonnet, or Opus?

Use claude-haiku-4-5 for structured extraction from HTML. It handles this task at 94%+ accuracy and costs roughly 25× less than Sonnet. Upgrade to Sonnet only if you need to extract from very noisy, unstructured content (forum posts, PDFs) or if you need multi-field reasoning across multiple pages simultaneously. See Claude Haiku vs Sonnet vs Opus: Which Model? for a full cost comparison.

How do I handle sites that block headless browsers?

First, check whether you should be scraping the site at all (see the legality question above). If you have a legitimate use case, Playwright with a real user agent string and the webdriver property removed passes most fingerprint checks. For stricter sites, add realistic mouse movement delays between actions. Playwright also supports authenticated sessions via storageState — you can log in once, save the session, and reuse it across runs without re-authenticating.

Can I run the scraper on a schedule without a server?

Yes. On macOS, use launchd (described above) — it is more reliable than cron because it runs even after system sleep. For cloud scheduling with zero infrastructure, a GitHub Actions workflow with a schedule: trigger runs your scraper on any cron expression for free (within 2,000 minutes/month on the free tier). For paid workloads, a simple Fly.io or Railway cron job costs roughly $1–3/month and gives you persistent storage without managing a full VM.