Web Scraping & Data Extraction Engine

# Web Scraping & Data Extraction Engine

## Quick Health Check (Run First)

Score your scraping operation (2 points each):

| Signal | Healthy | Unhealthy |
|--------|---------|-----------|
| Legal compliance | robots.txt checked, ToS reviewed | Scraping blindly |
| Architecture | Tool matches site complexity | Using Puppeteer for static HTML |
| Anti-detection | Rotation, delays, fingerprint diversity | Single IP, no delays |
| Data quality | Validation + dedup pipeline | Raw dumps, no cleaning |
| Error handling | Retry logic, circuit breakers | Crashes on first 403 |
| Monitoring | Success rates tracked, alerts set | No visibility |
| Storage | Structured, deduplicated, versioned | Flat files, duplicates |
| Scheduling | Appropriate frequency, off-peak | Hammering during business hours |

**Score: /16** → 12+: Production-ready | 8-11: Needs work | <8: Stop and redesign

---

## Phase 1: Legal & Ethical Foundation

### Pre-Scrape Compliance Checklist

```yaml
compliance_brief:
  target_domain: ""
  date_assessed: ""
  
  robots_txt:
    checked: false
    target_paths_allowed: false
    crawl_delay_specified: ""
    ai_bot_rules: ""  # Many sites now block AI crawlers specifically
    
  terms_of_service:
    reviewed: false
    scraping_mentioned: false
    scraping_prohibited: false
    api_available: false
    api_sufficient: false
    
  data_classification:
    type: ""  # public-factual | public-personal | behind-auth | copyrighted
    contains_pii: false
    pii_types: []  # name, email, phone, address, photo
    gdpr_applies: false  # EU residents' data
    ccpa_applies: false  # California residents' data
    
  legal_risk: ""  # low | medium | high | do-not-scrape
  decision: ""  # proceed | use-api | request-permission | abandon
  justification: ""
```

### Legal Landscape Quick Reference

| Scenario | Risk Level | Key Case Law |
|----------|-----------|--------------|
| Public data, no login, robots.txt allows | LOW | hiQ v. LinkedIn (2022) |
| Public...

# Web Scraping & Data Extraction Engine 🕸️

Complete web scraping methodology for AI agents and developers — from legal compliance to production-scale data pipelines.

## What This Skill Does

Turns your AI agent into a web scraping expert that:

- **Assesses legality** before touching any site (robots.txt, ToS, GDPR/CCPA)
- **Selects the right tool** (HTTP clients → Scrapy → Playwright → managed services)
- **Defeats anti-bot detection** with proxy rotation, fingerprint diversity, and stealth patterns
- **Builds data pipelines** with validation, deduplication, and structured storage
- **Monitors health** with breakage detection, success rate tracking, and alerting
- **Scales efficiently** from single-site to millions of pages

## Install

```bash
clawhub install afrexai-web-scraping-engine
```

## Quick Start

Tell your agent:
- "Check if I can scrape example.com/products"
- "Build a price monitoring scraper for 3 competitor sites"
- "My scraper keeps getting blocked — help"
- "Extract product data from this URL"

## What's Inside

- **Legal compliance framework** with case law references and decision rules
- **Tool selection matrix** (8 tools compared across 5 dimensions)
- **Anti-detection strategies** (proxy tiers, stealth configs, Cloudflare bypass)
- **Code patterns** for pagination, JS rendering, authentication, change detection
- **Data pipeline** with validation, deduplication, cleaning, and storage
- **5 complete scraping patterns** (e-commerce, jobs, news, social, real estate)
- **Production operations** — monitoring dashboard, breakage detection, runbook
- **100-point quality scoring** rubric

## ⚡ Level Up

This free skill covers methodology. For industry-specific data extraction strategies:

- **[SaaS Context Pack](https://afrexai-cto.github.io/context-packs/)** — Competitor monitoring, pricing intelligence
- **[Ecommerce Context Pack](https://afrexai-cto.github.io/context-packs/)** — Product data, price tracking at scale
- **[Real Estate Context Pack...