Web Scraping

ScrapeGraphAI

An open-source Python library that uses LLMs and graph logic to build intelligent web scraping pipelines — scrape any website with just a natural language prompt.

🔍 What is ScrapeGraphAI?

ScrapeGraphAI is an open-source Python library that revolutionizes web scraping by combining Large Language Models with graph-based pipelines. Instead of writing complex CSS selectors, XPath queries, or regex patterns, you simply describe what data you want in natural language, and the LLM extracts it automatically.

🗣️

Natural Language

Describe what you need

🔗

Graph Pipelines

Modular node-based flows

🤖

Multi-LLM

OpenAI, Gemini, Ollama

📦

Open Source

MIT License

💡 Key Insight: Traditional scraping breaks when websites change layout. ScrapeGraphAI uses LLMs to understand the page semantically, making it resilient to layout changes — the LLM adapts automatically.

⚡ Traditional Scraping vs ScrapeGraphAI

❌ Traditional (BeautifulSoup/Scrapy)

from bs4 import BeautifulSoup
import requests

resp = requests.get(url)
soup = BeautifulSoup(resp.text, 'html.parser')

# Fragile selectors that break easily
titles = soup.select('div.product-card h2.title')
prices = soup.select('span.price-current')
ratings = soup.select('div.star-rating')

• Breaks when HTML structure changes
• Requires deep knowledge of CSS/XPath
• Manual maintenance per site

✅ ScrapeGraphAI

from scrapegraphai.graphs import SmartScraperGraph

graph = SmartScraperGraph(
    prompt="Extract all product names, "
           "prices, and ratings",
    source=url,
    config={"llm": {"model": "gpt-4o-mini"}}
)
result = graph.run()

• Adapts to layout changes automatically
• Natural language — no selectors needed
• Works across different sites

📊 Available Graph Pipelines

ScrapeGraphAI provides several pre-built graph pipelines for different scraping scenarios:

Graph Type	Input Source	Description	Use Case
SmartScraperGraph	Single URL	Scrapes one page and extracts data via LLM	Product page, article
SearchGraph	Search query	Searches the web first, then scrapes results	Research, market analysis
SpeechGraph	URL → Audio	Scrapes content and generates audio summary	Podcast content, accessibility
ScriptCreatorGraph	URL	Generates a Python scraping script (not using LLM at runtime)	Production scripts, cost reduction
SmartScraperMultiGraph	Multiple URLs	Scrapes multiple pages in parallel	Bulk extraction, comparison
JSONScraperGraph	JSON / API	Extracts from JSON data using natural language	API responses, structured data
XMLScraperGraph	XML data	Parses XML documents with LLM understanding	RSS feeds, SOAP APIs, configs

💻 Code Examples

1. Basic Smart Scraper

from scrapegraphai.graphs import SmartScraperGraph

# Configuration
config = {
    "llm": {
        "api_key": "your-openai-key",
        "model": "openai/gpt-4o-mini",
    },
    "verbose": True,
    "headless": True,  # Run browser in headless mode
}

# Create and run the scraper
smart_scraper = SmartScraperGraph(
    prompt="Extract the title, author, publication date, "
           "and a brief summary of the main article",
    source="https://example.com/blog/article",
    config=config
)

result = smart_scraper.run()
print(result)
# Output: {"title": "...", "author": "...", "date": "...", "summary": "..."}

2. Using Local LLMs (Ollama)

from scrapegraphai.graphs import SmartScraperGraph

# Use Ollama for local, private scraping (no API costs)
config = {
    "llm": {
        "model": "ollama/llama3",
        "temperature": 0.0,
        "base_url": "http://localhost:11434",
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",
    },
    "verbose": True,
}

scraper = SmartScraperGraph(
    prompt="List all job positions with title, location, "
           "salary range, and required experience",
    source="https://example.com/careers",
    config=config
)

jobs = scraper.run()
for job in jobs.get("positions", []):
    print(f"📋 {job['title']} - {job['location']} - {job['salary']}")

3. Multi-Page Scraping

from scrapegraphai.graphs import SmartScraperMultiGraph

config = {
    "llm": {
        "model": "openai/gpt-4o-mini",
        "api_key": "your-key"
    },
}

# Scrape multiple competitor product pages at once
urls = [
    "https://competitor1.com/product",
    "https://competitor2.com/product",
    "https://competitor3.com/product",
]

multi_scraper = SmartScraperMultiGraph(
    prompt="Extract product name, price, key features, "
           "and customer rating",
    source=urls,
    config=config
)

results = multi_scraper.run()
# Compare prices and features across competitors

🤖 Supported LLM Providers

☁️ Cloud APIs

• OpenAI (GPT-4o, GPT-4o-mini)
• Google Gemini
• Anthropic Claude
• Azure OpenAI
• Groq (ultra-fast)

🖥️ Local (Self-Hosted)

• Ollama (Llama 3, Mistral, etc.)
• HuggingFace Transformers
• LM Studio
• llama.cpp / GGUF models

📊 Embeddings

• OpenAI Embeddings
• Ollama nomic-embed-text
• HuggingFace models
• Google Gecko embeddings

⚙️ How It Works Internally

┌─────────────┐     ┌──────────────┐     ┌─────────────┐     ┌─────────────┐
│  Fetch      │────▶│  Parse       │────▶│  RAG /      │────▶│  Generate   │
│  Node       │     │  Node        │     │  Chunk Node │     │  Node       │
│             │     │              │     │             │     │             │
│ Download    │     │ HTML → text  │     │ Split into  │     │ LLM extracts│
│ the page    │     │ Clean noise  │     │ chunks +    │     │ structured  │
│ (headless)  │     │ Extract body │     │ embed them  │     │ data via    │
│             │     │              │     │             │     │ prompt      │
└─────────────┘     └──────────────┘     └─────────────┘     └─────────────┘

Fetch Node

Downloads the page using Playwright (headless browser) or simple HTTP requests. Handles JavaScript-rendered content, cookies, and authentication.

Parse Node

Converts raw HTML to clean text. Removes navigation, ads, and boilerplate. Extracts the meaningful content from the page body.

RAG / Chunk Node

For large pages, splits content into chunks and uses embeddings to find the most relevant sections for the user's prompt. Reduces token usage.

Generate Node

Sends the relevant content + user prompt to the LLM, which extracts structured data (JSON) according to the natural language description.

🎯 Use Cases

🛒

Price Monitoring

Track competitor prices across e-commerce sites. Automatic adaptation to layout changes.

📰

News Aggregation

Extract headlines, summaries, and key facts from news sites for monitoring dashboards.

🔬

Research & Analysis

Gather data from academic papers, patents, or technical docs for analysis.

💼

Lead Generation

Extract company info, contacts, and job listings from business directories.

⭐

Review Monitoring

Collect and analyze product reviews, ratings, and customer feedback at scale.

📊

Data Enrichment

Enrich CRM data by scraping company websites for details like tech stack, size, and funding.

✅ Best Practices

Do's

Use gpt-4o-mini or local models for cost-effective bulk scraping
Be specific in your prompt — describe the exact fields you want
Use ScriptCreatorGraph for production to avoid per-run LLM costs
Respect robots.txt and rate-limit your requests
Cache results to avoid redundant LLM calls

Don'ts

Use GPT-4 for every single page scrape (use mini models)
Scrape pages too aggressively without delays
Trust LLM output blindly — always validate extracted data
Ignore legal considerations (ToS, GDPR, copyright)
Skip error handling — sites can block, timeout, or change

📚 Resources

GitHub Repository

Source code & documentation

Documentation

Official docs & API reference

Test Your Knowledge

Score 8/10 or higher to pass