Web Scraping

Firecrawl

Turn any website into LLM-ready markdown. A powerful API that crawls, scrapes, and converts web pages into clean, structured data for RAG and AI applications.

🔥 What is Firecrawl?

Firecrawl is a specialized scraping tool designed specifically for the AI era. Unlike traditional scrapers that return raw HTML full of navigational clutter, Firecrawl converts websites into clean Markdown. It handles complex tasks like crawling subpages, screenshotting, and PDF parsing automatically, making it the perfect data ingestion layer for RAG pipelines.

🧹

Clean MD

LLM-optimized markdown

🕸️

Crawler

Traverse sitemaps & links

🧠

Smart Extraction

Structured JSON extraction

🚀

API First

Simple SDK integration

💡 Why Markdown? Markdown is the native language of LLMs. By converting HTML to Markdown, Firecrawl removes token-wasting boilerplate (headers, footers, scripts) and preserves the semantic structure (headers, lists, tables) that helps models understand context.

✨ Key Features

HTML to Markdown

Automatically strips navigation, ads, and scripts. Converts tables to GFM (GitHub Flavored Markdown).

Crawl Mode

Give it a base URL, and it crawls extraction links, subpages, and sitemaps automatically.

Structured Extraction

Define a Pydantic schema or JSON structure, and Firecrawl uses LLMs to extract exact fields.

Anti-Bot Bypass

Handles dynamic content (JS rendering), rotating proxies, CAPTCHAs, and rate limits.

💻 Implementation Examples

1. Basic Scrape (URL to Markdown)

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a single URL into Markdown
scrape_result = app.scrape_url(
    "https://docs.python.org/3/library/functions.html", 
    params={
        "formats": ["markdown"],
    }
)

print(scrape_result["markdown"])
# Output:
# # Built-in Functions
# The Python interpreter has a number of functions...

2. Crawl and Map a Website

# Crawl a whole documentation site
crawl_status = app.crawl_url(
    "https://docs.firecrawl.dev",
    params={
        "limit": 100,
        "scrapeOptions": {"formats": ["markdown"]}
    }
)

print(f"Crawl ID: {crawl_status.get('id')}")

# Check status later
status = app.check_crawl_status(crawl_status.get('id'))
print(f"Status: {status['status']}, Pages: {status['completed']}")

3. Extraction with Schema

from pydantic import BaseModel, Field

class ArticleExtract(BaseModel):
    title: str = Field(description="The title of the article")
    author: str = Field(description="The author's name")
    key_points: list[str] = Field(description="List of 3 main takeaways")

scrape_result = app.scrape_url(
    "https://techcrunch.com/some-article",
    params={
        "formats": ["extract"],
        "extract": {
            "schema": ArticleExtract.model_json_schema()
        }
    }
)

print(scrape_result["extract"])
# Output: {"title": "...", "author": "...", "key_points": [...]}

🎯 Ideal Use Cases

📚

RAG Knowledge Bases

Ingest documentation, blogs, and wikis directly into vector databases as clean Markdown.

🤖

LLM Agents

Give agents the ability to browse the live web and read pages without hallucinating on HTML noise.

📊

Market Intelligence

Extract structured data from competitor sites, pricing pages, and news feeds.

🆚 Comparison

Feature	BeautifulSoup	Selenium / Puppeteer	Firecrawl
Output	Raw HTML/Text	Raw HTML/Text	Clean Markdown / JSON
JS Rendering	❌ No	✅ Yes (Heavy)	✅ Yes (Managed)
Anti-Bot	❌ DIY	⚠️ Hard to maintain	✅ Built-in
Cleaning	Manual Regex/selectors	Manual Regex/selectors	✅ Automatic

📚 Resources

Official Website

Pricing and API Keys

GitHub Repo

Open source core

Test Your Knowledge

Score 8/10 or higher to pass