Firecrawl

What it is

Firecrawl is an open-source web scraper designed for LLMs. Given a URL, it returns clean Markdown or structured JSON — handling JavaScript rendering, anti-bot measures, and crawling across many pages. It is widely used as a building block for RAG pipelines and research agents, and offers an MCP server that plugs into Claude, Cursor, and others.

Best for

• Turn any website into clean Markdown for RAG
• Crawl and extract data from many pages
• Power research agents that need fresh web data

Quick start

1. Cloud: sign up at firecrawl.dev → grab the API key from the dashboard.
2. Self-host: clone https://github.com/firecrawl/firecrawl, copy .env.example to .env, fill in OPENAI_API_KEY, then `docker compose up -d`.
3. Python SDK: `pip install firecrawl-py`; Node: `npm i @mendable/firecrawl-js`.
4. Single page: `client.scrape("https://example.com", formats=["markdown", "html"])` — get clean Markdown + structured JSON.
5. Crawl: `client.crawl("https://docs.example.com", limit=100)` — walks subpages, returns Markdown; AI extraction uses your schema (Pydantic / Zod).

Sample input / output

```python
from firecrawl import Firecrawl
from pydantic import BaseModel

class Product(BaseModel):
    name: str
    price: float
    in_stock: bool

client = Firecrawl(api_key="fc-...")
result = client.extract(
    urls=["https://shop.example.com/catalog"],
    prompt="Extract all products with name, price, availability",
    schema=Product,
)
print(result.json())

```json
[
  { "name": "Wireless Earbuds Pro", "price": 89.99, "in_stock": true },
  { "name": "USB-C Hub 7-in-1",  "price": 39.50, "in_stock": true },
  { "name": "Mechanical Keyboard", "price": 149.00, "in_stock": false }
]
```
One LLM call to a structured Pydantic schema, returns typed JSON — perfect for RAG pipelines.

Benchmarks

Pricing

Free self-host; cloud from free tier

Underlying models

Self-hosting

Yes — Firecrawl can be self-hosted under the AGPL-3.0 license. This gives you full control over data and deployment.