By Rajesh Kannan, Co-Founder & CTO, Gramosoft · Last updated: · Prices and crawl features checked on each vendor's own pages on November 14, 2026

A RAG pipeline is only as good as the pages you put into it, and most of those pages sit three or four clicks deep on someone's docs site, help center or blog. A web crawling API follows those links for you, renders each page and hands back Markdown or JSON you can chunk and embed. This guide compares the 10 crawling APIs worth shortlisting in 2026, starting with GcrawlAI, with prices checked on vendor pages, a cost script for 10,000 to 1,000,000 pages a month and a working crawl-to-JSON script.

TL;DR

  • Best overall for RAG: GcrawlAI. Every crawled page is one request with a real browser, residential proxy and stealth included, failed pages are never charged, and you get a Cloud API, a Python SDK, MCP for agents and an MIT core you can self-host.
  • Cheapest at 100,000 pages a month, on list prices: ScrapeGraphAI ($85), then Firecrawl and Olostep ($99). GcrawlAI is $101. Spider's usage billing is lower still on its own estimate.
  • Schema JSON costs extra almost everywhere: Firecrawl's JSON format adds 4 credits a page, Cloudflare's JSON runs on Workers AI, ScrapeGraphAI's Extract is 5 credits.
  • Need only the URL list first? GcrawlAI, Firecrawl, Tavily, Olostep and Bright Data export a site map from sitemaps and links before you crawl; the map script below writes it to sitemap.xml.
  • Re-crawls are where bills grow. Hash every page and re-embed only what changed; the script below does it.

What is the best recursive crawling API for RAG in 2026? GcrawlAI, for most RAG teams: it crawls a whole site through a real browser and residential proxies, returns Markdown plus page metadata, charges one request per page with failed pages free, and connects through a Python SDK, MCP or a self-hosted MIT core. Firecrawl, ScrapeGraphAI and Olostep cost slightly less on unprotected sites.

Quick comparison (details and sources for each API follow)

API Type Recursive crawl Markdown Schema JSON Billing unit Free plan
GcrawlAI Cloud API, SDK, MCP, MIT core Yes, sitemap + links Yes Via your LLM or extractors Request (1 per page) 50 requests/month, no card
Firecrawl Cloud API, AGPL core Yes Yes +4 credits/page Credit 1,000 credits/month
Crawl4AI Open-source library + Cloud Yes (deep crawl) Yes CSS or LLM strategies Cloud credit (0.1 to 2 per page) Library free; Cloud 1,000 credits/month
Spider Cloud API Yes Yes AI add-ons Bandwidth + compute Keyless playground
Tavily Crawl Cloud API Yes, with instructions Yes No Credit (map + extract) 1,000 credits/month
Apify Website Content Crawler Actor on a platform Yes Yes No Compute units Monthly platform credit
Cloudflare /crawl Browser Rendering endpoint (beta) Yes Yes Via Workers AI Browser hours 10 min/day on Workers Free
Olostep Cloud API Yes Yes Separate endpoint Request (1 per page) 500 requests
ScrapeGraphAI Cloud API Yes Yes Extract, 5 credits Credit 500 credits
Bright Data Crawl API Cloud API (async) Yes Yes ld_json output Record (1 per page) Free trial

The 10 best web crawling APIs for RAG in 2026

These are the crawling APIs that take one start URL, discover the rest of the site and return content an LLM can use, ranked for RAG work: output quality, scope controls, how pages that block bots are handled, and what the bill looks like when you re-crawl every week. GcrawlAI comes first; the others follow by fit.

1. GcrawlAI: one request per crawled page, browser and proxies included

Best for: RAG pipelines and AI agents that crawl docs, help centers and JavaScript or bot-protected sites · License: MIT core, managed Cloud API · Language: Python SDK (gcrawl-sdk 0.5.9), REST, MCP

GcrawlAI is a crawling, scraping and Search API for AI data work. A crawl starts by mapping the site from its sitemap and internal links, then loads every page in a stealth browser through residential IPs and returns clean Markdown, with SEO metadata alongside when you ask for it. Nothing is fetched with plain HTTP, so pages that need JavaScript or check for bots come back filled in. Each page costs one request whatever it needed.

GcrawlAI homepage with the logo and menu (Products, Playground, Extractors, Docs, Pricing, Blog, Use Cases, Custom Request): Transform any website into structured LLM-ready data, the open-source web scraping API for AI agents, RAG pipelines and data teams, with JavaScript rendering, anti-bot stealth and proxies included in every request and no credit system; tabs for Scrape, Map, Crawl, Batch, Screenshot, Search and Smart Answer
The GcrawlAI homepage: Scrape, Map, Crawl, Batch, Screenshot and Search in one place, rendering and proxies in every request (screenshot taken October 26, 2026).
  • Map, crawl and scrape: map and export a site's URLs first, crawl with page limits, same-domain and subdomain controls and canonical-URL duplicate removal, or scrape single pages.
  • Markdown built for retrieval: Fit-Markdown with menus, footers and ads stripped, plus HTML, SEO metadata, screenshots and offline HTML bundles from the same crawl.
  • Access included: real browser, rotating residential proxies with country targeting and stealth escalation in every request, no credit multipliers.
  • Async jobs with webhooks: large crawls and batches run in the background and report back when they finish.
  • Agents and search: an MCP server, the Search API and Deep Search, which merges several engines into one ranked list, on the same key.
  • Try it first: the Playground runs a crawl without code; the docs cover every endpoint.

This script crawls a docs site, splits every page by heading into chunks and writes JSON records with the URL, title, section and a content hash. On the next run it skips pages whose hash hasn't changed, so you only re-embed what moved.

crawl_to_rag.py

# pip install gcrawl-sdk
"""Recursive crawl -> Markdown -> JSON records for RAG, with GcrawlAI.
Each crawled page is one request (real browser, residential proxy and stealth included).
Failed pages are never charged. Pages whose content hasn't changed since the last run are skipped."""
import hashlib
import json
import os
import re
from datetime import datetime, timezone
from pathlib import Path

from gcrawl_sdk import GcrawlClient

client = GcrawlClient(api_key=os.getenv("GCRAWL_API_KEY", "gc-YOUR_API_KEY"))  # base_url="http://localhost:8000" if self-hosted
OUT, SEEN = Path("chunks.jsonl"), Path("seen_hashes.json")
MAX_CHARS = 1200


def chunks(markdown: str):
    """Split on ## / ### headings, then pack paragraphs into chunks of up to MAX_CHARS."""
    section = ""
    for block in re.split(r"\n(?=#{2,3} )", markdown):
        heading = re.match(r"#{2,3} (.+)", block)
        section = heading.group(1).strip() if heading else section
        buf = ""
        for para in block.split("\n\n"):
            if buf and len(buf) + len(para) > MAX_CHARS:
                yield section, buf.strip()
                buf = ""
            buf += para + "\n\n"
        if buf.strip():
            yield section, buf.strip()


def crawl_to_jsonl(start_url: str, limit: int = 200, geo: str = "US"):
    seen = json.loads(SEEN.read_text()) if SEEN.exists() else {}
    job = client.crawl(url=start_url, limit=limit, formats=["markdown", "seo"], geo=geo, wait=True)
    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
    written = skipped = 0
    with OUT.open("a", encoding="utf-8") as f:
        for page in job.pages:
            md = page.markdown or ""
            page_hash = hashlib.sha256(md.encode()).hexdigest()
            if not md or seen.get(page.url) == page_hash:
                skipped += 1                       # empty or unchanged since the last run: nothing to re-embed
                continue
            title = (page.seo_json or {}).get("title", "")
            for i, (section, text) in enumerate(chunks(md)):
                f.write(json.dumps({
                    "id": hashlib.sha1(f"{page.url}#{i}".encode()).hexdigest()[:16],
                    "url": page.url, "title": title, "section": section,
                    "text": text, "page_hash": page_hash, "crawled_at": now,
                }, ensure_ascii=False) + "\n")
                written += 1
            seen[page.url] = page_hash
    SEEN.write_text(json.dumps(seen, indent=1))
    print(f"{job.status}: {len(job.pages)} pages, {job.pages_failed or 0} failed (not billed), "
          f"{written} chunks written, {skipped} skipped (unchanged or empty)")


if __name__ == "__main__":
    crawl_to_jsonl("https://docs.example.com/", limit=200)

The offline check below runs that script, and the map script further down, against the real SDK with the network stubbed: a site map export, two crawls of three pages, one failed page that isn't billed, and a second run that writes records only for the page that changed. It checks the logic, not a live crawl.

demo_output.txt (printed output)

3 URLs exported to sitemap.xml and urls.txt
3 URLs in sitemap.xml
completed: 3 pages, 1 failed (not billed), 6 chunks written, 1 skipped (unchanged or empty)
completed: 3 pages, 1 failed (not billed), 4 chunks written, 2 skipped (unchanged or empty)
HTTP calls: [('POST', '/api/v1/links', 500, True), ('GET', '/crawler/results/m1'), ('POST', '/api/v1/crawl', 50, 'US', True, True), ('GET', '/crawler/results/c1'), ('POST', '/api/v1/crawl', 50, 'US', True, True), ('GET', '/crawler/results/c1')]
10 JSON records; first: {'url': 'https://docs.example.com/', 'title': 'Acme Docs', 'section': '', 'text': '# Docs\n\nWelcome.'}
Building cross-section with an elevator, titled one crawl, floor by floor. Floor 6 start: one URL plus scope, page limit, same domain, subdomains. Floor 5 map: sitemap and internal links, canonical duplicates dropped. Floor 4 render: real browser and residential IP, stealth escalation, 1 request per page. Floor 3 clean: Fit-Markdown and SEO metadata. Floor 2 chunk: split by heading into JSON records with url, title, section and page hash. Floor 1 index: embed changed pages only, vector database or agent via SDK or MCP. Basement: failed, blocked, timeout or 5xx, retried inside the platform, $0
One GcrawlAI crawl, floor by floor: GcrawlAI handles start to clean, your script chunks and indexes.

Pricing: a free plan of 50 requests a month with no card, paid plans from $0.67 to $1.45 per 1,000 requests, and a self-hosted open-source core that costs nothing. One crawled page is one request, and failed pages are never charged. Full plan table in the cost section.

Where it falls short: there's no plain-English extraction parameter, so schema JSON comes from your own LLM over the Markdown, from GcrawlAI's ready-made extractors, or from your own selectors. A re-crawl still fetches every page in scope; the hash check saves embedding work, not crawl requests. The ecosystem is younger, with fewer ready-made framework integrations, and self-hosting means bringing your own proxy accounts.

"Most RAG teams I talk to don't lose quality at the embedding step. They lose it at the crawl, when a third of the docs come back as a cookie banner or a loading spinner. We made every crawled page a real browser visit for exactly that reason."

Rajesh Kannan, Co-Founder & CTO at Gramosoft

2. Firecrawl: the best-known crawl-to-Markdown API

Best for: teams that want a mature crawl API with many SDKs and framework loaders · Type: Cloud API, open-source core (AGPL-3.0)

Firecrawl's crawl endpoint follows links from a start URL and returns each page as Markdown, HTML or JSON, with a separate map endpoint for URL discovery. It has official SDKs in several languages and loaders for LangChain and LlamaIndex, which is why it's the default many RAG tutorials reach for.

Pricing: crawl costs 1 credit a page and the JSON format adds 4 credits a page, per its pricing page and billing docs.

Plan Monthly Billed yearly Credits/month Rollover
Free $0 - 1,000 No
Hobby $19 $16/month 5,000 No
Standard $99 $83/month 100,000 No
Growth $399 $333/month 500,000 No
Scale $749 $599/month 1,000,000 One month

Where it falls short: structured JSON makes every page cost 5 credits, so 100,000 JSON pages need the $399 Growth plan. Credits don't roll over below Scale, and the self-hosted core is AGPL-3.0, which matters if you ship it inside a product.

3. Crawl4AI: open-source deep crawling in Python, now with a Cloud

Best for: Python teams that want to run the crawler themselves · Type: open-source library (version 0.9.5 on PyPI, October 2026) plus a hosted Cloud

Crawl4AI is a Python library that renders pages with Playwright and turns them into Markdown, with a filtered "fit" Markdown that drops boilerplate and deep-crawl strategies for following links across a site. Structured data comes from CSS rules or an LLM you configure. The project now also runs Crawl4AI Cloud, a hosted API with an MCP endpoint.

Pricing: the library is free. The Cloud gives 1,000 free credits a month and bills 0.1 credit for a plain page, 0.5 for a real-browser page and 2 for a hard site; its site crawl endpoint covers 100 pages by default and up to 10,000 URLs a job. The page lists extra credits from $0.67 to $1 per 1,000 but didn't show plan prices when we checked, so it isn't in the cost script.

Where it falls short: self-hosted, you run the browsers, queue, retries and proxies yourself, and anti-bot handling is only as good as the proxies you bring. On the Cloud, a protected page costs 20 times a plain one.

4. Spider: usage-based crawling, cheapest on paper

Best for: very large crawls of lightweight pages where cost per page matters most · Type: managed Rust crawler, pay as you go

Spider's crawl endpoint takes a start URL and a limit, follows links and returns every page in one response, in Markdown and other formats, with HTTP, smart and browser request modes.

Pricing: no per-page price. Its pricing guide bills $1 per GB of bandwidth plus compute at one credit ($0.0001) a minute, credits never expire, and it estimates a 1,000,000-page RAG crawl in Chrome mode with Markdown at about $80 to $160 a month.

Where it falls short: the bill depends on page weight and render time, so you only know it after the crawl. The guide also says a blocked or empty page still pays a small minimum, so check how many of your pages come back blocked before you rely on the estimate.

5. Tavily Crawl: crawling steered by instructions

Best for: agents that already use Tavily search and need part of a site · Type: Cloud API

Tavily's crawl endpoint walks a site in parallel from a root URL, with depth and page limits, path filters and an optional plain-English instruction that steers which pages it keeps. It returns extracted content, Markdown included.

Pricing: per its credit docs, a crawl costs the map plus the extract: 1 credit per 10 pages mapped (2 with instructions) and 1 credit per 5 pages extracted (2 on advanced), with failed extractions free. That's 0.3 credits a page on basic settings. Plans run from 1,000 free credits a month to $500 for 100,000, and pay as you go is $0.008 a credit.

Where it falls short: it gets expensive at volume ($220 for 100,000 pages in our script), there's no schema JSON output, and instructions or advanced extraction double parts of the cost.

6. Apify Website Content Crawler: the crawler with the most knobs

Best for: teams already on Apify that want fine control over how a crawl renders and cleans pages · Type: Actor on the Apify platform

Website Content Crawler is Apify's own Actor for crawling a site into Markdown, text or HTML for LLMs, with a choice of HTTP or headless-browser crawling, URL scoping and cleaning options. Runs land in Apify datasets you can export or pull through the API.

Pricing: the Actor is billed in Apify platform compute units rather than per page, so cost depends on how heavy the site is. Apify's pricing page didn't load for us on November 14, 2026; check apify.com/pricing for current plans.

Where it falls short: you can't price a crawl per page before it runs, and there's a platform to learn: Actors, runs, datasets and proxy groups.

7. Cloudflare /crawl: a crawl endpoint in Browser Rendering

Best for: teams on Cloudflare Workers crawling sites that allow it · Type: Browser Rendering REST endpoint, open beta since March 2026

One POST starts an asynchronous crawl job: set a page limit (up to 100,000), depth, whether URLs come from sitemaps, links or both, and HTML, Markdown or JSON output. You poll the job ID or subscribe to queue events, and results stay available for 14 days.

Pricing: crawls that render run on browser hours: 10 hours a month on Workers Paid, then $0.09 an hour, per Cloudflare's pricing page. Crawls with rendering off are unbilled during the beta. JSON output uses Workers AI, which bills separately. The Workers Free plan allows 10 browser minutes a day, five crawl jobs a day and 100 pages a crawl.

Where it falls short: it respects robots.txt and Content Signals and doesn't get past CAPTCHAs or Turnstile, so protected sites stop it. It's still in beta.

8. Olostep: per-page pricing that falls with volume

Best for: high-volume crawls on a flat per-page budget · Type: Cloud API

Olostep's crawl endpoint discovers subpages under include and exclude rules and returns a list of pages; you then pull each page's Markdown or HTML with its retrieve ID. A 50-page crawl counts as 50 requests, and only successfully processed pages are billed.

Pricing: per its pricing page, 500 free requests, then $9 for 5,000, $99 for 200,000, $399 for 1,000,000 and $1,100 for 3,000,000 a month, with JavaScript rendering and residential IPs from Starter up. Top-up packs last six months.

Where it falls short: crawling is two steps (crawl, then retrieve each page), and the pricing page doesn't say whether structured extraction costs extra.

9. ScrapeGraphAI: the cheapest Markdown crawl at mid volume

Best for: mid-size Markdown crawls plus LLM extraction on a budget · Type: Cloud API

ScrapeGraphAI crawls a site and scrapes each page to Markdown, and its Extract service pulls structured JSON with an LLM from a prompt or schema.

Pricing: per its pricing docs, 500 free credits, then $17 for 10,000, $85 for 100,000 and $425 for 750,000 credits a month. A Markdown page is 1 credit, Extract is 5, and a crawl adds 2 credits a job. Failed requests aren't charged.

Where it falls short: no published overage, so above 750,000 pages a month you're on a custom plan, and the docs don't list a separate price for extraction during a crawl.

10. Bright Data Crawl API: enterprise crawling as a dataset job

Best for: large one-off domain crawls delivered to storage · Type: asynchronous Cloud API

You trigger a crawl job for a root URL and receive the site's pages as Markdown, text, HTML or JSON, including ld_json, delivered as data files. Bright Data's docs now call it the Fetch API (formerly Crawl API).

Pricing: $1.50 per 1,000 records pay as you go, or a $499 monthly Growth plan listed at $1.30 per 1,000 (shown at $0.98 with a 25% discount when we checked), per its pricing page.

Where it falls short: it's a batch job rather than an interactive call, and it costs more per page than the API-first crawlers below 1,000,000 pages.

How a recursive crawl turns a site into RAG-ready Markdown and JSON

Botanical plate titled anatomy of a recursive crawl. From docs.example.com at depth 0, branches to /install, /api and /guides are crawled, /login is outside scope. At depth 2, /install/linux, /api/auth, /api/items and /guides/rag are crawled, /install?ref=nav is a duplicate canonical and dropped, /private/ is disallowed by robots.txt. At depth 3, /api/items/v2 is crawled and /guides/print is outside the path rules
What a recursive crawl keeps and drops: scope rules, duplicate canonicals and robots.txt decide which pages reach your index.

A recursive crawl starts from one URL, finds every page in scope through the sitemap and the links on each page, renders and cleans each one, then splits it into chunks with metadata. Scope rules decide what gets in, and content hashes decide what has to be embedded again on the next run.

  1. Discover. Read the sitemap, then follow internal links breadth first. Sitemaps find pages nothing links to; links find pages the sitemap forgot. Our tutorial How to Crawl an Entire Website tests both methods on a 73-page site.
  2. Scope. Stay on one domain, set a page limit and a depth, include the docs path, exclude tag pages, login pages and query-string duplicates. Normalise URLs and drop pages whose canonical you've already seen.
  3. Render. Load each page in a real browser, so content built by JavaScript is there. On sites that check for bots, this is where plain crawlers come back with challenge pages.
  4. Clean. Keep the main content as Markdown: headings, lists, tables and code blocks survive, navigation and footers go.
  5. Chunk and label. Split by heading, attach URL, title, section and crawl date to every chunk, and store a hash of the page.
  6. Refresh. On the next crawl, compare hashes and re-embed only changed pages. Delete chunks for pages that returned 404, not pages that merely failed this time.

Crawling APIs that export a site map before you crawl

Several crawl APIs return a site's full URL list without fetching every page, which is the cheapest way to size a RAG crawl, pick the sections worth indexing and spot orphan pages. GcrawlAI, Firecrawl, Tavily, Olostep and Bright Data document a map step; Cloudflare reads sitemaps inside its crawl.

API Map or URL discovery What it returns Cost
GcrawlAI Map (links) endpoint Every URL from robots.txt sitemaps, sitemap indexes and internal links, with duplicates and asset URLs dropped One request per map call
Firecrawl Map endpoint The site's URLs 1 credit per call
Tavily Map endpoint Site structure, optionally steered by instructions 1 credit per 10 pages (2 with instructions)
Olostep Maps endpoint All URLs on a site, paginated with a cursor Per request
Bright Data Crawl API Site mapping Site structure in one request Per record
Cloudflare /crawl Inside the crawl job (source: sitemaps, links or all) Crawled pages, not a separate URL list Browser hours

This script maps a site with GcrawlAI and writes the URLs to a standard sitemap.xml and a plain urls.txt, filtered to one section. Feed the list to your own scheduler, compare it with last week's to find new pages, or crawl only the section you need with the script above. The offline check above runs it too.

map_to_sitemap.py

# pip install gcrawl-sdk
"""Map a site with GcrawlAI and export its URLs as sitemap.xml and urls.txt, before (or instead of) a full crawl.
GcrawlAI reads robots.txt sitemap lines, sitemap indexes and internal links, and drops asset and duplicate URLs.
One map call is one request."""
import os
from xml.sax.saxutils import escape

from gcrawl_sdk import GcrawlClient

client = GcrawlClient(api_key=os.getenv("GCRAWL_API_KEY", "gc-YOUR_API_KEY"))  # base_url="http://localhost:8000" if self-hosted


def export_sitemap(start_url: str, limit: int = 5000, geo: str = "US", prefix: str = ""):
    result = client.links(url=start_url, limit=limit, same_domain_only=True, include_subdomains=False, geo=geo, wait=True)
    urls = sorted({(u.get("url") if isinstance(u, dict) else u) for u in (result.links or [])} - {None})
    urls = [u for u in urls if u.startswith(prefix or start_url)]       # e.g. keep only /docs/ for a RAG index
    with open("sitemap.xml", "w", encoding="utf-8") as f:
        f.write('<?xml version="1.0" encoding="UTF-8"?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n')
        f.writelines(f"  <url><loc>{escape(u)}</loc></url>\n" for u in urls)
        f.write("</urlset>\n")
    with open("urls.txt", "w", encoding="utf-8") as f:
        f.write("\n".join(urls) + "\n")
    print(f"{len(urls)} URLs exported to sitemap.xml and urls.txt")
    return urls


if __name__ == "__main__":
    export_sitemap("https://docs.example.com/", prefix="https://docs.example.com/")

Markdown or JSON: which output does your RAG pipeline need?

Most RAG pipelines want both: Markdown for the text you embed, and JSON for the metadata around it. Schema JSON, where an LLM fills fields you define, is a third thing, and it's the part that usually costs extra. Pick the output by what retrieval needs to filter on, not by what the API happens to return.

Output What it is Use it for Cost on the APIs above
Markdown The page's main content with structure kept Chunking and embedding Base price everywhere; GcrawlAI one request a page
JSON records Chunks with URL, title, section, date and hash Citations, filters, refresh Free: you build them from Markdown and metadata, as the script above does
Schema JSON Fields an LLM extracts to a schema (price, author, version) Product data, structured answers Firecrawl +4 credits a page; ScrapeGraphAI Extract 5 credits; Cloudflare via Workers AI; GcrawlAI through your own LLM or extractors

If you need structured fields from a few page types, our guide to extracting structured data shows the selector route, which costs nothing per page once written.

What crawling 10,000, 100,000 and 1,000,000 pages costs

On list prices, GcrawlAI is not the cheapest way to crawl unprotected pages to Markdown. ScrapeGraphAI, Firecrawl and Olostep cost less at 100,000 pages a month, and Spider's own estimate is far lower. GcrawlAI's price holds on protected and JavaScript sites, where every page is still one request and failed pages are free.

The script finds the cheapest published way to buy each monthly volume in each vendor's own unit: plans with overage or top-up packs, pay as you go, or the vendor's usage estimate. It's list-price arithmetic, not a benchmark.

cost_crawl.py

"""What crawling 10K, 100K and 1M pages a month to Markdown costs on each crawl API, at list price.
For each vendor: the cheapest published way to buy that volume in the vendor's own unit
(plan + overage or top-up packs, pay as you go, or a usage-based estimate the vendor publishes).
Vendor pages checked Nov 14, 2026. List-price arithmetic, not a benchmark."""


def plan_cost(n, plans, topup=None):
    """plans: [(price, units_included, overage_per_unit or None)]; topup: per-unit pack price usable on any plan."""
    best = None
    for price, incl, over in plans:
        extra = max(0, n - incl)
        rate = over if over is not None else topup
        if extra and rate is None:
            continue
        cost = price + (extra * rate if extra else 0)
        best = cost if best is None else min(best, cost)
    return best


K = 1_000
VENDORS = {
    # GcrawlAI: 1 request per page, rendering + residential proxy + stealth included; overage per request
    "GcrawlAI": lambda n: plan_cost(n, [(29, 20 * K, 1.60 / K), (79, 80 * K, 1.10 / K),
                                        (199, 250 * K, 0.90 / K), (499, 750 * K, 0.75 / K)]),
    # Firecrawl, monthly billing: crawl 1 credit/page; extra credits sold in packs per plan
    "Firecrawl": lambda n: min(plan_cost(n, [(19, 5 * K, 5 / K)]), plan_cost(n, [(99, 100 * K, 2.5 / K)]),
                               plan_cost(n, [(399, 500 * K, 2 / K)]), plan_cost(n, [(749, 1000 * K, 1 / K)])),
    # Tavily: crawl = map (1 credit / 10 pages) + basic extract (1 credit / 5 pages) = 0.3 credits/page
    "Tavily Crawl": lambda n: plan_cost(0.3 * n, [(0, 1 * K, 0.008), (30, 4 * K, 0.008), (100, 15 * K, 0.008),
                                                  (220, 38 * K, 0.008), (500, 100 * K, 0.008)]),
    # Olostep: 1 request per crawled page; 6-month credit packs ($5 per 2,000) top up a plan
    "Olostep": lambda n: plan_cost(n, [(9, 5 * K, None), (99, 200 * K, None), (399, 1000 * K, None),
                                       (1100, 3000 * K, None)], topup=5 / 2000),
    # ScrapeGraphAI: crawl = 2 credits per job + 1 credit per Markdown page; no published overage
    "ScrapeGraphAI": lambda n: plan_cost(n, [(17, 10 * K, None), (85, 100 * K, None), (425, 750 * K, None)]),
    # Bright Data Crawl API: $1.50 per 1,000 records pay as you go; $499 Growth plan at $1.30 list
    "Bright Data Crawl API": lambda n: min(1.5 * n / K, max(499, 1.3 * n / K)),
}
# Spider bills bandwidth + compute, not pages; its pricing guide estimates ~$80-$160 for 1M pages
# in Chrome mode with Markdown. Scaled linearly here as a range, labelled as the vendor's estimate.
SPIDER = (0.08, 0.16)

if __name__ == "__main__":
    vols = [10 * K, 100 * K, 1000 * K]
    print(f"{'Crawl API (Markdown out)':26}{'10K pages':>12}{'100K pages':>13}{'1M pages':>12}   per 1K at 100K")
    rows = []
    for name, fn in VENDORS.items():
        costs = [fn(v) for v in vols]
        rows.append((costs[1] if costs[1] is not None else 1e9, name, costs))
    for _, name, costs in sorted(rows):
        cells = [f"${c:,.0f}" if c is not None else "custom" for c in costs]
        per = f"${costs[1] / 100:.2f}" if costs[1] is not None else "-"
        print(f"{name:26}{cells[0]:>12}{cells[1]:>13}{cells[2]:>12}   {per}")
    lo, hi = SPIDER
    print(f"{'Spider (vendor estimate)':26}" + "".join(f"{'$' + format(lo * v / K, ',.0f') + '-' + format(hi * v / K, ',.0f'):>12}" for v in vols)
          + f"   ${lo:.2f}-{hi:.2f}")
    print("JSON per page (schema extraction): Firecrawl adds 4 credits (5 per page), so 100K JSON pages = $399 (Growth).")
    print("GcrawlAI returns Markdown + SEO metadata as JSON records; schema extraction runs on your own LLM.")
    print("Not modelled: Apify (compute units), Crawl4AI Cloud (plan prices not shown), Cloudflare /crawl (browser hours).")

cost_crawl_output.txt (printed output)

Crawl API (Markdown out)     10K pages   100K pages    1M pages   per 1K at 100K
ScrapeGraphAI                      $17          $85      custom   $0.85
Firecrawl                          $44          $99        $749   $0.99
Olostep                            $22          $99        $399   $0.99
GcrawlAI                           $29         $101        $686   $1.01
Bright Data Crawl API              $15         $150      $1,300   $1.50
Tavily Crawl                       $16         $220      $2,100   $2.20
Spider (vendor estimate)          $1-2       $8-16     $80-160   $0.08-0.16
JSON per page (schema extraction): Firecrawl adds 4 credits (5 per page), so 100K JSON pages = $399 (Growth).
GcrawlAI returns Markdown + SEO metadata as JSON records; schema extraction runs on your own LLM.
Not modelled: Apify (compute units), Crawl4AI Cloud (plan prices not shown), Cloudflare /crawl (browser hours).
Bookshelf chart of the monthly bill for crawling 100,000 pages to Markdown, book height equals cost: Spider estimate $8 to $16, ScrapeGraphAI $85, Firecrawl $99, Olostep $99, GcrawlAI $101, Bright Data $150, Tavily $220
The list-price bill for 100,000 crawled pages a month on each API.

Three things move these numbers. Schema JSON multiplies Firecrawl's per-page price by five. Spider's usage billing rises with page weight and render time. And on sites that block bots, plain or HTTP-mode crawls return challenge pages you still have to re-run, which GcrawlAI doesn't bill. For the wider scraping market, see our list of the cheapest reliable scraping APIs.

GcrawlAI's plans, from gcrawlai.com/pricing:

Plan Price Requests/month Concurrent Per 1,000 Overage per 1,000
Free $0, no card 50 2 - -
Starter $29/month 20,000 10 $1.45 $1.60
Growth $79/month 80,000 35 $0.99 $1.10
Pro $199/month 250,000 50 $0.80 $0.90
Scale $499/month 750,000 100 $0.67 $0.75
Enterprise Volume pricing 2M+ Custom - -
Self-hosted $0 (MIT) Your servers Your servers Infrastructure only -

Annual billing charges 10 months for 12, unused requests on paid plans roll over once, and Indian customers are billed in INR with a GST invoice. Failed requests are never charged on any plan.

"If your crawl is 100,000 clean docs pages, pick the cheapest line in that table and you'll be fine. Our pricing earns its keep on the other kind of site, the one that answers a plain crawler with a challenge page. There, a failed page costs you nothing and a good one costs one request."

Rajesh Kannan, Co-Founder & CTO at Gramosoft

Which web crawling API should you choose?

Choose by the sites you crawl first, then by output and budget. If any of your sources block bots or build content with JavaScript, start with GcrawlAI. If they're clean static docs and price per page is all that matters, the cheaper rows in the cost table are worth a test.

If you need… Pick Why
Docs and help centers into a RAG index GcrawlAI Sitemap + link discovery, Markdown and metadata, one request a page
Sites that block bots or need JavaScript GcrawlAI Real browser, residential IPs and stealth in every request; failed pages free
Weekly refresh of a large corpus GcrawlAI with content hashes Async crawls with webhooks; re-embed only changed pages
Agents that crawl on their own GcrawlAI via MCP Map, crawl, scrape and search as agent tools
Schema JSON from every page GcrawlAI + your LLM, or Firecrawl JSON Run extraction on the Markdown, or pay 5 credits a page
Data that must stay on your servers GcrawlAI self-hosted MIT core, no per-page fee
Lowest list price on unprotected pages GcrawlAI for protected sources; ScrapeGraphAI, Olostep or Spider for the rest Mix by source: cheap crawler on static docs, GcrawlAI where pages fight back
Already on Cloudflare Workers, sites that allow crawling Cloudflare /crawl; GcrawlAI for protected sites Browser hours you already pay for; it won't pass CAPTCHAs

When to run your own crawler instead

Run your own crawler when the corpus is large, steady and yours to keep, or when compliance rules out a third-party API and you need a custom web crawler API of your own. The open-source route with browser, proxies and Markdown already built is the MIT-licensed GcrawlAI core, running the same crawl as the Cloud API.

The self-hosted core is a FastAPI service with Celery workers on Redis, PostgreSQL and a stealth Playwright layer. It maps the sitemap and links before crawling, drops duplicate canonicals and rotates residential proxies by country, using proxy accounts you supply. Point the SDK's base_url at your server (http://localhost:8000 by default) and the script above runs unchanged. The setup steps are in the code download and on GitHub, and our list of the best open-source web crawlers compares the self-hosted options in detail.

"Self-hosting is the right call once crawling becomes part of your platform rather than a feature of it. We kept the core MIT so teams can make that move without rewriting their pipeline."

Rajesh Kannan, Co-Founder & CTO at Gramosoft

Crawling responsibly: robots.txt, terms and what to leave out

Crawl only public pages you're allowed to collect, follow the site's robots.txt and terms, and keep request rates gentle. Leave out logins, paywalls and personal data. For your own docs or a partner's, ask for a sitemap or an export first, because it's faster and more complete than any crawl.

Some crawlers enforce this for you: Cloudflare's endpoint honors robots.txt, crawl-delay and Content Signals. With any API, set a page limit, exclude account and checkout paths, and record where every chunk came from so you can delete it on request. Our explainer What Is Web Scraping? How It Works, Uses & Legality covers the legal basics. This article isn't legal advice.

Frequently asked questions

Which crawling APIs export a site map?

GcrawlAI, Firecrawl, Tavily and Olostep have map endpoints that return a site's URLs without crawling every page, and Bright Data's Crawl API maps site structure. GcrawlAI reads robots.txt sitemaps, sitemap indexes and internal links, drops duplicates, and one map call is one request.

Which crawl API converts whole websites into Markdown or JSON for RAG?

GcrawlAI for most teams, because every page goes through a real browser and residential proxies, the output is Markdown plus metadata, failed pages are free, and it connects through an SDK, MCP or a self-hosted MIT core. Firecrawl suits teams that want many SDKs; ScrapeGraphAI and Olostep cost less on unprotected sites.

Should I store crawled pages as Markdown or JSON?

Both. Embed the Markdown text, and keep each chunk as a JSON record with its URL, title, section, crawl date and a page hash. The JSON is what lets you cite sources, filter by section and re-embed only pages that changed.

How do I keep a RAG index fresh without re-embedding everything?

Hash each page's Markdown on every crawl and compare it with the last run. Re-embed changed pages, keep unchanged ones, and delete chunks only for pages that are really gone. The crawl itself still visits every page in scope.

Is GcrawlAI open source, and can I use it commercially?

Yes. The GcrawlAI core is MIT-licensed on GitHub, so you can self-host and use it commercially at no cost, bringing your own proxy accounts. The managed Cloud API adds hosted browsers and proxies, with 50 free requests a month and no card.

How much does crawling 100,000 pages a month cost?

On list prices checked November 14, 2026: about $85 on ScrapeGraphAI, $99 on Firecrawl or Olostep, $101 on GcrawlAI, $150 on Bright Data and $220 on Tavily. Spider estimates $8 to $16 on usage billing. Schema JSON raises Firecrawl to $399.

Can a crawling API get past Cloudflare and other bot checks?

Some can, on public pages. GcrawlAI loads every page in a real browser with residential IPs and stealth escalation, and doesn't bill pages that still fail. Cloudflare's own /crawl endpoint doesn't pass CAPTCHAs or Turnstile, and HTTP-mode crawlers usually return challenge pages.

RK

Rajesh Kannan M

Rajesh Kannan M is Co-Founder & CTO of Gramosoft in Chennai, where he leads the team building GcrawlAI, an open-source web scraping, crawling and Search API platform for developers and AI agents. LinkedIn

Crawl the whole site, pay per page that works

Start with 50 free requests a month, no card. GcrawlAI crawls sites into Markdown and metadata through a real browser and residential proxies, one request per page, through the Cloud API, Python SDK or MCP, or self-hosted under MIT. Failed pages are never charged.

CTA Illustration