By Rajesh Kannan, Co-Founder & CTO, Gramosoft · Last updated: · Versions and prices checked November 15, 2026; speed test run on our own machine the same day
The fastest web crawler is the one that brings pages back, not the one with the biggest number on a static test page. This guide ranks the nine best web crawler software tools in 2026, starting with GcrawlAI, and puts six open-source crawlers through the same local speed test, so you can see which ones wait on the network, which ones burn CPU, and what a browser costs you in pages per second.
TL;DR
- Best overall: GcrawlAI. A hosted crawl that renders every page in a real browser through residential IPs, up to 100 pages in parallel by plan, one request per page and failed pages never charged. Cloud API, Python SDK, MCP for AI agents and an MIT core you can self-host.
- Fastest in our HTTP test: Colly (Go) and spider-rs (Rust), both about 305 pages a second at 16 requests in flight and over 1,000 at 64, against a ceiling of 320 and 1,280.
- Fastest Python framework: Scrapy, at 214 pages a second, rising to 331 with more concurrency. A new Scrapy project ships capped at one request a second per site, so change two settings first.
- Browsers are the real slowdown: plain Playwright managed 20 pages a second at 4 tabs, Crawl4AI and Crawlee's Playwright crawler about 7, with 1.2 to 2 GB of memory.
- SEO audits: Screaming Frog SEO Spider, free up to 500 URLs and £199 a year after that.
What is the fastest web crawler software in 2026? For sites that block bots or need JavaScript, GcrawlAI, because it renders every page in a real browser through residential IPs, up to 100 pages in parallel. On open, plain-HTML sites, Colly (Go) and spider-rs (Rust) were fastest in our test, over 1,000 pages a second, with Scrapy the fastest Python option.
Quick comparison (speed figures are from our local test; details and sources follow)
| Tool | Type | Language | Our test: pages a second | Renders JavaScript | Price |
|---|---|---|---|---|---|
| GcrawlAI | Hosted crawl API, SDK, MCP, MIT core | Python | Not run locally (hosted); 2 to 100 pages in parallel by plan | Yes, every page in a real browser | Free 50 requests a month; $0.67 to $1.45 per 1,000 |
| Colly | Open-source library | Go | 306 (1,042 at 64 in flight) | No | Free (Apache 2.0) |
| Spider | Open-source library + cloud API | Rust | 303 (1,119 at 64) | Yes, on pages that need it | Free (MIT); cloud pay as you go |
| Scrapy | Open-source framework | Python | 214 (331 at 64) | With a plugin | Free (BSD-3-Clause) |
| Crawlee | Open-source framework | Node.js, Python | 172 (Node), 86 (Python) | Yes, with Playwright or Puppeteer | Free (Apache 2.0) |
| Crawl4AI | Open-source library | Python | 6.9 (browser, 4 at a time) | Yes | Free (Apache 2.0) |
| Firecrawl | Hosted crawl API | Any (REST, SDKs) | Not tested; 2 to 100 browsers by plan | Yes | 1,000 free credits; $19 to $749 a month |
| Apify | Cloud platform for crawlers | Node.js, Python | Not tested; 5 to 256 runs at once by plan | Depends on the Actor | $5 free usage; $19 to $999 a month plus usage |
| Screaming Frog SEO Spider | Desktop SEO crawler | Windows, macOS, Linux app | Not tested; 5 threads by default | Yes, optional rendering | Free up to 500 URLs; £199 a year |
The 9 best web crawler software tools in 2026
These nine tools cover three ways to crawl: a hosted API that handles browsers and blocks, an open-source crawler on your own servers, and a desktop app for SEO audits. They're ranked by how fast each gets usable pages back from real sites, with GcrawlAI first and measured speeds for the six we ran locally.
1. GcrawlAI: fastest route through protected and JavaScript sites
Best for: teams that need whole sites as Markdown or HTML, including sites that block bots or build pages with JavaScript. License: MIT core, plus a managed Cloud API. Language: Python (FastAPI, Celery, Redis, PostgreSQL, stealth Playwright).
Raw request speed stops mattering the moment a site answers with a challenge page. GcrawlAI is built for that case: you send one URL, it maps the site from its sitemaps and internal links, then renders each page in a real browser through an in-country residential IP, with stealth escalation and retries inside the same request. Pages come back as Markdown, HTML or JSON. The Cloud API runs up to 100 pages in parallel on the Scale plan, so a crawl's speed is set by your plan and the target site, not by your laptop.
- Map, then crawl: the map endpoint returns a site's URLs from robots.txt sitemaps, child sitemaps and internal links, with canonical duplicates dropped, so the crawl only fetches pages that exist.
- Parallel by plan: 2 pages at a time on Free, 10 on Starter, 35 on Growth, 50 on Pro and 100 on Scale, with async crawl jobs and webhooks from Growth up.
- One request includes everything: JavaScript rendering, rotating residential proxies, anti-bot handling for Cloudflare, DataDome, Akamai and PerimeterX, and Markdown or JSON output. No credit multipliers.
- Failed pages are free: blocked pages, CAPTCHAs, timeouts and server errors are never billed.
- Built for agents: a Python SDK, an MCP server for AI agents, the Search API and Deep Search, screenshots and offline HTML bundles on one account. Try a crawl without code in the Playground.
- Self-hostable: the same crawler runs on your servers under MIT, with more Celery workers for more pages in parallel.
Map the site, then crawl it in one job. The script times the crawl, so you can compare plans on your own targets:
gcrawlai_crawl.py
# pip install gcrawl-sdk
"""Map a site first, then crawl it in one async job. Every page is one request; failed pages are free."""
import time
from gcrawl_sdk import GcrawlClient
client = GcrawlClient(api_key="gc-YOUR_API_KEY") # or GCRAWL_API_KEY; base_url="http://localhost:8000" when self-hosted
if __name__ == "__main__":
start = "https://docs.example.com/"
# 1. Map: the site's URLs from its sitemaps and internal links, without rendering every page.
site_map = client.links(url=start, limit=5000, same_domain_only=True, wait=True)
print(len(site_map.links), "URLs found")
# 2. Crawl: GcrawlAI renders each page in a real browser through residential IPs, in parallel
# up to your plan's concurrency (2 on Free, 35 on Growth, 100 on Scale).
t0 = time.perf_counter()
job = client.crawl(url=start, limit=1000, formats=["markdown"], same_domain_only=True, wait=True)
took = time.perf_counter() - t0
print(f"{job.status}: {len(job.pages)} pages in {took:.0f}s "
f"({len(job.pages) / took:.1f} pages/s), {job.pages_failed} failed (not billed)")
for page in job.pages[:3]:
print(page.url, len(page.markdown or ""), "chars of Markdown")
To run the open-source core yourself, with your own proxy accounts:
selfhost.sh
# GcrawlAI open-source core (MIT): Python, PostgreSQL and Redis git clone https://github.com/GramosoftAI/GcrawlAI.git && cd GcrawlAI python -m venv venv && source venv/bin/activate pip install -r requirements.txt && playwright install cp .env.example .env # PostgreSQL, Redis and your proxy accounts python -m api.core.db_setup # creates the tables redis-server & celery -A web_crawler.crawler.celery_config worker -l info --concurrency 8 & # more workers = more pages in parallel uvicorn api.api:app --port 8000
Pricing: a free plan of 50 requests a month with no card, paid plans from $0.67 to $1.45 per 1,000 requests, and a self-hosted core that costs nothing. Every crawled page is one request, and failed pages are never charged. The full plan table is in the cost section below.
Where it falls short: it wasn't in our local speed test, because a hosted crawl depends on your plan's concurrency and the target site rather than your machine, so run your own sites through the free plan first. It has no prompt-driven browsing agent like Firecrawl's /agent, the ecosystem is younger with fewer ready-made integrations, and self-hosting means bringing your own proxy accounts.
"Pages a second on a test site is easy. What counts is pages a second that come back with content. A crawler that does 1,000 requests a second into a wall of 403s has crawled nothing, so we put the browser and the residential IP inside every request."
Rajesh Kannan, Co-Founder & CTO at Gramosoft
2. Colly: fastest HTTP crawler in our test
Best for: Go teams crawling open, plain-HTML sites at high volume. Version: 2.3.0. License: Apache 2.0.
Colly is a small Go library: you register callbacks for elements and responses, turn on async mode and set a parallelism limit per domain. In our test it hit 306 pages a second at 16 requests in flight, 96% of what the 50 ms delay allows, and 1,042 a second at 64, using 52 to 64 MB of memory. Its README promises over 1,000 requests a second on a single core, and that matched what we saw.
Where it falls short: it doesn't run JavaScript, has no proxy pool or anti-bot handling, and you write the queue persistence and retries yourself.
3. Spider: Rust speed with the smallest memory
Best for: Rust teams, and anyone who wants a cheap hosted crawl of open sites. Version: spider crate 2.53.9. License: MIT, plus Spider Cloud.
Spider calls itself "the fastest web crawler and scraper for Rust", and its README says it streams pages as they arrive and renders JavaScript only on pages that need it. In our test it was the fastest crawler at 64 in flight, 1,119 pages a second, and the lightest at every setting, 20 to 26 MB. The same engine runs Spider Cloud, which is pay as you go: $1 buys 10,000 credits, bandwidth is $1 per GB, and the page estimates $0.002 to $0.0155 per 100 pages depending on mode.
Where it falls short: the library needs Rust (our release build took six minutes on two cores), the pricing page publishes no concurrency figure for pay as you go, and proxies bill at 1.5 to 4 times the base bandwidth rate.
4. Scrapy: the fastest Python framework, once you lift its limits
Best for: Python teams running large, structured crawls. Version: 2.20.0 (November 6, 2026). License: BSD-3-Clause.
Scrapy brings the queue, retries, duplicate filtering, throttling and export pipelines that other tools leave to you. It crawled 214 pages a second at 16 in flight and 331 at 64, with about 110 MB of memory, which puts it well ahead of Crawlee for Python, a little behind a hand-written httpx loop at 16 in flight and more than twice as fast as that loop at 64. One catch: a project created with scrapy startproject sets one request at a time per site and a one-second delay, so a fresh spider crawls about one page a second until you change the settings shown in the speed section.
Where it falls short: no JavaScript without the scrapy-playwright plugin, no proxies or anti-bot handling built in, and past about 300 pages a second it was limited by CPU on our two cores.
5. Crawlee: one API for HTTP and browser crawling
Best for: Node.js or Python teams that need both HTTP and browser crawling with persistent queues. Version: 3.18.2 for Node.js, 1.11.2 for Python. License: Apache 2.0.
Crawlee, from the Apify team, swaps between a fast HTTP crawler and a Playwright or Puppeteer crawler behind the same router, with request queues, sessions and proxy rotation you can switch on. Its Node.js Cheerio crawler held 172 pages a second at both 16 and 64 in flight, and the Python Parsel crawler managed 86 and 110: in both cases the parsing, not the network, set the pace. Its Playwright crawler did 6.4 pages a second with 4 browser pages; by default Crawlee caps a run's memory at a quarter of the machine's, 1.5 GB on our box.
Where it falls short: slower than Scrapy and the compiled crawlers on plain HTML in our test, the Node.js crawler used about 460 MB, and you supply the proxies.
6. Crawl4AI: browser crawling straight into Markdown
Best for: Python teams feeding LLMs and RAG pipelines from their own servers. Version: 0.9.5. License: Apache 2.0.
Crawl4AI drives a headless browser and turns every page into clean Markdown, with breadth-first and best-first deep crawl strategies. In our test its BFS crawl of 300 pages, four at a time, ran at 6.9 pages a second with 1.2 GB of peak memory, about a third the speed of a bare Playwright loop, because it also converts every page to Markdown and its breadth-first strategy works through the site level by level. For LLM work that conversion is the point.
Where it falls short: a browser on every page makes it the slowest kind of crawler on plain HTML, and there's no proxy pool or unblocking built in.
7. Firecrawl: hosted crawling with browsers by plan
Best for: teams that want a hosted crawl-to-Markdown API with a large ecosystem. License: AGPL-3.0 core, plus a hosted API.
Firecrawl's speed is sold as concurrent browsers. Its pricing page lists 2 on Free, 5 on Hobby ($19 a month for 5,000 credits), 25 on Standard ($99 for 100,000), 50 on Growth ($399 for 500,000) and 100 on Scale ($749 for 1,000,000). A crawled page costs 1 credit, and the JSON format adds 4 more.
Where it falls short: structured JSON costs five times a Markdown page, and the 25 browsers on Standard are fewer than the 35 pages in parallel on GcrawlAI's cheaper Growth plan.
8. Apify: run many crawlers in the cloud at once
Best for: teams that want ready-made crawlers (Actors) or a place to host their own Crawlee and Scrapy code. License: commercial platform; Crawlee is open source.
Apify's speed limit is the number of Actor runs at once and their memory. Its pricing page allows 5 concurrent runs on Free, 32 on Starter ($19 a month), 128 on Scale ($199) and 256 on Business ($999), with compute at $0.20 down to $0.13 per compute unit (1 GB of RAM for an hour).
Where it falls short: the bill depends on run time, memory, proxies and each Actor's own fees, so you only know the cost after a test run.
9. Screaming Frog SEO Spider: desktop crawler for SEO audits
Best for: SEO teams auditing their own sites for broken links, redirects, titles and canonicals. Version: 24.3. License: commercial desktop app.
Screaming Frog crawls at 5 threads by default, and its speed settings let you raise the threads or cap URLs per second, with an auto throttle that backs off on 429s and 503s. It's free up to 500 URLs per crawl; a licence is £199 a year and unlocks the configuration, including speed.
Where it falls short: it's an audit tool, not a data pipeline: it runs on your desktop, the free version can't change crawl speed, and getting page content into your own code takes exports.
How fast is each crawler? Our local speed test
We ran six HTTP crawlers and three browser crawlers against the same local 5,001-page site, with a fixed 50 ms delay on every response to stand in for the network. On plain HTML, Colly and spider-rs came within a few percent of the ceiling; browsers ran 15 to 48 times slower than the fastest HTTP crawler.
How we tested. One machine with 2 vCPUs and 7 GB of RAM, on November 15, 2026. A small Go server served 5,000 linked pages of about 15 KB each and counted every request, so each run had to fetch all 5,001 URLs exactly once. Every crawler followed every same-site link and read the page title, with the concurrency set to 16 and then 64 requests in flight. Browser crawlers fetched 300 pages, four at a time. We took the median of three runs and the peak memory of the whole process tree. GcrawlAI, Firecrawl, Apify and Screaming Frog weren't run: the hosted ones depend on plan and network, and Screaming Frog is a desktop app.
lab_results.txt (printed output)
Local crawl-speed lab, November 15, 2026 (2 vCPU, 7 GB RAM, Linux; 50 ms per response) HTTP crawl: 5,001 pages, same site, median of 3 runs crawler conc seconds pages/s peak MB Colly 2.3 (Go) 16 16.4 305.9 52 spider-rs 2.53 (Rust) 16 16.5 303.1 20 httpx + asyncio (DIY) 16 17.3 288.7 67 Scrapy 2.20 16 23.4 214.2 108 Crawlee JS 3.18 (Cheerio) 16 29.2 171.5 458 Crawlee Python 1.11 (Parsel) 16 58.1 86.1 175 (ceiling at this concurrency) 16 320 spider-rs 2.53 (Rust) 64 4.5 1118.8 26 Colly 2.3 (Go) 64 4.8 1041.9 64 Scrapy 2.20 64 15.1 331.0 115 Crawlee JS 3.18 (Cheerio) 64 29.0 172.3 456 httpx + asyncio (DIY) 64 33.5 149.2 92 Crawlee Python 1.11 (Parsel) 64 45.7 109.5 193 (ceiling at this concurrency) 64 1280 Browser crawl: 300 pages, 4 at a time, median of 3 runs Playwright 1.65 (DIY) 4 14.9 20.1 1718 Crawl4AI 0.9.5 4 43.6 6.9 1196 Crawlee Python (Playwright) 4 47.2 6.4 2015 Every run fetched every page once (server-side count, 0 duplicates bar 1 for Scrapy's start URL).
Where does GcrawlAI fit in this test?
GcrawlAI isn't in the table because it can't take this test fairly. The Cloud API renders pages on GcrawlAI's servers through residential IPs, so it can't reach a test server on our own machine, and its speed depends on how many pages your plan renders at once and on the target site, not on your CPU. Every page also goes through a real browser, so it belongs with the browser crawlers, not the HTTP ones.
What you can work out is its ceiling. Pages a second is pages in parallel divided by seconds per page. If a protected, JavaScript-heavy page takes three seconds to render (an example, not a measurement), each plan gives you:
| GcrawlAI plan | Pages in parallel | Pages a second at 3 s a page | Pages a minute |
|---|---|---|---|
| Free | 2 | 0.7 | 40 |
| Starter | 10 | 3.3 | 200 |
| Growth | 35 | 11.7 | 700 |
| Pro | 50 | 16.7 | 1,000 |
| Scale | 100 | 33.3 | 2,000 |
Compare that with the browser crawlers above: four Playwright tabs reached 20 pages a second only because our test pages had no JavaScript, no bot checks and answered in 50 ms, and they needed 1.7 GB of memory on your machine. With GcrawlAI the browsers, residential IPs and retries run on GcrawlAI's side, and blocked pages cost nothing. To get a real number for your own sites, run the timed script in the GcrawlAI entry on the free plan: it prints pages a second at the end.
Three things stood out:
- Waiting beats parsing. With 16 requests in flight, the ceiling is 320 pages a second. Colly, spider-rs and the plain httpx loop sat close to it, because they spend their time waiting on the network. Crawlee for Python and Node.js didn't speed up with more concurrency: CPU spent on parsing and queueing was the limit.
- More concurrency isn't always faster. The compiled crawlers went from about 305 to over 1,000 pages a second at 64. Scrapy rose to 331. Our httpx loop got slower, from 289 to 149, as 64 connections fought over one Python process.
- Browsers cost seconds and gigabytes. A bare Playwright loop rendered 20 pages a second with four tabs and 1.7 GB of memory. Crawl4AI and Crawlee's Playwright crawler, doing more work per page, managed about 7.
The test and every crawler are in the code download, so you can rerun them on your own hardware. Treat the numbers as a comparison between tools on one machine, not a promise for your sites: real crawls are slowed by the site, not your CPU.
"The language mattered less than I expected. What separated the fast crawlers from the slow ones was whether they spent their time waiting on the network or burning CPU on every page. Add a browser and none of that matters any more: the page itself becomes the bottleneck."
Rajesh Kannan, Co-Founder & CTO at Gramosoft
What makes a web crawler fast?
Crawl speed is pages in parallel divided by seconds per page. Every crawler on this list is fast or slow for one of four reasons: how many requests it keeps in flight, how long each page takes to arrive, whether it renders JavaScript, and how hard it must slow down to stay welcome on the site.
- Concurrency: 16 requests at 50 ms each is at most 320 pages a second; 64 is 1,280. Raise it until the site or your CPU pushes back.
- Latency: a 500 ms server makes every crawler ten times slower than our test. You can't fix the site, so add concurrency or crawl less.
- Rendering: a browser adds seconds and hundreds of megabytes per page. Fetch plain HTML where the content is already in it, and render only pages that need it.
- Politeness: delays, robots.txt and rate limits cap speed on purpose. A crawler that ignores them gets 429s and blocks, which is slower in the end.
In Scrapy, these are the settings that decide speed. A new project starts with one request per site and a one-second delay:
scrapy_fast_settings.py
# settings.py: the Scrapy 2.20 settings that decide crawl speed. # A project made with `scrapy startproject` ships with CONCURRENT_REQUESTS_PER_DOMAIN = 1 and DOWNLOAD_DELAY = 1, # which caps a one-site crawl at about one page a second. Raise them only on sites that allow it. CONCURRENT_REQUESTS = 64 # default 16: requests in flight across all sites CONCURRENT_REQUESTS_PER_DOMAIN = 16 # template 1, default 8: the real cap when you crawl one site DOWNLOAD_DELAY = 0 # template 1 second: seconds between requests to the same site AUTOTHROTTLE_ENABLED = True # slows down by itself when the site's responses slow down AUTOTHROTTLE_TARGET_CONCURRENCY = 8.0 # default 1.0: average requests in flight it aims for DOWNLOAD_TIMEOUT = 30 # default 180: stop waiting on slow pages sooner COOKIES_ENABLED = False # default True: skip cookie handling you don't need LOG_LEVEL = "INFO" # default DEBUG logs every request and costs CPU ROBOTSTXT_OBEY = True # template True: keep it on for sites you don't own
To find which pages need a browser before you crawl, see how to crawl an entire website with sitemap and link discovery, and to keep a fast crawler from being flagged, match a real browser's identity as covered in our guide to user agents for web scraping.
What hosted crawlers cost, and how many pages they run at once
On a hosted crawler you buy speed as concurrency: how many pages render at the same time. GcrawlAI runs 2 to 100 pages in parallel by plan, Firecrawl 2 to 100 browsers, and Apify 5 to 256 Actor runs. Per page, Spider Cloud's published estimates are the cheapest on open sites.
| Service | Pages in parallel, by plan | Price | Billing unit |
|---|---|---|---|
| GcrawlAI | 2 Free, 10 Starter, 35 Growth, 50 Pro, 100 Scale | Free 50 requests; $29 to $499 a month | 1 request per page, browser and residential IP included; failed pages free |
| Firecrawl | 2 Free, 5 Hobby, 25 Standard, 50 Growth, 100 Scale | 1,000 free credits; $19 to $749 a month | 1 credit per page; JSON +4 credits |
| Apify | 5, 32, 128 or 256 Actor runs at once | $5 free usage; $19, $199 or $999 a month plus usage | Compute units ($0.20 to $0.13), proxies, Actor fees |
| Spider Cloud | Not published for pay as you go | $1 per 10,000 credits, no subscription | Bandwidth ($1 per GB) plus compute; about $0.02 to $0.155 per 1,000 pages |
| Screaming Frog | 5 threads by default, more with a licence | Free up to 500 URLs; £199 a year | Per user licence |
As a worked example, not a measurement: at 35 pages in parallel and three seconds per rendered page, a crawl moves at about 12 pages a second, or 700 a minute. Prices are from each vendor's page, checked November 15, 2026.
GcrawlAI's plans, with the pages each one runs in parallel:
| Plan | Price | Requests a month | Pages in parallel | Per 1,000 | Overage per 1,000 |
|---|---|---|---|---|---|
| Free | $0, no card | 50 | 2 | – | – |
| Starter | $29 a month | 20,000 | 10 | $1.45 | $1.60 |
| Growth | $79 a month | 80,000 | 35 | $0.99 | $1.10 |
| Pro | $199 a month | 250,000 | 50 | $0.80 | $0.90 |
| Scale | $499 a month | 750,000 | 100 | $0.67 | $0.75 |
| Enterprise | Volume pricing | 2M+ | Custom | – | – |
| Self-hosted | $0 (MIT) | Your servers | Your workers | Infrastructure only | – |
Annual billing charges 10 months for 12, unused requests on paid plans roll over once, and Indian customers are billed in INR with a GST invoice. The honest read: on open sites, Spider Cloud costs less per page and Firecrawl Standard ($99 for 100,000 pages) costs about the same as GcrawlAI ($101 on Growth with overage). GcrawlAI's price includes the real browser, residential IPs and anti-bot handling on every page, and failed pages cost nothing.
Which web crawler software should you choose?
Pick by what your sites throw back at you. For protected or JavaScript-heavy sites, a crawler with browsers and residential IPs built in; for open HTML at high volume, a compiled HTTP crawler or Scrapy; for SEO audits, a desktop app. GcrawlAI fits next to each of them for the pages they can't reach.
| Your situation | Pick | Where GcrawlAI fits |
|---|---|---|
| Sites that block bots or build pages with JavaScript | GcrawlAI | Real browser and residential IPs in every request, up to 100 pages in parallel |
| AI agents that crawl on their own | GcrawlAI with MCP | Agents call map, crawl and scrape as tools |
| Open, plain-HTML sites at very high volume | Colly or spider-rs | Send the blocked or JavaScript pages to GcrawlAI |
| Python team, large structured crawls | Scrapy | Keep Scrapy's crawl, fetch hard pages through the GcrawlAI SDK |
| Node.js team, HTTP and browser crawling | Crawlee | GcrawlAI replaces the browser and proxy fleet you'd run |
| Markdown for an LLM from your own servers | Crawl4AI | GcrawlAI returns Markdown too, with proxies and anti-bot included |
| A hosted crawl-to-Markdown API | GcrawlAI or Firecrawl | GcrawlAI: one request per page, JSON at no extra cost, failed pages free |
| SEO audit of your own site | Screaming Frog | Use GcrawlAI for the page content and screenshots your audit needs |
| Data must stay on your servers | GcrawlAI self-hosted | MIT core, more workers for more speed |
When crawling faster is the wrong goal
A crawler that runs at 1,000 pages a second can take a small site down. Speed is only useful inside what the site allows: its robots.txt, its rate limits and its terms. Map the site first, crawl only what changed, and back off on 429 and 503 responses. That's faster overall than retrying blocked pages.
Screaming Frog's own guide points out that a million-page site at five URLs a second takes more than two days, and that most servers don't want to be crawled faster. Agree a crawl rate with the site owner when you can, keep robots.txt on for sites you don't own, and don't crawl behind logins or collect personal data without a lawful basis. For the legal basics, see What Is Web Scraping?
"The fastest crawl is the one you don't repeat. Map the site, fetch what changed since last time, and the site barely notices you. Hammering it at full speed every night is how you end up crawling block pages."
Rajesh Kannan, Co-Founder & CTO at Gramosoft
Frequently asked questions
What is the fastest web crawler software in 2026?
For protected or JavaScript-heavy sites, GcrawlAI, because every page renders in a real browser through residential IPs, up to 100 pages in parallel. On open, plain-HTML sites, Colly and spider-rs were fastest in our local test, over 1,000 pages a second at 64 requests in flight, and Scrapy was the fastest Python framework.
Is GcrawlAI open source, and can I use it commercially?
Yes. The GcrawlAI core is MIT licensed on GitHub, so you can self-host it and use it commercially. The managed Cloud API adds residential IPs and stealth escalation, with a Python SDK and MCP for AI agents, a free plan of 50 requests a month and paid plans from $0.67 to $1.45 per 1,000 requests.
Why is my Scrapy crawler so slow?
Usually its settings. A project made with scrapy startproject sets CONCURRENT_REQUESTS_PER_DOMAIN to 1 and DOWNLOAD_DELAY to 1, about one page a second per site. Raise both on sites that allow it, turn on AutoThrottle, lower DOWNLOAD_TIMEOUT and set LOG_LEVEL to INFO. In our test Scrapy then crawled 214 to 331 pages a second.
How much slower is a browser crawler than an HTTP crawler?
In our test, a bare Playwright loop rendered 20 pages a second with four tabs, against about 306 for the fastest HTTP crawler at 16 requests in flight, and it used 1.7 GB of memory instead of about 50 MB. Crawl4AI and Crawlee's Playwright crawler ran at about 7 pages a second.
Which web crawler uses the least memory?
spider-rs, at 20 to 26 MB in our test, then Colly at 52 to 64 MB. Scrapy used about 110 MB, Crawlee's Node.js crawler about 460 MB, and the browser crawlers 1.2 to 2 GB. A hosted crawler such as GcrawlAI keeps the browsers off your machine entirely.
How fast can I crawl a website without getting blocked?
There's no safe number; it depends on the site. Follow robots.txt and any crawl-delay, start slow, and back off on 429 and 503 responses. Screaming Frog defaults to 5 threads for that reason. For protected sites, GcrawlAI paces and retries pages inside each request, and failed pages are never charged.
Which web crawler software is free?
Related guides
- Best Open-Source Web Crawlers in 2026 (Self-Hosted)
- How to Crawl an Entire Website: Sitemap and Link Discovery
- Web Scraping vs Web Crawling: What's the Difference?
- Why Companies Are Switching from Firecrawl to GcrawlAI: The Hidden Cost of 5x Credit Multipliers
- 9 Best Apify Alternatives in 2026: Scraping Without Compute Units
- Web Scraping for RAG: The Complete Engineering Guide to Clean LLM Data Ingestion in 2026
- Model Context Protocol (MCP) Web Scraping: How to Connect AI Agents to the Live Web
- 10 Best User Agents for Web Scraping in 2026: Current Strings, Matching Headers and Rotation
- Fix 405 Method Not Allowed and 500 Errors in Web Scraping
- 7 Cheapest Reliable Web Scraping APIs for High Volume in 2026
- What Is Web Scraping? How It Works, Uses & Legality