feat: add fastCRW search provider

[Recep S (us)]

What

Adds crw (fastCRW) to the SearchAPI options, alongside tavily and the native search modes.

Why

It runs 100% locally — including stealth, JS rendering, and proxy rotation

fastCRW is a fully open-source (AGPL) scraping engine in a single ~8MB Rust binary (~6MB RAM at idle). Anti-bot/stealth, BYO-proxy rotation, and JS/SPA rendering all ship in the open core — no extra services, no flags, no asterisks.

Firecrawl's OSS build works for plain HTTP pages, but its real anti-bot path (fire-engine — Cloudflare JS-challenge bypass, stealth UA rotation, headless fallback) is gated behind a cloud-only flag. Self-hosting Firecrawl gives you a multi-service stack that still falls back to bare Playwright on protected or JS-heavy sites. fastCRW ships all of that in the open core in one binary, so a self-hosted fastCRW instance can reach JS-heavy and bot-protected pages that a self-hosted Firecrawl cannot.

Higher truth-recall and faster than Firecrawl on Firecrawl's own benchmark

On Firecrawl's published benchmark dataset: truth-recall 63.74% vs 56.04%, with faster median latency (p50 ~1.9s vs ~2.3s).

Search = SearXNG breadth + a calibrated accuracy layer on top

crw is not an alternative to SearXNG — it is built on top of it. SearXNG is the metasearch aggregator underneath; crw adds a quality layer above: query expansion (multi-variant rewrite), content-aware reranking (re-scoring by fetched content rather than SearXNG's content-blind ordering), and category routing (research queries fan out to arXiv / Semantic Scholar / Google Scholar, code queries to GitHub). The result is SearXNG's breadth plus a measurable accuracy improvement, all open-source (AGPL) and self-hostable with configurable engines — giving open_deep_research a flat-priced (1 credit = 1 page, no charge on failure) search backend with no per-query rate limits.

Why the diff is tiny

fastCRW's /v1/search is API-compatible with Firecrawl, so the integration mirrors the existing tavily_search dispatch with no new abstractions.

Changes (additive only)

• src/open_deep_research/configuration.py: crw added to the SearchAPI enum.
• src/open_deep_research/utils.py: a crw_search() implementation + dispatch branch mirroring tavily_search.
• tests/test_crw_search.py, .env.example, README note.

Config

SEARCH_API=crw, CRW_API_KEY from https://fastcrw.com/dashboard (free tier).

Happy to adjust — I maintain the integration and can provide free credits.