Web Search Engine Demo

This project is a full-stack, self-hosted web search engine prototype.

It crawls content, stores raw pages and links, computes ranking signals (including PageRank), indexes documents in Elasticsearch, and serves ranked results through a FastAPI API and a React frontend.

What this project is

• A practical end-to-end search stack you can run locally with Docker Compose.
• A playground for experimenting with crawler quality, ranking, retrieval, and query UX.
• A benchmarkable system with evaluation scripts for relevance (hit@k, MRR) and latency.

End goal

The end goal is to build a reliable, explainable search system that can:

1. Continuously ingest useful web content at scale.
2. Rank results with better relevance than naive keyword matching.
3. Serve fast, stable search responses through a simple product UI.
4. Improve over time using measurable metrics (quality + performance) rather than guesswork.

In short: move from a crawler/indexing prototype to a production-ready retrieval foundation.

Architecture

• crawler/ — Scrapy spider (scrapy-redis based), URL discovery, page extraction, and Postgres writes.
• indexer/ — Reads pages/links from Postgres, computes PageRank, and indexes to Elasticsearch.
• api/ — FastAPI service (/search, /suggest, /chat, /health) backed by Elasticsearch.
• frontend/ — React + Vite search UI.
• scripts/ — evaluation and operational utilities (search_eval.py, stress_eval.py, etc.).

Core infrastructure (via docker-compose.yml):

• Redis (crawl frontier)
• PostgreSQL (raw pages + link graph)
• Elasticsearch (search index)
• Crawler, indexer, API, frontend services

Search pipeline

1. Seed URLs are loaded (curated/broad/Tranco or explicit seed file/env values).
2. Crawler fetches pages (respecting robots/politeness), extracts text/metadata/links.
3. Pages + link graph are stored in Postgres.
4. Indexer computes PageRank + site-level authority signals.
5. Documents are indexed into Elasticsearch.
6. API executes search + suggestions and returns ranked results to the frontend.

Quick start

From the repo root:

docker compose up -d --build redis postgres elasticsearch
docker compose up -d --build crawler
docker compose run --rm indexer python sync.py
docker compose up -d --build api frontend

Then open:

• Frontend: http://localhost:5173
• API docs: http://localhost:8000/docs

Evaluate quality and performance

Run from the repo root once API is up:

python scripts/search_eval.py
python scripts/stress_eval.py

• search_eval.py tracks retrieval quality with hit@1, hit@3, hit@5, and MRR.
• stress_eval.py tracks latency and failure patterns under a fixed query suite.

Current status

• Works as a local full-stack demo for crawling + indexing + search.
• Includes relevance/latency evaluation loops.
• Suitable as a base for ranking improvements, recrawl freshness strategies, and future AI-assisted retrieval features.