Production-minded RAG project with:
- A notebook orchestrator for interactive workflows.
- A modular Python package for testable core logic.
- CI quality gates, smoke checks, benchmarks, and structured docs.
rag-pipeline is designed for teams that want notebook speed without notebook fragility.
Core logic is moved into src/rag_pipeline, while the notebook remains a thin orchestration layer.
- Hybrid retrieval architecture (dense + BM25 + reranking).
- Manifest-based ingestion safety with atomic writes.
- Chunked stale cleanup with bounded-memory deletion.
- Offline and env-gated live smoke checks.
- Notebook validation and architectural pattern auditing.
- CLI-first developer workflow for testing, auditing, and benchmarking.
.
|-- Python3finale.ipynb
|-- src/rag_pipeline/
| |-- cli.py
| |-- query_engine.py
| |-- chroma_pipeline.py
| |-- processing.py
| |-- enrichment.py
| |-- bm25_utils.py
| |-- manifest.py
| |-- cleanup.py
| |-- storage.py
| |-- config.py
| |-- observability.py
| `-- ...
|-- tests/
|-- scripts/
|-- docs/
`-- .github/workflows/ci.yml
python -m pip install -r requirements-dev.txt
python -m pip install -e .# test suite
rag-pipeline test
# notebook checks
rag-pipeline validate-notebook Python3finale.ipynb
rag-pipeline audit-notebook Python3finale.ipynb
# smoke checks
rag-pipeline smoke
rag-pipeline smoke --live
# benchmark
rag-pipeline benchmark --samples 20000 --output benchmarks/latest.jsonUse Python3finale.ipynb for interactive iteration.
Keep logic changes in src/rag_pipeline and use notebook wrappers only for orchestration.
CI validates:
- tests
- notebook schema/syntax
- notebook architectural patterns
- offline smoke
- CLI smoke
Main runtime knobs can be supplied via environment variables (for example):
PIPELINE_VERSIONGROQ_TIMEOUTRETRIEVAL_TOP_KRERANK_CANDIDATESFINAL_TOP_KMAX_CHECKPOINT_BYTES
This project is licensed under the MIT License.