Data Ingestion Pipeline

[emincalyakaisskar]

WHY: As a user, I want to have a relevant chatbot in the sense that the data accessible by the RAG must be accessible quickly and updated with a real data ingestion system.

DoD:

• Identify the data sources for the different types (audio, video, text, web, pdf) in our use case
• We want a pipeline for each type of data: Video/Audio Pipeline, PDF Pipeline, Web Pipeline, Plain Text Pipeline
• Add connectors capable of retrieving data from these sources (with the right rights, take all data, update data), one connector per pipeline
• Define triggers according to source type (Event driven, Cron, manual)
• Manage data transformation (Video -> Audio -> Transcript, PDF -> OCR -> Text, Html parsing -> Text, ...)
• Chunking strategies: Intelligent according to data type
• Quality control
• Metadata (origin and provenance of data)

[emincalyakaisskar]

[onestardao]

your ambition here is 🔥 especially the multi-pipeline handling of audio/video/PDF/web as if they're native to RAG.

we’ve been down that road, and ran into a few semantic traps:

• pdf-to-ocr paths often distort chunk boundaries (No.1 on our internal failure map)
• raw media embeddings collapse the reasoning logic unless normalized properly (No.6)
• inconsistent text structure after HTML parsing or whisper can trigger silent breakdowns (No.4)

we ended up building a modular ingestion engine to stabilize those formats before even reaching the retriever.

happy to share the whole setup if you're exploring that direction — just say the word.