feat: add TwelveLabs video data connector (Pegasus)#5912
Open
mohit-twelvelabs wants to merge 2 commits into
Open
feat: add TwelveLabs video data connector (Pegasus)#5912mohit-twelvelabs wants to merge 2 commits into
mohit-twelvelabs wants to merge 2 commits into
Conversation
Adds an opt-in data connector that analyzes a video with the TwelveLabs Pegasus model and imports the resulting transcript/summary/scene breakdown as a workspace document. - collector: TwelveLabs loader (upload by URL -> index with Pegasus -> analyze) mirroring the existing YouTube transcript connector, plus the /ext/twelvelabs-video processor route - server: /ext/twelvelabs/video extension endpoint (admin/manager gated) - frontend: TwelveLabs connector option, data connector client method, icon, and i18n strings (en + seeded across all locales) - collector: offline unit tests for the loader (key/URL guards, index reuse-vs-create, prompt forwarding, document persistence) API key is read from request body or the TWELVELABS_API_KEY env var; no defaults are changed and existing connectors are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
Pull Request Type
Relevant Issues
No tracking issue exists yet. I'm aware
CONTRIBUTING.mdasks for a corresponding issue before a PR is merged — happy to open one (or have a maintainer link an existing request) if you'd like this considered. Opening the PR first so there's concrete, tested code to evaluate.Description
Adds an opt-in video data connector powered by the TwelveLabs Pegasus model. A user pastes a public video URL; the connector uploads it to a TwelveLabs index (reused if it already exists, otherwise created with Pegasus enabled), waits for indexing, runs
analyze, and saves the resulting transcript / summary / on-screen-context as a normal workspace document that can be embedded like any other source.This fills a real gap: AnythingLLM can already ingest YouTube transcripts (caption text only), but has no way to ingest the actual content of an arbitrary video — spoken words plus visual scenes and on-screen text. Pegasus produces all of that from any direct video URL, so users can chat over webinars, recordings, demos, and footage that have no captions.
It mirrors the existing YouTube connector end-to-end so it slots into the established patterns:
utils/extensions/TwelveLabsloader (upload → index → analyze →writeToServerDocuments, with the same path-safety/isWithinguards as the YouTube loader) + the/ext/twelvelabs-videoprocessor route./ext/twelvelabs/videoextension endpoint, admin/manager gated, behindvalidatedRequestlike every other connector, with a telemetry event.dataConnectorclient method, an icon, and i18n strings (English authored; seeded across all locales sotranslations:verifypasses — happy to have these properly translated).Non-breaking & opt-in: no defaults change, no existing connector is touched. The API key is read from the request body or the
TWELVELABS_API_KEYenv var; the key is never logged.The TwelveLabs JS SDK (
twelvelabs-js) is added only to thecollectorand lazy-required inside the handler, so it's never loaded unless this connector is actually invoked.Visuals (if applicable)
N/A (functional connector; UI mirrors the existing YouTube connector layout).
Additional Information
Tested:
collector/__tests__/utils/extensions/TwelveLabs, jest, SDK mocked): no-key and invalid-URL guards, index reuse-vs-create, Pegasus model/option on creation, custom-prompt forwarding, env-var fallback, and document persistence shape (chunkSource: twelvelabs://...). All 8 pass; full collector suite stays green (45/45).You can grab a free API key at https://twelvelabs.io — there's a generous free tier.
Developer Validations
yarn lint(prettier, the repo formatting authority) on all changed files & committed changesanythingllm-docs