Skip to content

feat: add TwelveLabs video data connector (Pegasus)#5912

Open
mohit-twelvelabs wants to merge 2 commits into
Mintplex-Labs:masterfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat: add TwelveLabs video data connector (Pegasus)#5912
mohit-twelvelabs wants to merge 2 commits into
Mintplex-Labs:masterfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

Pull Request Type

  • ✨ feat (New feature)

Relevant Issues

No tracking issue exists yet. I'm aware CONTRIBUTING.md asks for a corresponding issue before a PR is merged — happy to open one (or have a maintainer link an existing request) if you'd like this considered. Opening the PR first so there's concrete, tested code to evaluate.

Description

Adds an opt-in video data connector powered by the TwelveLabs Pegasus model. A user pastes a public video URL; the connector uploads it to a TwelveLabs index (reused if it already exists, otherwise created with Pegasus enabled), waits for indexing, runs analyze, and saves the resulting transcript / summary / on-screen-context as a normal workspace document that can be embedded like any other source.

This fills a real gap: AnythingLLM can already ingest YouTube transcripts (caption text only), but has no way to ingest the actual content of an arbitrary video — spoken words plus visual scenes and on-screen text. Pegasus produces all of that from any direct video URL, so users can chat over webinars, recordings, demos, and footage that have no captions.

It mirrors the existing YouTube connector end-to-end so it slots into the established patterns:

  • collectorutils/extensions/TwelveLabs loader (upload → index → analyze → writeToServerDocuments, with the same path-safety/isWithin guards as the YouTube loader) + the /ext/twelvelabs-video processor route.
  • server/ext/twelvelabs/video extension endpoint, admin/manager gated, behind validatedRequest like every other connector, with a telemetry event.
  • frontend — a TwelveLabs connector option in the Data Connectors modal, the dataConnector client method, an icon, and i18n strings (English authored; seeded across all locales so translations:verify passes — happy to have these properly translated).

Non-breaking & opt-in: no defaults change, no existing connector is touched. The API key is read from the request body or the TWELVELABS_API_KEY env var; the key is never logged.

The TwelveLabs JS SDK (twelvelabs-js) is added only to the collector and lazy-required inside the handler, so it's never loaded unless this connector is actually invoked.

Visuals (if applicable)

N/A (functional connector; UI mirrors the existing YouTube connector layout).

Additional Information

Tested:

  • Offline unit tests (collector/__tests__/utils/extensions/TwelveLabs, jest, SDK mocked): no-key and invalid-URL guards, index reuse-vs-create, Pegasus model/option on creation, custom-prompt forwarding, env-var fallback, and document persistence shape (chunkSource: twelvelabs://...). All 8 pass; full collector suite stays green (45/45).
  • Live end-to-end against the real API: a public MP4 was uploaded, indexed, and analyzed, returning a correct description of the video (~80s for a short clip).

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Developer Validations

  • I ran yarn lint (prettier, the repo formatting authority) on all changed files & committed changes
  • Relevant documentation has been updated (if applicable) — no per-connector README section exists; public-API docs live in anythingllm-docs
  • I have tested my code functionality (offline unit tests + a live end-to-end run)
  • Docker build succeeds locally — not run in this environment

Adds an opt-in data connector that analyzes a video with the TwelveLabs
Pegasus model and imports the resulting transcript/summary/scene
breakdown as a workspace document.

- collector: TwelveLabs loader (upload by URL -> index with Pegasus ->
  analyze) mirroring the existing YouTube transcript connector, plus the
  /ext/twelvelabs-video processor route
- server: /ext/twelvelabs/video extension endpoint (admin/manager gated)
- frontend: TwelveLabs connector option, data connector client method,
  icon, and i18n strings (en + seeded across all locales)
- collector: offline unit tests for the loader (key/URL guards, index
  reuse-vs-create, prompt forwarding, document persistence)

API key is read from request body or the TWELVELABS_API_KEY env var; no
defaults are changed and existing connectors are untouched.
@timothycarambat timothycarambat added the Integration Request Request for support of a new LLM, Embedder, or Vector database label Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration Request Request for support of a new LLM, Embedder, or Vector database

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants