Embedding pipeline mechanics: broker safety, chunking, dispatch

[mbertrand]

What are the relevant tickets?

Addresses mitodl/hq#12453. Related: mitodl/hq#12015 (infra concurrency cap + grace period), mitodl/hq#12008 (Qdrant performance settings), mitodl/hq#12172 (embedding pipeline failures).

Description (What does it do?)

Phase 1 of the content-file embedding load reduction — task/broker mechanics that amplify legitimate work into storms. No change to what gets embedded.

• Broker visibility_timeout → 6h (CELERY_BROKER_TRANSPORT_OPTIONS). The default 3600s is shorter than a long summarize+embed task, so in-flight tasks were redelivered en masse during backlogs (the redelivery-storm driver).
• Split chunk sizes: QDRANT_CHUNK_SIZE 10→100 (resources, cheap per item); new QDRANT_CONTENT_FILE_CHUNK_SIZE=25 (content files do inline summarization, kept small).
• ignore_result=True on fire-and-forget embedding tasks; drop the finalize_embeddings tail and the embed_errors counter plumbing. Failure signal is now log.exception in generate_embeddings plus the weekly presence-based embeddings_healthcheck.
• Bounded backfill dispatch: start_embed_resources / embed_learning_resources_by_id go from self.replace(chain(...)) over the whole catalog (O(N²) message bytes) to a loop of apply_async batches. Per-run dispatch becomes a group of chunk signatures.
• embed_run_content_files filters to published files — stop embedding files the very next chained task deletes.
• generate_embeddings.rate_limit from a new CELERY_EMBEDDINGS_RATE_LIMIT setting (same 200/m default) so ops can tune without a deploy.
• try_with_retry_as_task dispatches via apply_async(retry=True, retry_policy=…) instead of re-.delay() on any exception (which could double-publish during a broker hiccup).
• Also batched Qdrant reads + conditional payload writes in vector_search/utils.py.

How can this be tested?

1. Config applied:

   docker compose run --rm web python manage.py shell -c \
     "from main.celery import app; from django.conf import settings; \
      print(app.conf.broker_transport_options.get('visibility_timeout'), \
            settings.QDRANT_CHUNK_SIZE, settings.QDRANT_CONTENT_FILE_CHUNK_SIZE, \
            settings.CELERY_EMBEDDINGS_RATE_LIMIT)"
   

   Expect 21600 100 25 200/m.
2. ignore_result — a dispatched embedding task stores no result key:
   docker compose exec -T redis redis-cli EXISTS celery-task-meta-<id> → 0.
3. Mixed published/unpublished ingest (hooks on): embed_run_content_files embeds only the published files, dispatched as a group of ⌈published/25⌉ chunks, points land in Qdrant with no redelivered-task warnings.
4. Test suites: docker compose run --rm web uv run pytest vector_search/tasks_test.py vector_search/utils_test.py learning_resources_search/plugins_test.py — all pass.

Additional Context

Ops note before merge: confirm the prod env doesn't pin QDRANT_CHUNK_SIZE in ol-infrastructure — its default is what changes here. This is the app-side complement to the infra-side fixes in mitodl/hq#12015.

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR adjusts the Celery/Qdrant embedding pipeline’s dispatch mechanics to reduce broker amplification during backlogs, improve batching behavior, and make embedding execution more operationally tunable—without changing which resources/content files are embedded.

Changes:

• Increased broker safety by configuring a longer broker visibility_timeout, and made generate_embeddings rate limiting configurable via settings.
• Reduced broker/result-backend load by making embedding tasks fire-and-forget (ignore_result=True) and removing the prior “finalize” tail/counter plumbing.
• Improved embedding efficiency by batching Qdrant point retrievals and skipping some redundant payload writes; updated related tests accordingly.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
main/settings.py	Updates embedding chunk-size defaults and introduces a dedicated content-file chunk size setting.
main/settings_celery.py	Adds broker visibility_timeout transport options and makes embedding task rate-limit configurable.
vector_search/tasks.py	Refactors embedding task dispatch patterns (groups vs chains), adds ignore_result on embedding tasks, filters run embedding to published files.
vector_search/utils.py	Adds batched Qdrant retrieve helper and uses retrieved points to reduce per-document Qdrant calls and some payload writes.
learning_resources_search/plugins.py	Switches to publish-time retry on task dispatch via apply_async(retry=..., retry_policy=...).
vector_search/tasks_test.py	Updates expectations for new dispatch semantics and removal of finalize/counter behavior.
vector_search/utils_test.py	Adds coverage for batched retrieval and payload-write skipping behavior.