feat: "newest courses first" Algolia sort — config-gated replica registry + fail-safe

[macdiesel]

Summary

Backend support for a "newest courses first" sort on the enterprise Learner Portal search page, via a recency-sorted Algolia virtual replica. This PR builds and configures the replica; the waffle flag (edx-enterprise) and the MFE pointing search at it ship in separate PRs.

It also generalizes replica handling into one settings-driven registry and makes replica configuration fail-safe, so the replica feature and its safe configuration ship together.

What's here

• Recency replica — recently_released_timestamp (a course's earliest course-run start of any status — the Discovery release date, ENT-11386) added to course records; a replica whose customRanking leads with desc(recently_released_timestamp). Undated courses get 0 so they sort last under a desc ranking (deliberately not the far-future ALGOLIA_DEFAULT_TIMESTAMP, which would float them to the top).
• Unified, settings-driven replica registry — ALGOLIA['ADDITIONAL_VIRTUAL_REPLICA_INDEX_SETTINGS'] (an index_name -> Algolia index settings map in settings/base.py) is the single source of truth for all additional sort replicas, alongside the base REPLICA_INDEX_NAME "by duration" replica. _get_algolia_replica_names, _configured_replicas, configure_algolia_index, and the secured-key restrictIndices (replica_index_names) all read it. A replica is declared/configured/restricted only when its index-name key is set — inert by default (no virtual(None)), so it's safe to deploy ahead of ops enabling it. Adding a future sort is one new map entry plus the field its ranking sorts on. ALGOLIA is in DICT_UPDATE_KEYS, so the deployment YAML is merged (not replaced) — code-defined replica settings are preserved while ops sets per-environment names/credentials.
• Virtual replicas — replicas are declared virtual(name), so they mirror the primary's records (no added record count or cost). The virtual-vs-standard trade-off (relevance-biased sort under a text query) is documented in ADR 0014.
• Unified set_index_settings — folded the old primary_index boolean / separate replica method into one name-based set_index_settings(index_settings, index_name=None) routing through _get_index.
• Fail-safe configuration — each replica's set_index_settings is wrapped so an AlgoliaException is logged and skipped; one replica failing never aborts the reindex (primary + other replicas stay configured), so the degraded state is always the safe base sort. configure_algolia_index fails loudly if the primary index isn't initialized, so a misconfigured reindex can't silently no-op.
• Docs — ADR 0014 records the design (settings-driven registry, config-vs-flag layering, virtual-vs-standard trade-off, the missing-index open question + operational mitigation); docs/how_to/add_an_algolia_sort_replica.rst walks through adding a future sort end to end.
• Fixed incremental_reindex_algolia for the unified set_index_settings signature, with a TODO (ENT-11982) to extend it to the full registry and drop the two-client hack at the incremental-reindex cutover.

Test

• test_algolia_utils.py / test_algolia.py / command tests green (only the 3 pre-existing time-sensitive test_get_course_run* failures remain, which fail on master too). pycodestyle / pylint / isort clean.

Rollout (see ADR 0014)

Deploy + reindex_algolia here before ops sets the MFE env var / enables the enterprise.search_default_sort_newest waffle flag.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.96%. Comparing base (aa42e89) to head (9303a22).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #118      +/-   ##
==========================================
+ Coverage   85.74%   85.96%   +0.22%     
==========================================
  Files         109      109              
  Lines        6735     6771      +36     
  Branches      833      838       +5     
==========================================
+ Hits         5775     5821      +46     
+ Misses        830      821       -9     
+ Partials      130      129       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds backend support for a “newest courses first” Algolia sort by introducing a recency-based replica index, while also consolidating replica handling behind a config-gated registry and making replica configuration resilient to partial failures.

Changes:

• Adds recently_released_timestamp to course Algolia records and defines a recency-sorted replica (desc(recently_released_timestamp)).
• Introduces a shared, config-gated replica registry (ALGOLIA_REPLICA_CONFIG_KEYS) and updates index configuration + secured-key restrictions to loop over configured replicas only.
• Makes replica configuration fail-safe (replica failures are logged and skipped so primary configuration continues) and updates incremental reindex tooling/tests for the unified set_index_settings API.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
enterprise_catalog/settings/base.py	Documents and adds the new RECENTLY_RELEASED_REPLICA_INDEX_NAME setting key.
enterprise_catalog/apps/search/management/commands/incremental_reindex_algolia.py	Updates incremental reindex flow to configure replica settings via index_name=.
enterprise_catalog/apps/search/management/commands/tests/test_incremental_reindex_algolia.py	Adjusts tests for lazy replica initialization via name-based settings calls.
enterprise_catalog/apps/catalog/algolia_utils.py	Adds replica registry plumbing, recency replica settings, fail-safe per-replica configuration, and recently_released_timestamp computation.
enterprise_catalog/apps/catalog/tests/test_algolia_utils.py	Adds coverage for recency timestamp, configured-replica gating, and safe-fail configuration behavior.
enterprise_catalog/apps/api_client/constants.py	Defines ALGOLIA_REPLICA_CONFIG_KEYS as the shared replica registry.
enterprise_catalog/apps/api_client/algolia.py	Implements name-based set_index_settings routing and secured-key restriction over all configured replicas.
enterprise_catalog/apps/api_client/tests/test_algolia.py	Adds tests for set_index_settings(index_name=...) and restrictIndices behavior with/without the recency replica.
docs/how_to/add_an_algolia_sort_replica.rst	Adds a step-by-step guide for adding future Algolia sort replicas via the registry pattern.
docs/decisions/0014-newest-courses-sort-replica.rst	Adds ADR documenting the design, rollout strategy, and operational tradeoffs.

[pwnage101]

Logic LGTM! Just have some nits and strong desires to rename things.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[pwnage101]

Selecting virtual replicas would be an Alternative Considered, not part of the Context section. Please move this below.

Also I don't think I agree with this as a foundation for rejecting virtual replicas. Virtual replicas simply don't support fully customizable sort orders, since there's always a built-in "relevance" factor as the top priority sort order. We'd use virtual replicas to save money and record usage, but the tradeoff is less precise and non-deterministic sorting.

[pwnage101]

AI docs do this a lot where they explain low-level details, rather than what actually matters and what the end goals are. Very few people will read this in the first place, let alone understand it and have the correct takeaway, even if it is factually true.

My version is simpler:

Suggested change

	* ``ALGOLIA`` is *replaced* (not merged) from the deployment YAML, so a replica
	is only live once ops sets its index-name key in ``edx-internal`` and a
	``reindex_algolia`` run declares it on the primary.
	* The fundamental design of the enterprise-catalog logic assumes we will only provision one replica, so there is only one setting (``ALGOLIA['REPLICA_INDEX_NAME']``) and one python function to provision that one replica.

[pwnage101]

There is one missing consequence: cost. Another full replica (non-virtual) doubles our record count.

[iloveagent57]

the amount of misdirection here is tripping me up. can we make it more directly settings-driven, something like:

# settings/base.py
ALGOLIA = {
    'INDEX_NAME': '',
    # Base replica: the long-standing sort the learner-portal MFE points its video search
    # (SEARCH_INDEX_IDS.VIDEOS) at -- the MFE's ALGOLIA_REPLICA_INDEX_NAME env var. Its
    # customRanking lives in ALGOLIA_REPLICA_INDEX_SETTINGS.
    'REPLICA_INDEX_NAME': '',
    'ADDITIONAL_REPLICA_INDEX_SETTINGS': {'newest_courses_index_repl': 'ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS'}, 
    'APPLICATION_ID': '',
    'API_KEY': '',
}

this defines both the additional replica index names that are enabled and which algolia index setting to use.

[iloveagent57]

but actually, that's still a lot of indirection, which is a code smell (with the original settings/algolia_utils.py relationship, not with your code). can we just yoink the settings values from algolia_utils into settings?

ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS = {
    'customRanking': [
        'desc(recently_released_timestamp)',
        'asc(metadata_language)',
        'asc(visible_via_association)',
        'asc(created)',
        'desc(course_bayesian_average)',
        'desc(recent_enrollment_count)',
    ],
}
ALGOLIA = {
    'INDEX_NAME': '',
    # Base replica: the long-standing sort the learner-portal MFE points its video search
    # (SEARCH_INDEX_IDS.VIDEOS) at -- the MFE's ALGOLIA_REPLICA_INDEX_NAME env var. Its
    # customRanking lives in ALGOLIA_REPLICA_INDEX_SETTINGS.
    'REPLICA_INDEX_NAME': '',
    'ADDITIONAL_REPLICA_INDEX_SETTINGS': {'newest_courses_index_repl': ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS}, 
    'APPLICATION_ID': '',
    'API_KEY': '',
}

in a later PR, we could clean up the rest of the algolia settings later to follow a similar pattern. I think that will be much more straightforward everywhere.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

[macdiesel]

Fixed in 9303a22 — removed the inaccurate "ALGOLIA is replaced (not merged)" justification (it's merged via DICT_UPDATE_KEYS). The fail-loud guard stands on its own: a missing/empty primary index name would otherwise pass as a successful reindex.

[macdiesel]

Clarified in 9303a22 — the doc now states the ALGOLIA merge is shallow, so setting ADDITIONAL_VIRTUAL_REPLICA_INDEX_SETTINGS in edx-internal replaces the whole map rather than overriding individual entries.

[macdiesel]

Changed to "Accepted" in 9303a22 to match the convention in 0012/0013 (this PR implements the decision).

[pwnage101]

Nit: this "consequence" isn't a consequence if the decision is actually followed. Consider just dropping it.

[macdiesel]

Dropped in 9303a22. The missing-index open question is still recorded in the Decision section, so we don't lose the trade-off.

[pwnage101]

Future us will really appreciate current us adding "VIRTUAL" to the setting name.

Suggested change

	'ADDITIONAL_REPLICA_INDEX_SETTINGS': {
	'enterprise_catalog_recently_released_desc': ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS,
	},
	'ADDITIONAL_VIRTUAL_REPLICA_INDEX_SETTINGS': {
	'enterprise_catalog_recently_released_desc': ALGOLIA_RECENTLY_RELEASED_REPLICA_INDEX_SETTINGS,
	},

[macdiesel]

Done in 9303a22 — renamed ADDITIONAL_REPLICA_INDEX_SETTINGS → ADDITIONAL_VIRTUAL_REPLICA_INDEX_SETTINGS everywhere (settings, client, utils, tests, docs). Key stays enterprise_catalog_recently_released_desc.

[pwnage101]

Nit: all the wording on this one is just weird. If I'm reading it correctly, it's just saying that we make a best effort to configure a replica, and any errors on configuration won't crash initialization. That's an implementation detail, not a consequence. Please drop this bullet too.

[macdiesel]

Agreed — dropped in 9303a22.

[pwnage101]

This is a code smell beget by another code smell. Normal usage of an API client shouldn't require re-plumbing the client internals. This seems like a hack to workaround the original code having a hack which was to create two algolia clients (sdk_client + algolia_client) to workaround the wrapper AlgoliaSearchClient not having parameterized init_index to support passing an arbitrary index name.

This is layered, so I'd understand if you wanted to defer a fix. Eventually this management command will be used in production and will need to be refactored anyway, so at least add a comment explaining that this hack should be a temporary workaround, and also point out that sdk_client = new_search_client_or_error() above is also a hack.

[macdiesel]

Added a comment in 9303a22 calling out that both the client-internals re-plumbing and the second new_search_client_or_error() client are temporary workarounds for AlgoliaSearchClient.init_index() not taking an arbitrary index name. Filed ENT-11982 for the real fix (parameterize init_index()) at the incremental-reindex cutover: https://2u-internal.atlassian.net/browse/ENT-11982

[pwnage101]

You and/or claude has decided to just give up on extending new approach to this management command. I'd like to point out that this management command is not some legacy thing, it triggers the new and improved incremental indexing approach Alex has been working on. If you don't want to enhance it right now, that's fine, but at least put in a TODO comment with an ENT ticket pointer so we don't lose track of the tech debt.

[macdiesel]

Added a TODO (ENT-11982) in 9303a22. ENT-11982 tracks extending this command to loop the unified replica registry (it currently configures only the base replica) and removing the two-client hack: https://2u-internal.atlassian.net/browse/ENT-11982

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.