Skip to content

Optimize global digest lookups - #89

Open
moacirmoda wants to merge 1 commit into
seasonedcc:mainfrom
moacirmoda:codex/optimize-global-digests
Open

Optimize global digest lookups#89
moacirmoda wants to merge 1 commit into
seasonedcc:mainfrom
moacirmoda:codex/optimize-global-digests

Conversation

@moacirmoda

Copy link
Copy Markdown

Summary

  • replace three global window-ranked revision scans in digest reads with correlated latest-row lookups
  • add a reversible index for global catch-up filtering by Discord creation time
  • preserve deterministic createdAt desc, id desc latest-event semantics with regression coverage for same-instant ties

Why

The global catch-up query ranked every message, channel, and member revision before applying its guild and time filters. As revision history grows, that makes a small incremental digest pay the cost of sorting unrelated history.

The correlated lookups constrain each latest-row search to its parent entity. The new timestamp index lets selective global catch-ups begin with the messages in range.

Query-plan evidence

Against the exact compiled global digest query on a disposable store with 100,000 synthetic messages, SQLite selected:

  • messages_discord_created_at_index (discord_created_at>?)
  • message_revisions_message_id_created_at_index (message_id=?)
  • channel_detail_revisions_channel_id_created_at_index (channel_id=?)
  • member_detail_revisions_member_id_created_at_index (member_id=?)

Removing the new index changed the outer access path from one global time-range lookup to repeated channel/time lookups through the pre-existing compound index.

Verification

  • pnpm run lint
  • pnpm run tsc
  • pnpm run test:unit twice: 589/589 passed each time
  • pnpm run test:e2e: 58/58 specs passed; 20/20 tools reached through real MCP traffic
  • pnpm run test:seed-coverage: 17/20 tools demonstrated; the existing 3 live-Discord tools remained explicitly unseedable
  • migration verified with migrate -> rollback -> migrate on a disposable store
  • focused real MCP stdio session verified messages_catch_up after the final commit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant