fix: keep watched document re-syncs from breaking pinned document de-duplication#6029
Open
ArslanRasheed60 wants to merge 1 commit into
Open
Conversation
…duplication A pinned document is filtered out of similarity search results by comparing the sourceIdentifier of the document on disk (title + published) against the identifier of every returned chunk. The re-sync job embedded the new content with the old published timestamp but then wrote a new one to disk, so after the first sync the identifiers no longer matched and a document that was both pinned and watched was injected into the context twice - once as full text and again as RAG chunks. Stamp the vectors, the source document and every other workspace referencing it with the same timestamp. resolves Mintplex-Labs#5334
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #5334
Description
Reproduction
RAG chunks, so the same content is injected into the context twice.
Cause
Pinned documents are de-duplicated out of similarity search results with
sourceIdentifier()(server/utils/chats/index.js), which keys ontitle+published. The pinned side of that comparison is read from the document JSON ondisk; the RAG side is the metadata stored on each chunk.
server/jobs/sync-watched-documents.jswrote two different values forpublishedduring asingle sync: the payload handed to
addDocumentToNamespacecarried the old timestampfrom
currentDocumentData, whileupdateSourceDocumentimmediately wrote a new one todisk. From the first re-sync onwards the two identifiers can never match, the pin filter
silently stops filtering, and the document is injected twice.
Fix
Compute the timestamp once and stamp the workspace vectors, the source document and every
other workspace that references the same file with that same value. The primary
addDocumentToNamespacecall runs withskipCache: trueand is what writes the vectorcache file, so the bloomed workspaces (which read that cache) inherit the corrected
timestamp as well.
Scope
This is limited to the double-injection / de-duplication bug. The issue also raises a
secondary feature request — that a document which is both watched and pinned should always
have its latest version read from disk at chat time — which is deliberately not
included here, as that is a product design decision for the maintainers.
Visuals (if applicable)
N/A — server-side only.
Additional Information
Added
server/__tests__/jobs/sync-watched-documents.test.js, which drives the sync job withmocked collectors/models/vector DB and asserts that the vector payload, the on-disk payload
and the bloomed workspace payload all resolve to the same
sourceIdentifier(). It fails oncurrent
master(vector payload keeps the old timestamp, disk payload gets a new one) andpasses with this change.
Developer Validations
yarn lintfrom the root of the repo & committed changes