Skip to content

fix: keep watched document re-syncs from breaking pinned document de-duplication#6029

Open
ArslanRasheed60 wants to merge 1 commit into
Mintplex-Labs:masterfrom
ArslanRasheed60:fix/5334-watched-document-pinned-dedup
Open

fix: keep watched document re-syncs from breaking pinned document de-duplication#6029
ArslanRasheed60 wants to merge 1 commit into
Mintplex-Labs:masterfrom
ArslanRasheed60:fix/5334-watched-document-pinned-dedup

Conversation

@ArslanRasheed60

Copy link
Copy Markdown

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #5334

Description

Reproduction

  1. Add a link document to a workspace and turn on "watch" for it.
  2. Pin the same document in that workspace.
  3. Let the watched-document sync job run once with changed source content.
  4. Chat with the workspace: the document now comes back both as pinned full text and as
    RAG chunks, so the same content is injected into the context twice.

Cause

Pinned documents are de-duplicated out of similarity search results with
sourceIdentifier() (server/utils/chats/index.js), which keys on
title + published. The pinned side of that comparison is read from the document JSON on
disk; the RAG side is the metadata stored on each chunk.

server/jobs/sync-watched-documents.js wrote two different values for published during a
single sync: the payload handed to addDocumentToNamespace carried the old timestamp
from currentDocumentData, while updateSourceDocument immediately wrote a new one to
disk. From the first re-sync onwards the two identifiers can never match, the pin filter
silently stops filtering, and the document is injected twice.

Fix

Compute the timestamp once and stamp the workspace vectors, the source document and every
other workspace that references the same file with that same value. The primary
addDocumentToNamespace call runs with skipCache: true and is what writes the vector
cache file, so the bloomed workspaces (which read that cache) inherit the corrected
timestamp as well.

Scope

This is limited to the double-injection / de-duplication bug. The issue also raises a
secondary feature request — that a document which is both watched and pinned should always
have its latest version read from disk at chat time — which is deliberately not
included here, as that is a product design decision for the maintainers.

Visuals (if applicable)

N/A — server-side only.

Additional Information

Added server/__tests__/jobs/sync-watched-documents.test.js, which drives the sync job with
mocked collectors/models/vector DB and asserts that the vector payload, the on-disk payload
and the bloomed workspace payload all resolve to the same sourceIdentifier(). It fails on
current master (vector payload keeps the old timestamp, disk payload gets a new one) and
passes with this change.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

…duplication

A pinned document is filtered out of similarity search results by comparing the
sourceIdentifier of the document on disk (title + published) against the identifier of
every returned chunk. The re-sync job embedded the new content with the old published
timestamp but then wrote a new one to disk, so after the first sync the identifiers no
longer matched and a document that was both pinned and watched was injected into the
context twice - once as full text and again as RAG chunks.

Stamp the vectors, the source document and every other workspace referencing it with the
same timestamp.

resolves Mintplex-Labs#5334
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Pinned and Watched Documents without RAG

1 participant