Bug
Keyword indexing builds one tsvector over the whole page, and Postgres hard-caps a tsvector at 1,048,575 bytes. A single markdown file over that size fails with string is too long for tsvector — and instead of failing just that file, the whole source sync reports Sync BLOCKED and the checkpoint cannot advance until the file is fixed or skipped.
Evidence from v0.42.56.0 (macOS, Postgres 17)
Sync-failures ledger entries for a mailing-list feed source containing large exported Google-Docs markdown (1.1–7.5 MB):
"error":"string is too long for tsvector (1888898 bytes, max 1048575 bytes)"
"error":"string is too long for tsvector (3547984 bytes, max 1048575 bytes)"
Sync output:
Sync BLOCKED at 0e1313a1: 5 file(s) failed to parse.
See ~/.gbrain/sync-failures.jsonl for details, or run 'gbrain doctor'.
- Other files in the same run DID import, but the checkpoint stays pinned.
- With
--retry-failed the same files re-fail every run.
- The 3-consecutive-failure auto-skip eventually moves past them, but for a scheduled collector that means hours of blocked cycles per oversized file, per source.
Notably, embeddings are already chunk-level — only the keyword tsvector is whole-page, so this limit looks like an implementation artifact rather than inherent.
Why this matters
Sources fed by automated exporters (Drive docs, email archives, large tables) routinely produce multi-MB markdown. One such file silently degrades an entire source's freshness, and the failure mode ("Sync BLOCKED") reads like a sync-infrastructure problem rather than "one file is too big for the keyword index".
Expected fix
Any of:
- Build
search_vector per chunk (matching the embedding granularity), or
- Truncate/cap the tsvector input for oversized pages (index the first ~1MB, warn, keep full text in the page body), or
- At minimum: treat tsvector overflow as a per-file ledger entry that does NOT block the source checkpoint.
Workaround
Cap markdown exports at the collection layer (we truncate to ≤900KB with a marker before gbrain ever sees the file).
Bug
Keyword indexing builds one
tsvectorover the whole page, and Postgres hard-caps a tsvector at 1,048,575 bytes. A single markdown file over that size fails withstring is too long for tsvector— and instead of failing just that file, the whole source sync reportsSync BLOCKEDand the checkpoint cannot advance until the file is fixed or skipped.Evidence from v0.42.56.0 (macOS, Postgres 17)
Sync-failures ledger entries for a mailing-list feed source containing large exported Google-Docs markdown (1.1–7.5 MB):
Sync output:
--retry-failedthe same files re-fail every run.Notably, embeddings are already chunk-level — only the keyword tsvector is whole-page, so this limit looks like an implementation artifact rather than inherent.
Why this matters
Sources fed by automated exporters (Drive docs, email archives, large tables) routinely produce multi-MB markdown. One such file silently degrades an entire source's freshness, and the failure mode ("Sync BLOCKED") reads like a sync-infrastructure problem rather than "one file is too big for the keyword index".
Expected fix
Any of:
search_vectorper chunk (matching the embedding granularity), orWorkaround
Cap markdown exports at the collection layer (we truncate to ≤900KB with a marker before gbrain ever sees the file).