Skip to content

Fix/149 structural chunker drops documents with noheader - #883

Open
flyingtony1424 wants to merge 7 commits into
ascherj:mainfrom
flyingtony1424:fix/149-structural-chunker-drops-documents-with-noheader
Open

Fix/149 structural chunker drops documents with noheader#883
flyingtony1424 wants to merge 7 commits into
ascherj:mainfrom
flyingtony1424:fix/149-structural-chunker-drops-documents-with-noheader

Conversation

@flyingtony1424

Copy link
Copy Markdown

Issue

Closes #149

Changes

  • ingestion/chunking/structural_chunker.py: _extract_sections() now appends content lines regardless of whether a heading has been seen yet, and always attempts to flush the trailing section (guarded by a non-empty-after-strip check so blank sections aren't emitted).
  • tests/unit/test_issue_149_reproduction.py: reframed from "expected to fail" reproduction tests to permanent regression tests (docstrings/comments updated; no assertions changed) now that the underlying bug is fixed.
  • JOURNAL.md: added Week 9 check-ins documenting progress.

Testing

  • Unit tests pass (make test-unit)
  • Integration tests pass (make test-integration) — not applicable, no integration surface touched
  • Linter passes (make lint) — clean on the files changed in this PR; see note below on pre-existing repo-wide failures
  • Type checker passes (make typecheck) — clean on the files changed in this PR; see note below
  • New/updated tests cover the changes

How to manually verify:

  1. Run the targeted regression tests: .venv/Scripts/python -m pytest tests/unit/test_issue_149_reproduction.py tests/unit/test_structural_chunker.py -v — all 18 should pass, including test_document_with_no_headings and the three issue-149 tests.
  2. In a Python shell, confirm a heading-less document now produces a chunk instead of []:
    from ingestion.chunking.structural_chunker import StructuralChunker
    c = StructuralChunker()
    result = c.chunk("Plain paragraph text with no markdown headings at all.", {"source": "readme"})
    print(len(result), result[0].text)  # before the fix: 0, IndexError; after: 1, the full text
  3. Confirm existing heading-based chunking is unchanged by running the full suite: .venv/Scripts/python -m pytest tests/unit -m unit -q and diffing against a run on main — the same 52 pre-existing failures should appear in both (see Notes below), with no new failures introduced by this branch.

Pre-existing failures observed (unrelated to this change):
Before making any changes, the local venv was missing several declared dev dependencies (redis, structlog, pypdf, python-jose, etc.); after running pip install -e ".[dev]", the full tests/unit suite has 52 pre-existing failures across unrelated modules (test_bias_detector.py, test_pii_scrubber.py, test_resume_parser.py, test_review_service.py, test_skill_extractor.py, test_tech_detector.py, etc.). I confirmed via git stash that these failures exist identically before and after this branch's changes — none touch structural_chunker or issue #149. Separately, ruff check has one pre-existing unused-variable warning in structural_chunker.py (current_level, present before this PR) that I left untouched as out of scope, and mypy fails repo-wide due to a numpy/Python 3.14 typestub incompatibility (Type statement is only supported in Python 3.12 and greater) unrelated to this change.

Screenshots / Demo

N/A — backend chunking logic change, no UI surface.

Notes for Reviewers

  • The main design decision (flagged as a risk in PLAN.md): heading-less sections get heading_path="" (via " > ".join([])) and heading_level=0. I grepped rag/ and api/ for heading_path consumers and found none outside the chunking module itself, so an empty string should be safe — but flagging in case there's a downstream consumer I missed.
  • Preserving preamble text means documents that previously had text before their first heading will now produce one additional chunk. This could shift chunk_index values for previously-ingested documents; may be worth a follow-up re-ingestion pass for existing indexed docs, but I didn't see an existing re-ingestion script wired to run automatically, so I left that out of scope for this fix.

flyingtony1424 and others added 7 commits July 21, 2026 22:37
… docs

Add failing reproduction tests showing that StructuralChunker.chunk()
returns an empty list for any document without markdown headings, so
heading-less READMEs are silently excluded from the RAG index via the
source_type=readme ingestion path. Also documents a related loss:
preamble text before the first heading is discarded.

Repro: .venv/Scripts/python -m pytest tests/unit/test_issue_149_reproduction.py -v
(3 failed as expected; fix planned in PLAN.md for Week 9)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StructuralChunker._extract_sections() only collected content lines
after a heading had been seen, and only saved the final section when
the heading stack was non-empty. A document with no markdown headings
therefore yielded zero sections, so chunk() returned [] and the doc
was silently excluded from the RAG index. The same logic also dropped
any preamble text before a document's first heading.

Collect content lines unconditionally and always flush the trailing
section (skipping ones that are empty/whitespace-only, so adjacent
headings with no body don't produce blank chunks). Heading-less
sections get an empty heading_path/level 0, consistent with how
downstream code already treats top-level sections.

Fixes ascherj#149
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Structural chunker silently drops documents that contain no headings

1 participant