fix: keep document listings when pruning link farms

[ebarkhordar]

On the page in #900, extract() returns only the first content block: 873 characters against 1633 on 2.1.0.

The cause is the link farm branch added in #887 (link_density_test, htmlprocessing.py). It treats a large, link-dense element as boilerplate unless the average link text reaches 100 characters. The reporter's page is a listing of PDF manuals with one short title per link, so it never clears that exemption: the container holds 63 links and 1715 of its 1844 characters are link text, averaging 27.2 characters. prune_unwanted_sections calls delete_by_link_density(tree, "div", backtracking=True), which walks every div in document order, so the outermost match is dropped with its whole subtree and the rest of the article goes with it.

Average link length cannot separate the two cases on its own, because a nav sidebar and a document listing both use short link text. The markup differs instead: a farm runs its links together inside the container, while a listing gives each link its own paragraph. is_paragraph_listing tests for that shape and exempts the element when more than half of its links stand alone in a paragraph. Links that share a paragraph do not count, so wrapping a farm in a p element buys no exemption.

Verification:

• test_link_density_paragraph_listing_kept fails on master and passes on this branch; test_link_density_links_sharing_a_paragraph_pruned pins the other side of the exemption and passes on both.
• The reporter's page goes from 873 to 1631 characters, matching 2.1.0 output. Both link farm tests from fix: more targeted boilerplate pruning #887 still pass.
• unit_tests.py, realworld_tests.py, filters_tests.py, metadata_tests.py, baseline_tests.py and xml_tei_tests.py: 234 passed, 3 skipped. ruff check . and ruff format --check --diff trafilatura tests are clean.
• Precision and recall over evaldata.json for the 100 pages present in tests/cache are identical before and after (P 0.908, R 0.918, F1 0.913), so the pruning trade-off does not move outside the reported case. I did not run mypy or the docs tests against the full extras, so those are unverified here.

Fixes #900

[ebarkhordar]

The one red check here, build (ubuntu-latest, 3.13, false), fails in its Type checking step rather than in the tests:

trafilatura/utils.py:56: error: Incompatible types in assignment (expression has type "None", variable has type "Callable[[bytes], DetectionResult]")  [assignment]

This branch does not touch utils.py. It changes trafilatura/htmlprocessing.py and tests/unit_tests.py only.

It is not coming from this PR. On master at c1bc953 with no patch applied, in a clean python:3.13-slim container with .[dev] plus the [all] dependencies, mypy -p trafilatura gives the same single error and the same Found 1 error in 1 file (checked 22 source files). I also ran mypy 2.1.0, 2.2.0 and 2.3.0 against that same tree, and all three report it, so the mypy version is not the trigger.

The dependency that changed is faust-cchardet. Holding everything else fixed on master:

• faust-cchardet 3.1.0 (released 2026-08-04): no utils.py:56 error
• faust-cchardet 3.2.0 (released 2026-08-10 02:54 UTC): the error appears

3.2.0 ships py.typed and _cchardet.pyi, so detect now carries a real signature and the cchardet_detect = None fallback in the except ImportError branch becomes an incompatible assignment. The last Tests run on master was 2026-07-31, before that release, which is why master still looks green. Any PR whose type-check job resolves 3.2.0 will hit the same error.

I have not written a fix, since it sits outside this PR. Happy to send one separately if that is useful.