[Bug]: rag/res/term.freq is loaded by term_weight but not shipped — every lowercase Latin token gets the same IDF

[danielgines]

Summary

rag/nlp/term_weight.py loads rag/res/term.freq to build its document-frequency table. That file does not exist in the repository and is not produced by the build, so the load always fails, self.df stays empty, and the fallback assigns the same document frequency to every lowercase Latin-script token.

The result is that term weighting cannot distinguish a stop word from a content word for English, Portuguese, Spanish, French, and any other language written in the Latin alphabet. Every startup logs Load term.freq FAIL! and the warning has been silently accepted as noise.

Observed on v0.26.4.

Code

rag/nlp/term_weight.py, in __init__:

fnm = os.path.join(get_project_base_directory(), "rag/res")
self.ne, self.df = {}, {}
try:
    with open(os.path.join(fnm, "ner.json"), "r") as f:
        self.ne = json.load(f)
except Exception:
    logging.warning("Load ner.json FAIL!")
try:
    self.df = load_dict(os.path.join(fnm, "term.freq"))
except Exception:
    logging.warning("Load term.freq FAIL!")

rag/res/ in the repository contains only ner.json, synonym.json and deepdoc/. There is no term.freq, and the Dockerfile does not download one. ner.json loads fine, so this is not a packaging or permissions problem on our side — the file simply does not exist anywhere in the project.

With self.df empty, df() in weights() degenerates:

def df(t):
    if num_space_pattern.match(t):
        return 5
    if t in self.df:          # never true
        return self.df[t] + 3
    elif letter_pattern.match(t):
        return 300            # <-- every lowercase Latin token lands here
    elif len(t) >= 4:
        ...
    return 3

and idf2, which carries 70% of the weight, becomes a constant:

idf1 = np.array([idf(freq(t), 10000000) for t in tks])
idf2 = np.array([idf(df(t), 1000000000) for t in tks])
wts = (0.3 * idf1 + 0.7 * idf2) * np.array([ner(t) * postag(t) for t in tks])

freq() has the same return 300 fallback for tokens the tokenizer's trie does not know, so idf1 collapses too for these languages. ner() and postag() both return 1 for non-CJK tokens, so nothing else differentiates them.

Reproduction

from rag.nlp.term_weight import Dealer
d = Dealer()
print(len(d.df))   # 0
for t, w in sorted(d.weights(["what was the largest supplier of hospital equipment in Salvador"],
                             preprocess=True), key=lambda x: -x[1]):
    print(f"{t:<14} {w:.4f}")

Output on v0.26.4:

Salvador       0.1565
what           0.1202
was            0.1202
the            0.1202
largest        0.1202
supplier       0.1202
hospital       0.1202
equipment      0.1202

the and was carry exactly the same weight as supplier and equipment. Only Salvador differs, because its capital letter makes it miss letter_pattern. The same happens with a Portuguese sentence: every content word ties, and only one- and two-letter tokens are demoted, by the short_letter_pattern multiplier rather than by IDF.

Impact

These weights are emitted as per-term boosts in the generated full-text query (term^weight, plus bigram phrases boosted by max(w_left, w_right) * 2), so the lexical half of hybrid retrieval ranks as if every term were equally rare. It affects dataset retrieval and memory search alike.

Suggested fix

Either ship a term.freq (tab-separated term<TAB>document_frequency, which is what load_dict expects) covering at least the common function words of the supported languages, or make the fallback language-aware instead of returning a single constant for the whole Latin script. At minimum, the warning should be an error at startup, since today it announces a silently degraded ranking that reads as normal operation.

[yzl0ng]

I'd like to work on this, but I want to confirm the expected design before adding a lexical resource.

I found that the same missing term.freq dependency and constant Latin-token fallback also exist in internal/service/nlp/term_weight.go, so a Python-only fix would leave the two retrieval implementations inconsistent.

My proposed scope is:

1. Add focused regression tests for Python and Go showing that common function words and content words must not collapse to identical lexical weights.
2. Replace the silent missing-file degradation with a deterministic fallback, keeping Python and Go behavior aligned.
3. If a lexical resource is added, document its corpus provenance and license explicitly and keep it small enough to ship.
4. Add a parity fixture so future Python/Go changes cannot drift unnoticed.

Would maintainers prefer shipping a documented frequency resource, or a small maintained multilingual fallback (for example, function-word priors) when corpus DF data is unavailable? Once the direction is confirmed, I can implement it as focused, reviewable PRs.

[yzl0ng]

Implemented this in #18470 with matching Python and Go behavior.

The fix uses a bounded OOV prior only after both the explicit dictionary and tokenizer frequency miss: alphabetic words keep the old frequency 300 through three letters, then the estimate halves every two letters with a floor of 10. It supports Latin/Greek/Cyrillic (including case and accents), while CJK and invalid mixed tokens stay on their existing tokenizer paths. This avoids shipping a corpus table before its provenance/license and language semantics are agreed.

Both implementations consume one shared parity fixture, cover the reported ordering (was < largest < supplier < equipment), and verify that a future term.freq entry still takes precedence. Python tests and the full Go NLP package pass; a five-run Go benchmark showed no runtime regression and fewer allocations than main.