Skip to content

feat(rag): add context overflow protection with TRUNCATE/RAISE/MAP_REDUCE strategies#143

Open
wanna-money wants to merge 1 commit into
jd-opensource:mainfrom
wanna-money:main
Open

feat(rag): add context overflow protection with TRUNCATE/RAISE/MAP_REDUCE strategies#143
wanna-money wants to merge 1 commit into
jd-opensource:mainfrom
wanna-money:main

Conversation

@wanna-money

Copy link
Copy Markdown

Summary

为 RAGAgent 新增上下文超限保护能力,支持三种溢出处理策略(TRUNCATE/RAISE/MAP_REDUCE),并引入 KnowledgeTruncator 工具类用于 token 检测、截断和分块。同时修复了 LocalAgent.deepcopy 在子类有 exclude=True 必填字段时重建失败的问题。

Related Issue(s)

N/A

Changes

  • Feature: RAG 召回内容超限检测与处理

  • 在 RAGAgent 新增 max_knowledge_tokens 和 overflow_strategy 字段,支持以下四种策略:

    • IGNORE(默认):原有行为,透传给 LLM
    • TRUNCATE:截取最大合适前缀并告警
    • RAISE:超限时直接抛出 ValueError
    • MAP_REDUCE:将内容分块,逐块调 LLM 摘要,递归折叠直到符合预算
  • Feature: KnowledgeTruncator 工具类

  • 新增 oxygent/utils/token_utils.py 中的 KnowledgeTruncator,提供:

    • fits(text, budget):token 数检测
    • truncate(text, budget):二分查找截断 + 段落边界对齐
    • split_to_chunks(text, chunk_size):按段落分块,超大段自动硬切
  • Fix: LocalAgent.deepcopy 重建子类时丢失 exclude=True 字段

  • 原先 model_dump() 会跳过 exclude=True 字段(如 func_retrieve_knowledge),导致 team_size > 1 时 deepcopy 抛 ValidationError 或缺失 _semaphore。现在统一通过 type(self).model_fields 补回被排除字段,并改用 self.class(**fields) 走完整 init 流程。

  • Tests: 单测补充

    • test_knowledge_truncator.py:覆盖 KnowledgeTruncator 的 fits/truncate/split 逻辑
    • test_rag_agent.py:覆盖四种策略的同步/异步路径、MAP_REDUCE 多轮折叠、并发执行、边界情况(空输入、retry 耗尽、chunk 失败等),共 52 个用例

Screenshots (if UI changes)

N/A

Checklist

  • been self-reviewed.
    • Code is formatted
    • Tests added/updated
    • All CI checks passed
    • Documentation updated
  • added documentation for new or modified features or behaviors.
  • added new features, such as new agents, new flows, etc.
  • added or updated version, license, or notice information.
  • added or updated demo, integration tests, unit tests, or others.
  • added or updated ui, or had changes in frontend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant