feat(rag): add context overflow protection with TRUNCATE/RAISE/MAP_REDUCE strategies#143
Open
wanna-money wants to merge 1 commit into
Open
feat(rag): add context overflow protection with TRUNCATE/RAISE/MAP_REDUCE strategies#143wanna-money wants to merge 1 commit into
wanna-money wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
为 RAGAgent 新增上下文超限保护能力,支持三种溢出处理策略(TRUNCATE/RAISE/MAP_REDUCE),并引入 KnowledgeTruncator 工具类用于 token 检测、截断和分块。同时修复了 LocalAgent.deepcopy 在子类有 exclude=True 必填字段时重建失败的问题。
Related Issue(s)
N/A
Changes
Feature: RAG 召回内容超限检测与处理
在 RAGAgent 新增 max_knowledge_tokens 和 overflow_strategy 字段,支持以下四种策略:
Feature: KnowledgeTruncator 工具类
新增 oxygent/utils/token_utils.py 中的 KnowledgeTruncator,提供:
Fix: LocalAgent.deepcopy 重建子类时丢失 exclude=True 字段
原先 model_dump() 会跳过 exclude=True 字段(如 func_retrieve_knowledge),导致 team_size > 1 时 deepcopy 抛 ValidationError 或缺失 _semaphore。现在统一通过 type(self).model_fields 补回被排除字段,并改用 self.class(**fields) 走完整 init 流程。
Tests: 单测补充
Screenshots (if UI changes)
N/A
Checklist