[Cherry-Pick][User Experience] Improve VMM OOM diagnostics by zhangting2020 · Pull Request #79446 · PaddlePaddle/Paddle

zhangting2020 · 2026-07-09T06:30:17Z

PR Category

User Experience

PR Types

Not User Facing

Description

Improve VMM OOM diagnostics

是否引起精度变化

否

devPR:#79435

PaddlePaddle-bot

🤖 Paddle-CI-Agent | pr_review | 2026-07-09 14:39:03

📋 Review 摘要

PR 概述：改进 CUDA VMM OOM 信息，扩大 VMM VA 预留并在 OOM 中补充 pool/free 统计。
变更范围：Phi CUDA/VMM allocator、memory visitor、VMM allocator C++ 单测。
影响面 Tag：User Experience Execute Infrastructure

问题

级别	文件	概述
🟡 建议	`paddle/phi/core/memory/allocation/virtual_memory_auto_growth_best_fit_allocator.cc:310`	`native_compact` 重分配失败时 pool stats 读取的是 compact 前的 `free_blocks_`

📝 PR 规范检查

标题缺少 Tag；目标分支是 release/3.4，Description 也缺少 release cherry-pick 要求的 develop 原 PR 链接。

标题建议（可直接复制）：

[Cherry-Pick][User Experience] Improve VMM OOM diagnostics(#79435)

PR 描述建议（点击展开，可直接复制）

### PR Category
<!-- One of [ User Experience | Execute Infrastructure | Operator Mechanism | CINN | Custom Device | Performance Optimization | Distributed Strategy | Parameter Server | Communication Library | Auto Parallel | Inference | Environment Adaptation ] -->
User Experience

### PR Types
<!-- One of [ New features | Bug fixes | Improvements | Performance | BC Breaking | Deprecations | Docs | Devs | Not User Facing | Security | Others ] -->
Not User Facing

### Description
<!-- Describe what you’ve done -->
Improve VMM OOM diagnostics.

devPR:https://github.com/PaddlePaddle/Paddle/pull/79435

### 是否引起精度变化
<!-- one of the following [ 是 | 否 ]-->
否

总体评价

变更方向合理，VMM OOM 信息会更易定位物理显存耗尽与池内碎片问题。但 native_compact 后再次 OOM 的诊断路径需要修正，否则新增 pool stats 可能在该分支输出 compact 前的旧状态。

PaddlePaddle-bot · 2026-07-09T06:41:02Z

+  try {
+    allocate_result = AllocateOrCompact(size);
+  } catch (const paddle::memory::allocation::BadAlloc &e) {
+    auto stats = SumLargestFreeBlockSizes(free_blocks_.size());


🟡 建议 native_compact 分支在 compact 后的重分配失败会让这里读到 stale free_blocks_。

AllocateOrCompact() 在底层分配 OOM 后会调用 memory_compactor_->Compact(...)，而 TotalMemoryCompactor::Compact() 会用新的 all_blocks_ 替换原 list；但 free_blocks_ 只有在返回 std::nullopt 后的分支才 clear()/emplace()。如果 compact 后 underlying_allocator_->Allocate(size - free_block->size_) 再次抛 BadAlloc，会直接进入这个 catch，此时 SumLargestFreeBlockSizes(free_blocks_.size()) 仍基于 compact 前的 map 计算，OOM 提示的 total_free/max_free 会误报碎片状态。

建议在统计前先根据当前 all_blocks_ 计算/重建 free 信息，或在 compact 完成后、任何可能再次抛错的 re-alloc 之前同步刷新 free_blocks_；同时补一个 FLAGS_native_compact=true 且 re-alloc 失败的单测。

PaddlePaddle-bot · 2026-07-09T08:39:26Z

🤖 Paddle-CI-Agent | ci_status_monitor | 2026-07-10 19:13:10 UTC+08:00

CI报告基于以下代码生成（30分钟更新一次）:
PR commit: 3b085ea | Merge base: eaa369b (branch: release/3.4)

1 Required任务 : 23/27 通过

总执行（rerun次数）	总任务	✅ 通过	❌ 失败	⏳ 运行中	⏸️ 等待中	跳过
78(31)	47	42	3	0	0	2

任务	错误类型	置信度	日志
`Check PR Template / Check`	PR问题：release PR 模板检查缺少 devPR 链接	高	Job
`Check PR Template / Check`	PR问题：release PR 描述缺少 devPR 链接	高	Job

2 失败详情

🔴 Check PR Template / Check — PR问题（置信度: 高）

分析器: 通用分析(fallback)
失败用例: PR 模板检查

用例	错误摘要
`tools/CheckPRTemplate.py`	release 分支 PR 在该失败 run 中缺少 develop PR 链接，exit code 7

关键日志:

receive data : pr_num: 79446, title: [Cherry-Pick][User Experience] Improve VMM OOM diagnostics, user: zhangting2020
check_pr_template: False pr: 79446
ERROR MESSAGE: The PR link does not exist. To merge into the release branch, you need to merge into the develop branch first and then cherry-pick it to the release branch. Please merge into develop first and fill in the PR link in the Description. Use this format: devPR:https://github.com/PaddlePaddle/Paddle/pull/xxxx
EXCODE: 7
##[error]Process completed with exit code 7.

根因摘要: release PR 模板检查缺少 devPR 链接
该失败 run 执行在 release/3.4，模板检查要求 PR Description 包含 develop 分支 PR 链接，格式为 devPR:https://github.com/PaddlePaddle/Paddle/pull/xxxx。日志中 CheckPRTemplate.py 判定 check_pr_template: False 并返回 exit code 7。

状态补充: 当前 fetch_pr 已看到 PR Description 包含 devPR:https://github.com/PaddlePaddle/Paddle/pull/79435，且快速状态中存在后续 Check PR Template 成功 run；这条失败记录对应旧 run/旧描述状态。

修复建议:

保留当前 PR Description 中的 devPR:https://github.com/PaddlePaddle/Paddle/pull/79435。
如 GitHub required check 仍显示阻塞，等待状态刷新或重新触发模板检查。

关联变更: PR body/Description

🔴 Check PR Template / Check — PR问题（置信度: 高）

分析器: 通用分析(fallback)
失败用例: PR 模板检查

用例	错误摘要
`tools/CheckPRTemplate.py`	release 分支 PR 缺少 develop PR 链接

关键日志:

receive data : pr_num: 79446, title: Improve VMM OOM diagnostics, user: zhangting2020
check_pr_template: False pr: 79446
ERROR MESSAGE: The PR link does not exist. To merge into the release branch, you need to merge into the develop branch first and then cherry-pick it to the release branch. Please merge into develop first and fill in the PR link in the Description. Use this format: devPR:https://github.com/PaddlePaddle/Paddle/pull/xxxx
EXCODE: 7
##[error]Process completed with exit code 7.

根因摘要: release PR 描述缺少 devPR 链接
当前 PR 目标分支是 release/3.4，模板检查要求先合入 develop 再 cherry-pick 到 release，并在 Description 中填写 develop PR 链接。fetch_pr 返回的 PR 描述没有 devPR:https://github.com/PaddlePaddle/Paddle/pull/xxxx 字段，因此 tools/CheckPRTemplate.py 返回 exit code 7。

修复建议:

更新 PR Description，补充对应 develop 分支 PR 链接，格式为 devPR:https://github.com/PaddlePaddle/Paddle/pull/xxxx。

关联变更: PR body/Description

codecov-commenter · 2026-07-09T10:43:11Z

Codecov Report

❌ Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release/3.4@eaa369b). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...re/memory/allocation/cuda_virtual_mem_allocator.cc	25.00%	3 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##             release/3.4   #79446   +/-   ##
==============================================
  Coverage               ?   84.21%           
==============================================
  Files                  ?        3           
  Lines                  ?       19           
  Branches               ?        0           
==============================================
  Hits                   ?       16           
  Misses                 ?        3           
  Partials               ?        0

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sneaxiy

LGTM for coverage due to error messages in OOM.

paddle-bot · 2026-07-14T14:38:31Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Improve VMM OOM diagnostics

3b085ea

PaddlePaddle-bot reviewed Jul 9, 2026

View reviewed changes

zhangting2020 changed the title ~~Improve VMM OOM diagnostics~~ [Cherry-Pick][User Experience] Improve VMM OOM diagnostics Jul 10, 2026

sneaxiy approved these changes Jul 10, 2026

View reviewed changes

DanielSun11 approved these changes Jul 10, 2026

View reviewed changes

swgu98 added skip-ci: coverage skip-ci: api-benchmark labels Jul 10, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Cherry-Pick][User Experience] Improve VMM OOM diagnostics#79446

[Cherry-Pick][User Experience] Improve VMM OOM diagnostics#79446
zhangting2020 wants to merge 1 commit into
PaddlePaddle:release/3.4from
zhangting2020:release-3.4-cp

zhangting2020 commented Jul 9, 2026 •

edited

Loading

Uh oh!

PaddlePaddle-bot left a comment

Uh oh!

PaddlePaddle-bot Jul 9, 2026

Uh oh!

PaddlePaddle-bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 9, 2026 •

edited

Loading

Uh oh!

sneaxiy left a comment

Uh oh!

paddle-bot Bot commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

zhangting2020 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

是否引起精度变化

Uh oh!

PaddlePaddle-bot left a comment

Choose a reason for hiding this comment

📋 Review 摘要

问题

📝 PR 规范检查

总体评价

Uh oh!

PaddlePaddle-bot Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

PaddlePaddle-bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1 Required任务 : 23/27 通过

2 失败详情

Uh oh!

codecov-commenter commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sneaxiy left a comment

Choose a reason for hiding this comment

Uh oh!

paddle-bot Bot commented Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zhangting2020 commented Jul 9, 2026 •

edited

Loading

PaddlePaddle-bot commented Jul 9, 2026 •

edited

Loading

codecov-commenter commented Jul 9, 2026 •

edited

Loading