Skip to content

Post nightly AI failure analysis to Slack - #10795

Merged
jrhemstad merged 1 commit into
NVIDIA:mainfrom
jrhemstad:codex/nightly-ai-failure-slack
Aug 17, 2026
Merged

Post nightly AI failure analysis to Slack#10795
jrhemstad merged 1 commit into
NVIDIA:mainfrom
jrhemstad:codex/nightly-ai-failure-slack

Conversation

@jrhemstad

@jrhemstad jrhemstad commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • run the existing AI failure analysis for failed nightly workflows
  • publish a full GitHub report to the workflow summary
  • publish an abbreviated Slack summary to the CI alerts channel

Validation

@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cccl-authenticator-app cccl-authenticator-app Bot moved this from Todo to In Progress in CCCL Aug 13, 2026
@jrhemstad
jrhemstad force-pushed the codex/nightly-ai-failure-slack branch 2 times, most recently from df7afc4 to 4b06a9d Compare August 13, 2026 21:56
@jrhemstad
jrhemstad marked this pull request as ready for review August 13, 2026 23:42
@jrhemstad
jrhemstad requested a review from a team as a code owner August 13, 2026 23:42
@jrhemstad

Copy link
Copy Markdown
Collaborator Author

/ok to test 4b06a9d

@cccl-authenticator-app cccl-authenticator-app Bot moved this from In Progress to In Review in CCCL Aug 13, 2026
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c29342b6-1da5-464a-9dbb-04e07d7f6c66

📥 Commits

Reviewing files that changed from the base of the PR and between 4b06a9d and 967a7e3.

📒 Files selected for processing (1)
  • .github/workflows/ci-workflow-nightly.yml
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/ci-workflow-nightly.yml

📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Added automated CI failure analysis for nightly workflow failures.
    • Failure reports are posted to pull requests with clearer GitHub-formatted results.
    • Nightly failure summaries are shared through Slack alerts.
    • Analysis now provides separate detailed GitHub reports and concise Slack summaries.
  • Bug Fixes

    • Improved validation prevents incomplete or empty reports from being published.
    • Added safeguards for invalid workflow metadata and oversized Slack summaries.

Walkthrough

The CI failure-analysis action now uses a configured output directory, validates workflow context, generates separate GitHub and Slack reports, and exports both. Pull-request and nightly workflows consume the corresponding outputs.

Changes

CI failure analysis

Layer / File(s) Summary
Validate context and render reports
.github/actions/ci-failure-analysis/render.py
The renderer validates run identifiers, commit SHAs, repository names, and analysis references. It generates bounded GitHub and Slack outputs with sanitization and omission handling.
Wire action inputs and outputs
.github/actions/ci-failure-analysis/action.yml
The action stores analysis and workflow metadata in output_dir, renders both report formats, validates nonempty files, and exports github_report and slack_summary.
Update pull-request publishing
.github/workflows/ci-workflow-pull-request.yml
The pull-request workflow uses a temporary output directory and publishes github_report in the PR comment.
Add nightly failure publishing
.github/workflows/ci-workflow-nightly.yml
The nightly workflow analyzes eligible failures, writes the GitHub report to the workflow summary, and posts the Slack summary to the alert channel.

Suggested reviewers: bernhardmgruber

Merge Risk: 🟡 Moderate · up to 967a7

A validation failure can still exit successfully and allow stale nightly reports to be published to GitHub and Slack, potentially misleading responders. Fix or explicitly accept this bounded reporting risk before merging.


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
.github/workflows/ci-workflow-nightly.yml (1)

232-241: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

suggestion: Set an explicit timeout for publish-failure-analysis.

The job invokes a third-party Slack action but sets no timeout. GitHub applies a 360-minute default job timeout, so a stalled request can occupy a runner for hours. Add a short timeout-minutes value, or verify that the pinned action enforces a shorter timeout. (docs.github.com)

As per path instructions, this addresses workflow status/check behavior in repository automation.

Sources: Path instructions, MCP tools


ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4046357b-f66b-4e3e-a0c2-7e20d16731a2

📥 Commits

Reviewing files that changed from the base of the PR and between 5274138 and 4b06a9d.

📒 Files selected for processing (4)
  • .github/actions/ci-failure-analysis/action.yml
  • .github/actions/ci-failure-analysis/render.py
  • .github/workflows/ci-workflow-nightly.yml
  • .github/workflows/ci-workflow-pull-request.yml

Comment on lines 500 to +507
if __name__ == "__main__":
try:
main()
except ValidationError as error:
safe_error = str(error).replace("\r", "\\r").replace("\n", "\\n")
print(f"error: invalid CI triage output: {safe_error}", file=sys.stderr)
safe_error = str(error).encode("unicode_escape").decode("ascii")
print(
f"error: could not render CI triage output: {safe_error}", file=sys.stderr
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

important: Exit with failure after ValidationError.

The handler prints the validation error and then exits with status 0. If OUTPUT_DIR contains old report files, the export step can publish stale reports after rendering fails. Terminate with a nonzero status.

     except ValidationError as error:
         safe_error = str(error).encode("unicode_escape").decode("ascii")
         print(
             f"error: could not render CI triage output: {safe_error}", file=sys.stderr
         )
+        sys.exit(1)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if __name__ == "__main__":
try:
main()
except ValidationError as error:
safe_error = str(error).replace("\r", "\\r").replace("\n", "\\n")
print(f"error: invalid CI triage output: {safe_error}", file=sys.stderr)
safe_error = str(error).encode("unicode_escape").decode("ascii")
print(
f"error: could not render CI triage output: {safe_error}", file=sys.stderr
)
if __name__ == "__main__":
try:
main()
except ValidationError as error:
safe_error = str(error).encode("unicode_escape").decode("ascii")
print(
f"error: could not render CI triage output: {safe_error}", file=sys.stderr
)
sys.exit(1)

Comment thread .github/workflows/ci-workflow-nightly.yml Outdated
@github-actions

Copy link
Copy Markdown
Contributor

⏱️ CCCL compile-time benchmark comparison: Public headers compile-time bench

Result: 0 regression row(s), 2 improvement row(s) above threshold.

Run Value
Config public-headers-gcc13
Baseline origin/main
Preset all-dev
Targets cub.headers.base, thrust.cpp.cuda.headers.base, libcudacxx.test.public_headers
GPU / launch args rtx2080 / --cuda 13.3 --host gcc13

Artifacts: reports and traces

Direct file processing

-f file-processing exclusive --sort total

🟢 Direct file processing — Improvements
Rank Improvement impact Selected Δ Baseline Current Event Matched traces
1 0.651257 -0.651257 5.420129 4.768872 Processing Header File: libcudacxx/include/cuda/std/__cccl/prologue.h 552
2 0.226003 -0.226003 1.688815 1.462812 Processing Header File: libcudacxx/include/cuda/std/__cccl/epilogue.h 552

@github-actions

Copy link
Copy Markdown
Contributor

🥳 CI Workflow Results

🟩 Finished in 1h 45m: Pass: 100%/531 | Total: 5d 10h | Max: 1h 33m | Hits: 97%/639928

See results here.

@jrhemstad
jrhemstad force-pushed the codex/nightly-ai-failure-slack branch from 4b06a9d to 967a7e3 Compare August 14, 2026 19:29
@jrhemstad
jrhemstad merged commit 639b534 into NVIDIA:main Aug 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant