Skip to content

fix(code-review): post review comment deterministically#26

Open
Satendra-SR wants to merge 1 commit into
mainfrom
fix/deterministic-review-comment
Open

fix(code-review): post review comment deterministically#26
Satendra-SR wants to merge 1 commit into
mainfrom
fix/deterministic-review-comment

Conversation

@Satendra-SR

Copy link
Copy Markdown
Member

Problem

The automated code review reusable workflow is flaky about whether it actually posts a PR comment. On dost-ai-voicebot #21 the review ran successfully but never appeared on the PR — it only landed in the Actions step summary. The same workflow posts fine on other repos (admindashboard #1935, iages-backend #936).

Root cause, confirmed by reading the run logs on all three repos:

  • Label-triggered reviews are auto-detected by claude-code-action as agent mode (Auto-detected mode: agent).
  • In agent mode the action does not create or manage a tracking/sticky comment — use_sticky_comment is effectively a no-op. There is no action step that posts the review.
  • The only path to a PR comment was Claude itself choosing to run gh pr comment (it's in --allowed-tools, and the prompt asked for it).
Repo / PR mode model ran gh pr comment? comment posted?
admindashboard #1935 agent
iages-backend #936 agent ✅ (×2)
dost-ai-voicebot #21 agent ❌ (step summary only)

Same caller workflow, same @main reusable workflow, same action version, same claude[bot] auth on every repo. The difference was a single non-deterministic model decision per run. (Re-applying the label on #21 produced a run where the model did post — confirming it's run-to-run flakiness, not a repo misconfiguration.)

Fix

Stop leaving the comment to the model's discretion:

  1. Reframe the prompt — Claude returns the review as its final message instead of being told to post a comment itself.
  2. Drop Bash(gh pr comment:*) from the default allowed-tools, so we don't get a duplicate comment now that the workflow posts.
  3. Add a Post review comment step that reads the action's execution_file output, extracts the final result, and publishes it as one PR comment via the action's github_token output (preserves claude[bot] authorship). It honors use_sticky_comment by updating the same marked comment on re-runs.

Net effect: "the model usually posts" → "the workflow always posts."

Notes / risk

  • No change required in consuming repos — they pin @main.
  • execution_file and github_token are documented outputs of anthropics/claude-code-action@v1.0.8 (already pinned here).
  • The post step is if: always() && execution_file != '' && pull_request.number so it still runs if Claude's turn ends without an error, and no-ops cleanly when there's no result text.
  • Falls back to github.token if the App token output is empty.

🤖 Generated with Claude Code

In agent mode (the auto-detected mode for label-triggered reviews),
claude-code-action does not manage a PR comment. Whether a review
comment appeared therefore depended on Claude voluntarily running
`gh pr comment` as its final step. Most runs did; some did not — those
only wrote the review to the Actions step summary and left the PR with
no comment (observed on dost-ai-voicebot #21).

Make posting deterministic:
- Reframe the prompt so Claude returns the review as its final message
  rather than instructing it to post a comment itself.
- Drop `gh pr comment` from the model's allowed-tools (prevents a
  duplicate comment now that the workflow posts).
- Add a "Post review comment" step that reads the action's
  execution_file output, extracts the final result, and publishes it as
  one PR comment using the action's github_token (keeps claude[bot]
  authorship). Honors use_sticky_comment by updating the same comment on
  re-runs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant