feat(notify): generic handling of 429 in retrier

[TheMeier]

Add generic handling of HTTP responses with 429 status code in Retrier. This will allow to remove any handling of 429 in the individual notifiers and instead rely on the Retrier to handle it (TBD in follow-up PRs).

This is a re-implementation of #5088 but only the backend part. It adds a, currently unused, new receiver (CheckResponse()) to notifiy.Retrier that is meant as future replacment for the Check() receiver

Pull Request Checklist

Please check all the applicable boxes.

• Please list all open issue(s) discussed with maintainers related to this change
  • Refs: Implement proper retries for integrations #2205, slack: retry on 429 and honor Retry-After header #5048
• Is this a new Receiver integration? NO
• Is this a bugfix? NOT YET
• Is this a new feature?
  • I have added tests that test the new feature's functionality
• Does this change affect performance? NO
• Is this a breaking change? NO
• I have signed-off my commits
• I will follow best practices for contributing to this project

Which user-facing changes does this PR introduce?

NONE

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 87ce443a-3c12-486c-909d-b5d3c5d26a6a

📥 Commits

Reviewing files that changed from the base of the PR and between 5c81fb0 and cd68bb6.

📒 Files selected for processing (4)

• notify/notify_test.go
• notify/retry_stage.go
• notify/util.go
• notify/util_test.go

🚧 Files skipped from review as they are similar to previous changes (4)

• notify/util_test.go
• notify/retry_stage.go
• notify/notify_test.go
• notify/util.go

---

📝 Walkthrough

Walkthrough

Retry handling parses Retry-After from HTTP 429 responses, propagates the delay through ErrorWithReason, and schedules retries with per-attempt timers. Tests cover response classification, delay parsing, and retry timing.

Changes

Retry-After-aware retries

Layer / File(s)	Summary
Response retry contract notify/util.go, notify/util_test.go	Retrier.CheckResponse classifies responses, parses numeric or HTTP-date Retry-After values for 429 responses, and exposes the delay through ErrorWithReason.RetryAfter; tests cover response and parsing behavior.
Per-attempt retry scheduling notify/retry_stage.go, notify/notify_test.go	RetryStage.exec uses resettable timers, honors positive retry delays, adjusts fallback backoff by failed-attempt duration, and tests explicit, recalculated, and exponential retry timing.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant HTTPResponse
  participant Retrier
  participant RetryStageExec
  participant AttemptTimer
  HTTPResponse->>Retrier: provide status and Retry-After header
  Retrier-->>RetryStageExec: return retry decision and ErrorWithReason
  RetryStageExec->>AttemptTimer: schedule RetryAfter or backoff delay
  AttemptTimer-->>RetryStageExec: trigger next attempt

Loading

Possibly related PRs

• prometheus/alertmanager#5267: Relocates RetryStage, connecting to this PR’s changes to its implementation and execution logic.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: generic HTTP 429 handling in the notification retrier.
Description check	✅ Passed	The description explains the change, links related issues, confirms tests, and provides release notes, with only minor checklist omissions.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

notify/notify_test.go (1)

582-617: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a comment explaining the timing assumption, matching sibling tests.

Unlike TestRetryStageHonorsRetryAfter and TestRetryStageWithoutRetryAfterUsesExponentialBackoff, this test doesn't document why 2 attempts (and not more) fit in the 200ms window — it relies on the same coupling to backoff.NewExponentialBackOff() defaults (the second NextBackOff() call producing a delay that exceeds the remaining budget). A short comment would help future maintainers understand the timing assumption if backoff defaults change.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@notify/notify_test.go` around lines 582 - 617, Add a short comment in
TestRetryStageRecalculatesBackoffAfterRetryAfter near the 200ms context timeout
or attempt assertions, documenting that the test expects exactly two attempts
because the default exponential backoff’s second delay exceeds the remaining
timeout budget. Match the wording and placement style used by
TestRetryStageHonorsRetryAfter and
TestRetryStageWithoutRetryAfterUsesExponentialBackoff.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@notify/util.go`:
- Around line 244-264: Update parseRetryAfter to prevent overflow when
converting integer Retry-After values to time.Duration: validate or clamp secs
before multiplying by time.Second, using the maximum representable duration as
the upper bound. Preserve the existing zero result for absent, invalid, or
past-date values and keep valid in-range delays unchanged.

---

Nitpick comments:
In `@notify/notify_test.go`:
- Around line 582-617: Add a short comment in
TestRetryStageRecalculatesBackoffAfterRetryAfter near the 200ms context timeout
or attempt assertions, documenting that the test expects exactly two attempts
because the default exponential backoff’s second delay exceeds the remaining
timeout budget. Match the wording and placement style used by
TestRetryStageHonorsRetryAfter and
TestRetryStageWithoutRetryAfterUsesExponentialBackoff.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2c0f3f1a-9692-4b52-8e1f-25a18a89f8e0

📥 Commits

Reviewing files that changed from the base of the PR and between 9e50fad and 1af5b26.

📒 Files selected for processing (4)

• notify/notify_test.go
• notify/retry_stage.go
• notify/util.go
• notify/util_test.go