in_kubernetes_events: bound watch connection lifetime#12105
Conversation
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe Kubernetes events input adds a configurable watch timeout, applies it to Kubernetes watch requests and streaming HTTP read idle handling, and adds an integration test covering stalled-watch reconnection and timeout URL parameters. ChangesKubernetes watch timeout handling
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 602e7948c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| flb_sds_printf(&url, "?watch=1&resourceVersion=%" PRIu64, max_resource_version); | ||
| if (ctx->watch_timeout > 0) { | ||
| flb_sds_printf(&url, "&timeoutSeconds=%d", ctx->watch_timeout); |
There was a problem hiding this comment.
Advance resource versions before timing out watches
With the new default positive watch timeout, every watch request asks the API server to close after 600s, which makes the next collector cycle perform a full list again. In DB-less configurations, watched events are not used to advance ctx->last_resource_version (it is only updated from list metadata in check_and_init_stream()), so any events delivered by the watch and still retained by Kubernetes have resource versions greater than the stale filter and are emitted a second time on each forced relist. Please update the tracked resource version from watched events or avoid enabling periodic relists by default.
Useful? React with 👍 / 👎.
Problem
The Kubernetes events watch request can remain silently half-open forever when the API server or an intermediary stops sending data without closing the connection. The plugin accepts the incomplete HTTP response as an active streaming client, while the default network I/O timeout is disabled, so the collector never reconnects.
Fixes #12019.
Changes
kube_watch_timeout, defaulting to10m; setting it to0disables the timeout.timeoutSecondsquery parameter to watch requests.The default bounds recovery time while avoiding frequent API relists. Existing behavior can be retained with
kube_watch_timeout 0.Validation
All checks passed:
The strict Valgrind run reported zero leaks and zero errors.
Summary by CodeRabbit
New Features
0to disable this behavior.Bug Fixes
Tests