Skip to content

in_kubernetes_events: bound watch connection lifetime#12105

Open
edsiper wants to merge 2 commits into
masterfrom
kubernetes-events-hang-12019
Open

in_kubernetes_events: bound watch connection lifetime#12105
edsiper wants to merge 2 commits into
masterfrom
kubernetes-events-hang-12019

Conversation

@edsiper

@edsiper edsiper commented Jul 15, 2026

Copy link
Copy Markdown
Member

Problem

The Kubernetes events watch request can remain silently half-open forever when the API server or an intermediary stops sending data without closing the connection. The plugin accepts the incomplete HTTP response as an active streaming client, while the default network I/O timeout is disabled, so the collector never reconnects.

Fixes #12019.

Changes

  • Add kube_watch_timeout, defaulting to 10m; setting it to 0 disables the timeout.
  • Add the corresponding Kubernetes timeoutSeconds query parameter to watch requests.
  • Apply the same value as the HTTP client's read-idle timeout so Fluent Bit can recover if the server-side timeout is not honored.
  • Add an integration regression test that stalls a watch connection and verifies that the plugin reconnects and relists.

The default bounds recovery time while avoiding frequent API relists. Existing behavior can be retained with kube_watch_timeout 0.

Validation

All checks passed:

ctest --test-dir build -R '^flb-rt-in_kubernetes_events$' --output-on-failure
tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/in_kubernetes_events/tests/test_in_kubernetes_events_001.py -q
VALGRIND=1 VALGRIND_STRICT=1 tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/in_kubernetes_events/tests/test_in_kubernetes_events_001.py -q
GITHUB_EVENT_NAME=pull_request GITHUB_BASE_REF=master tests/integration/.venv/bin/python .github/scripts/commit_prefix_check.py

The strict Valgrind run reported zero leaks and zero errors.

Summary by CodeRabbit

  • New Features

    • Added a configurable Kubernetes watch timeout, defaulting to 10 minutes.
    • Applied the timeout to watch requests and inactive streaming connections.
    • Set the timeout to 0 to disable this behavior.
  • Bug Fixes

    • Improved recovery when Kubernetes event watch connections become stalled or expire.
  • Tests

    • Added coverage verifying watch reconnection and timeout parameters.

edsiper added 2 commits July 15, 2026 10:26
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
Signed-off-by: Eduardo Silva <eduardo@chronosphere.io>
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a8ec6c04-ea87-4913-a2e3-326c2d5d1a64

📥 Commits

Reviewing files that changed from the base of the PR and between 0c61016 and 602e794.

📒 Files selected for processing (3)
  • plugins/in_kubernetes_events/kubernetes_events.c
  • plugins/in_kubernetes_events/kubernetes_events.h
  • tests/integration/scenarios/in_kubernetes_events/tests/test_in_kubernetes_events_001.py

📝 Walkthrough

Walkthrough

The Kubernetes events input adds a configurable watch timeout, applies it to Kubernetes watch requests and streaming HTTP read idle handling, and adds an integration test covering stalled-watch reconnection and timeout URL parameters.

Changes

Kubernetes watch timeout handling

Layer / File(s) Summary
Watch timeout configuration and runtime application
plugins/in_kubernetes_events/kubernetes_events.h, plugins/in_kubernetes_events/kubernetes_events.c
Adds a 10-minute default, stores watch_timeout, appends timeoutSeconds to watch requests, and configures the streaming client read idle timeout.
Stalled-watch reconnection coverage
tests/integration/scenarios/in_kubernetes_events/tests/test_in_kubernetes_events_001.py
Adds a threaded mock Kubernetes API and verifies repeated watch requests, list requests, and timeoutSeconds=1 URL parameters.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: cosmo0920

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: bounding the Kubernetes events watch lifetime.
Linked Issues check ✅ Passed The change addresses #12019 by timing out stalled watches and reconnecting so event collection continues instead of hanging.
Out of Scope Changes check ✅ Passed The added timeout/config and regression test are directly related to the watch-hang fix and stay within scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch kubernetes-events-hang-12019

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 602e7948c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


flb_sds_printf(&url, "?watch=1&resourceVersion=%" PRIu64, max_resource_version);
if (ctx->watch_timeout > 0) {
flb_sds_printf(&url, "&timeoutSeconds=%d", ctx->watch_timeout);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Advance resource versions before timing out watches

With the new default positive watch timeout, every watch request asks the API server to close after 600s, which makes the next collector cycle perform a full list again. In DB-less configurations, watched events are not used to advance ctx->last_resource_version (it is only updated from list metadata in check_and_init_stream()), so any events delivered by the watch and still retained by Kubernetes have resource versions greater than the stale filter and are emitted a second time on each forced relist. Please update the tracked resource version from watched events or avoid enabling periodic relists by default.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

indefinite hang on processing kubernetes events

1 participant