PartitionEOF: carry topic + offset so consumers can detect full catch-up#784
Open
florin-akermann wants to merge 1 commit into
Open
PartitionEOF: carry topic + offset so consumers can detect full catch-up#784florin-akermann wants to merge 1 commit into
florin-akermann wants to merge 1 commit into
Conversation
Author
|
Hi @fede1024 any chance we can get this merged? |
florin-akermann
force-pushed
the
master
branch
10 times, most recently
from
2312148 to
e37b062
Compare
Previously `KafkaError::PartitionEOF` carried only the partition as an `i32`. For a consumer subscribed to more than one topic, that partition number is ambiguous. From the EOF signal alone there is no way to tell which topic-partition reached the end. This change replaces the payload with a `TopicPartitionOffset` struct (topic, partition, offset). Callers now get the full coordinates on every EOF signal. This makes it possible to track, from the EOF stream alone, whether the consumer reached the end for all of its topic-partitions. At EOF the offset is the log-end position, so the catch-up point comes for free. A concrete use case is an app that builds and serves state derived from Kafka topics. It can now detect when it has fully caught up: EOF observed on every relevant topic-partition. Only then does it mark itself ready. This enables smoother blue/green deployments. A freshly started instance warms up its state first. It begins serving traffic only once it is up to speed. This catch-up detection is also achievable today without this change. You can compare the consumed position against the high watermark per partition (e.g. via `fetch_watermarks`). The value here is ergonomics. The EOF approach is event-driven. It needs no separate watermark polling loop. It avoids the race between reading the watermark and consuming up to it. EOF is a point-in-time signal. It suits an initial warm-up gate rather than a continuous liveness check.
florin-akermann
changed the title
Author
|
Gave this a rebase and a cleanup. It's green locally on the pinned 1.85 toolchain — fmt, clippy Would appreciate a look when you get a chance. It's a breaking change to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now
KafkaError::PartitionEOFonly hands you the partition number as ani32. Once a consumer is subscribed to more than one topic that's ambiguous — you get an EOF for partition 3, but nothing tells you which topic it belongs to or the offset it stopped at. This PR changes the variant to carry a smallTopicPartitionOffsetstruct (topic, partition, offset).Motivation: with the full coordinates you can track whether you've reached EOF on every partition you care about. With just the partition number you can't, since there's no way to tie an EOF back to a topic.
A concrete example is an app that builds state off a set of Kafka topics. It can watch for EOF across all its partitions and only flip to "ready" once it's caught up — which is handy for blue/green rollouts, where a new instance warms its state before it starts taking traffic. As a bonus the offset at EOF is the log-end position, so you also learn where you caught up to.
None of this is strictly new: you can already get the same signal by comparing your position to the high watermark via
fetch_watermarks. This just makes the event-driven version pleasant — no polling loop, no watermark-vs-consume race. EOF is a one-shot signal, so it's better suited to an initial warm-up gate than a continuous health check.It's a breaking change to
PartitionEOF. If you'd rather keep it additive I'm happy to rework it.