egw-scale-utils: bound the stress test by wall-clock duration - #359
Conversation
The stress client marks itself ready only once its stress loop finishes, and that loop finishes only after three cumulative dial failures accrue, with no time bound. Under a datapath that is heavily loaded but healthy those failures are rare and sporadic, so the loop can keep running for well over fifteen minutes and the client never reports ready in time for the WaitForRunningPods step on the cilium side. The coupling is backwards: a more broken datapath produces the three failures quickly and the pod turns ready fast, while a healthy one keeps the loop running until the caller times out. Add a stress-duration flag that also bounds the loop by wall-clock time. The loop still exits early on repeated failures, so that signal is kept, but a healthy run now completes deterministically once the duration elapses. Zero preserves the previous behaviour of stopping on failures only, so existing callers are unaffected. This commit was prepared with AIL:3. Signed-off-by: André Martins <andre@cilium.io>
|
Thanks André for looking into this. Taking a step back, the goal of this stress test is to validate how many connections can be established in parallel. That is currently evaluated by sequentially opening new connections; a connection establishment is considered to have failed after a five seconds timeout (which accounts for ~2 SYN retries), and the test stops after three failures. The main downside of this approach is that the test can take a significant amount of time if a lot of connections actually succeed after the first SYN retry, as that implies at least 1 second delay (which also appears to not be configurable). Looking at one of the latest runs, there have been ~600 connections that succeeded after the first SYN retry, hence accounting for more that 10 minutes of wall-clock time due to that. While adding a global timeout ensures that the test always terminates on time, I wonder whether it would be more appropriate to slightly change the test methodology, and treat every connection that takes more than one second (i.e., incurs in a SYN retry) as a failure, and maybe raise the failure threshold to say 25 or 50, to avoid that a few spurious errors can cause the test to terminate too early. That should guarantee that each test completes within 1-2 minutes, which is much better than 15. WDYT? |
2 participants
The stress client marks itself ready only once its stress loop finishes, and that loop finishes only after three cumulative dial failures accrue, with no time bound. Under a datapath that is heavily loaded but otherwise healthy those failures are rare and sporadic, so the loop can keep running for well over fifteen minutes, and the client never reports ready in time for the WaitForRunningPods step on the cilium side. The coupling is backwards: a more broken datapath produces the three failures quickly and the pod turns ready fast, while a healthy one keeps the loop running until the caller times out. This is what has been failing the Scale Test Egress Gateway workflow on the stress-base client.
This adds a stress-duration flag that also bounds the loop by wall-clock time. The loop still exits early on repeated failures, so that signal is preserved, but a healthy run now completes deterministically once the duration elapses. The connection counts and latency histograms are still exported for Prometheus to scrape after the run, so a genuine egress gateway regression remains visible. Zero preserves the previous behaviour of stopping on failures only, so existing callers are unaffected.
The consuming side in cilium/cilium will pass the new flag from the CL2 stress-test module and bump egw_utils_ref to an image built from this change. That bump has to happen before the cilium side starts passing the flag, otherwise the client rejects the unknown flag at startup.
This change was prepared with AIL:3.