detect network issues and disconnect/reconnect as necessary

[oliver-sanders]

It has been suggested that there may be scenarios where UIS-Flow communications might need to be reset as the result of network outages.

I think ZMQ should attempt to re-establish the connection on the next publish event, so I'm not sure whether we need to disconnect/reconnect the subscriber explicitly at our layer, but am unsure over the scenario.

[oliver-sanders]

I've done some digging on the TCP/ZMQ front...

If we are experiencing broken connections after network dropouts, there's good news:

• ZMQ appears to have baked-in heartbeat (aka ping/pong) functionality - http://api.zeromq.org/4-2:zmq-setsockopt#toc38
• And auto-reconnection (with exponential backoff) - http://api.zeromq.org/4-2:zmq-setsockopt#toc17
• Which means we should be able to handle all of this in the ZMQ layer!
• There's a lengthy write up in the ZMQ guide https://zguide.zeromq.org/docs/chapter4/
• That goes into low-level implementation stuff.
• I've not yet managed to confirm/deny whether hearbeat/timeout/auto-reconnect are turned on or off for PUB/SUB by default (via pyzmq).
• But it looks like we can explicitly configure it along these lines:

  sock.setsockopt(zmq.CONNECT_TIMEOUT, 10 * 1000)
  sock.setsockopt(zmq.RECONNECT_IVL_MAX, 5000)
  sock.setsockopt(zmq.ROUTER_MANDATORY, 1)
  sock.setsockopt(zmq.ROUTER_HANDOVER, 1)

• It looks like this has to be configured from both ends which makes sense.

But I also found this, which might be pertinent:

If we use a TCP connection that stays silent for a long while, it will, in some networks, just die. Sending something (technically, a “keep-alive” more than a heartbeat), will keep the network alive.

I.E, if a workflow sits for a long time without publishing any new deltas, the network, if configured appropriately, might decide the connection is inactive and chop it. We've seen the exact same thing with websockets which lead to #586 (activate ping/pong within the websocket protocol).

They suggest turning on keep-alive, but presumably heartbeats would also solve this problem (I haven't read deeply enough yet).

@dwsutherland If this is the case, it should be super easy to test for. Try starting a scheduler in --paused mode, connect the GUI and come back to it in the morning, trigger a task and see what happens...

[dwsutherland]

I'll look into what you've posted.. soon

I've not yet heard of issues with ZMQ connections, have you got any details on this or approaches to reproduce or methods to test?

I've opened a stub issue for this problem, please add any details you can: #857

Well I'm not sure I have too much more to add, I just noticed that some of our workflows no longer update/send-deltas-to our UIS after weeks (could be less?) of inactivity... This is probably a site policy/default that's "chop"ing the connection like you said..

I don't know if it would be easy to test in a short space of time.. However, we have a socket options method that gets called in our setup, so we could just be explicit and add those options.

A ping-pong seems like a reasonable solution to this problem. Out of interest, why did you go down the route of implementing a bespoke low-level interface for this rather than using the basic GraphQL request interface used by cylc scan --ping?

I went down this route because I wanted to test both REQ/RES and PUB/SUB, and we don't currently have the ability to request a specific publish..
Also I thought it would be useful to test if the workflow is responsive (which requires more than just the ZMQ socket options). However, as you mentioned we'd need more than just /active => active to handle this because it will just loop around and keep doing this.. Perhaps we need a "naughty" bin/corner to put irresponsive workflows for 5-10min before trying again (and flash up a warning to the front end).

[hjoliver]

Perhaps we need a "naughty" bin/corner to put irresponsive workflows ...

🤣 the naughty workflows have the most fun

[dwsutherland]

sock.setsockopt(zmq.ROUTER_MANDATORY, 1)
sock.setsockopt(zmq.ROUTER_HANDOVER, 1)

These are for the ROUTER/DEALER pattern combinations (not PUB/SUB, and I couldn't see an equivalent)..

WRT the OS killing connections, these options would be relevant to override the OS default (if able):

(from the doc ref you posted)

Although it may not work on some OSs (as stated), so I think the (no updated) Ping/Pong / irresponsive handling in #851 will more than cover it.
And I will add the socket options:

sock.setsockopt(zmq.CONNECT_TIMEOUT, 10 * 1000)
sock.setsockopt(zmq.RECONNECT_IVL_MAX, 5000)

to cylc/cylc-flow#7355 where appropriate (in a couple hours)

[oliver-sanders]

Well I'm not sure I have too much more to add, I just noticed that some of our workflows no longer update/send-deltas-to our UIS after weeks (could be less?) of inactivity... This is probably a site policy/default that's "chop"ing the connection like you said..

It would be good to confirm that this is the case before we attempt to fix it any particular way, as there could be other reasons why this happened. I did briefly look into testing network interruption at the TCP level, but it seems a bit complicated and requires sudo, this might be handy for testing auto-reconnection though.

These are for the ROUTER/DEALER pattern combinations (not PUB/SUB, and I couldn't see an equivalent)..

Sorry, I think I quickly grabbed those from a router/dealer example I found on GH, I don't think the last two lines are relevant.

I haven't dug into the ZMQ stuff at depth, but there is also stuff like ZMQ_HEARTBEAT_IVL, ZMQ_HEARTBEAT_TIMEOUT, etc which will hopefully be useful.

[dwsutherland]

Yes, the HEARTBEAT types will be useful.

Have a workflow right now with timed out connection:

2026-07-06T17:17:50Z INFO - [2026-08/control_start_13516879/01:running] => succeeded
2026-07-07T05:14:56Z INFO - Command "hold" received. ID=40f689ac-e3da-4274-a95e-7e7ba09b436e
    hold(tasks=['2026-07/process_site_csv_09010876'])
2026-07-07T05:14:56Z INFO - Command "hold" actioned. ID=40f689ac-e3da-4274-a95e-7e7ba09b436e

But task not held in WUI, the publish didn't get through

In addition to the socket level PING/PONG (which I'll put in), I think the higher level one (including the irresponsive delta) is also worth having.

[oliver-sanders]

Have a workflow right now with timed out connection:

Right, before we jump to conclusions, please can you check that the TCP socket has closed (e.g, try netstat)?

If the diagnosis is correct, the TCP socket has closed (possibly due to inactivity). The subscriber has not detected this because it's just waiting for data and there is no heartbeat/timeout.

(Note, the publisher should be able to tell that the socket has closed on the next push, so if you can get it to yield a delta (e.g, removing the contact file should force a shutdown delta), you should either get an error, or a reconnection attempt from the workflow?)

If this is the case, a ZMQ-level heartbeat and reconnection will resolve this situation. To test, you could try starting two workflows in the paused state, add the relevant ZMQ flags to one, but not the other. If TCP inactivity is the cause, then both should loose connection at the same time, if the ZMQ heartbeat fixes it, only one should loose connection.

I don't think I can force a TCP socket to close on my box, but might try on a dev box so I can review the fix (if able).

In addition to the socket level PING/PONG (which I'll put in), I think the higher level #851 (comment) (including the irresponsive delta) is also worth having.

Sooo, I can't currently think of a problem that a higher level check (Cylc over ZMQ over TCP) would solve that the lower level solution wouldn't. This is basically just resetting the ZMQ client which only needs to be done if one of these fields change:

https://github.com/cylc/cylc-flow/blob/ca1b22378328882577889a1a0299c5d75f39b020/cylc/flow/network/client.py#L84-L91

Which can only happen if a workflow restarts, which is a case we are already detecting/handling.

Networking is a dark art so I'm inclined to leave it to the TCP pros (why we use ZMQ in the first place), we shouldn't have to worry about this stuff, but might need to configure things better.

[dwsutherland]

Have a workflow right now with timed out connection:

Right, before we jump to conclusions, please can you check that the TCP socket has closed (e.g, try netstat)?

A connected UI Server subscription:

[ecox_oper@ec-cylc01.slv ~]$ cat ~/cylc-run/eco_sync/run1/.service/contact
CYLC_API=5
CYLC_VERSION=8.6.3
CYLC_WORKFLOW_COMMAND=/opt/niwa/share/cylc/cylc-8.6.3/bin/python /opt/niwa/share/cylc/cylc-8.6.3/bin/cylc play --host=ec-cylc01.slv eco_sync/run1 --host=localhost --color=always
CYLC_WORKFLOW_HOST=ec-cylc01.slv.hpc.niwa.co.nz
CYLC_WORKFLOW_ID=eco_sync/run1
CYLC_WORKFLOW_OWNER=ecox_oper
CYLC_WORKFLOW_PID=161257
CYLC_WORKFLOW_PORT=43157
CYLC_WORKFLOW_PUBLISH_PORT=43195
CYLC_WORKFLOW_RUN_DIR_ON_WORKFLOW_HOST=/home/ecox_oper/cylc-run/eco_sync/run1
CYLC_WORKFLOW_UUID=eb009159-fc45-44dc-a9cf-166bddc02018
SCHEDULER_CYLC_PATH=None
SCHEDULER_SSH_COMMAND=ssh -oBatchMode=yes -oConnectTimeout=8 -oStrictHostKeyChecking=no
SCHEDULER_USE_LOGIN_SHELL=True
[ecox_oper@ec-cylc01.slv ~]$ netstat --tcp | grep 43195
tcp        0      0 ec-cylc01.slv.hpc:43195 ec-admin03.slv.hp:53368 ESTABLISHED

A broken one (tn_esp):

[ecox_oper@ec-cylc01.slv ~]$ cat ~/cylc-run/tn_esp/run1/.service/contact
CYLC_API=5
CYLC_VERSION=8.6.3
CYLC_WORKFLOW_COMMAND=/opt/niwa/share/cylc/cylc-8.6.3/bin/python /opt/niwa/share/cylc/cylc-8.6.3/bin/cylc play --pause tn_esp/run1 --host=localhost --color=never
CYLC_WORKFLOW_HOST=ec-cylc01.slv.hpc.niwa.co.nz
CYLC_WORKFLOW_ID=tn_esp/run1
CYLC_WORKFLOW_OWNER=ecox_oper
CYLC_WORKFLOW_PID=1729057
CYLC_WORKFLOW_PORT=43345
CYLC_WORKFLOW_PUBLISH_PORT=43452
CYLC_WORKFLOW_RUN_DIR_ON_WORKFLOW_HOST=/home/ecox_oper/cylc-run/tn_esp/run1
CYLC_WORKFLOW_UUID=983b77eb-8658-43e9-9ce0-2fd43f063fd7
SCHEDULER_CYLC_PATH=None
SCHEDULER_SSH_COMMAND=ssh -oBatchMode=yes -oConnectTimeout=8 -oStrictHostKeyChecking=no
SCHEDULER_USE_LOGIN_SHELL=True
[ecox_oper@ec-cylc01.slv ~]$ netstat --tcp | grep 43495
[ecox_oper@ec-cylc01.slv ~]$

If the diagnosis is correct, the TCP socket has closed (possibly due to inactivity). The subscriber has not detected this because it's just waiting for data and there is no heartbeat/timeout.

Well, it could be that the subscriber thread has thrown an exception (but unless explicitly sent to log, we won't see anything raised on a thread)... Either way, the connection needs to be re-established.

(Note, the publisher should be able to tell that the socket has closed on the next push, so if you can get it to yield a delta (e.g, removing the contact file should force a shutdown delta), you should either get an error, or a reconnection attempt from the workflow?)

With the PUB/SUB pattern, the publisher doesn't track it's subscribers.. It just publishes to any listening:
https://zguide.zeromq.org/docs/chapter5/#Pros-and-Cons-of-Pub-Sub

• 

If this is the case, a ZMQ-level heartbeat and reconnection will resolve this situation. To test, you could try starting two workflows in the paused state, add the relevant ZMQ flags to one, but not the other. If TCP inactivity is the cause, then both should loose connection at the same time, if the ZMQ heartbeat fixes it, only one should loose connection.

The reconnect might be more relevant from the subscriber end, in light of the above, but heartbeat from the publisher (?).

In addition to the socket level PING/PONG (which I'll put in), I think the higher level #851 (comment) (including the irresponsive delta) is also worth having.

Sooo, I can't currently think of a problem that a higher level check (Cylc over ZMQ over TCP) would solve that the lower level solution wouldn't. This is basically just resetting the ZMQ client which only needs to be done if one of these fields change:

https://github.com/cylc/cylc-flow/blob/ca1b22378328882577889a1a0299c5d75f39b020/cylc/flow/network/client.py#L84-L91

Which can only happen if a workflow restarts, which is a case we are already detecting/handling.

Networking is a dark art so I'm inclined to leave it to the TCP pros (why we use ZMQ in the first place), we shouldn't have to worry about this stuff, but might need to configure things better.

The higher level one can detect (and send delta) with irresponsive workflow, all the way to the main loop (both threads, not just networking thread)... Whereas the ZMQ one only looks at the socket connections.

So higher level PING/PONG has more broader detection than just the ZMQ level... And the irresponsive delta and flagging of a workflow shown here is also useful in notifying the user and avoiding a loop of reconnect attempts..

Also, as long as we can envision a potential problem outside of the ZMQ layer (i.e. workflow irresponsive for any one of no doubt many potential reasons a workflow could break/be-corrupted (hardware or software side of things)) we should be justified in including a way of testing/handling it. We don't need to wait for a real specific thing to fix/handle, just a realistic hypothetical.
This reliability section spells out the same thing, and gives a similar prescription (2nd point) to what I'm trying to achieve here.

[dwsutherland]

I haven't dug into the ZMQ stuff at depth, but there is also stuff like ZMQ_HEARTBEAT_IVL, ZMQ_HEARTBEAT_TIMEOUT, etc which will hopefully be useful.

https://zguide.zeromq.org/docs/chapter4/#Heartbeating

Heartbeating solves the problem of knowing whether a peer is alive or dead. This is not an issue specific to ZeroMQ. TCP has a long timeout (30 minutes or so), that means that it can be impossible to know whether a peer has died, been disconnected, or gone on a weekend to Prague with a case of vodka, a redhead, and a large expense account.

Haha, that last sentence 😆 ...
One-way pings/heartbeats:

For pub-sub, this does work, and it’s the only model you can use. SUB sockets cannot talk back to PUB sockets, but PUB sockets can happily send “I’m alive” messages to their subscribers.

So we only need the publisher to send a heartbeat (setting ZMQ_HEARTBEAT_IVL ) to keep the OS from killing it ..

However, I'm not sure we want the connection to timeout/close if a heartbeat doesn't arrive (setting ZMQ_HEATBEAT_TIMEOUT), because we'd need to disconnect/reconnect at a higher level anyway, and it has back-compat implications (although this could be handled either end with a version check before setting the socket option)...
So we might as well just use the higher level REQ/RES/PUB/SUB PING/PONG I've implemented (it already handles the disconnect/reconnect and irresponsive workflow), which also handles back compatibility..

Implemented publisher ping/one-way-heartbeat:

sutherlander@cortex-hyper:network$ git diff
diff --git a/cylc/flow/network/publisher.py b/cylc/flow/network/publisher.py
index 361d623df..ffbdd4f57 100644
--- a/cylc/flow/network/publisher.py
+++ b/cylc/flow/network/publisher.py
@@ -65,6 +65,8 @@ class WorkflowPublisher(ZMQSocketBase):
         # allow more time for connect/reconnect
         self.socket.setsockopt(zmq.CONNECT_TIMEOUT, 10 * 1000)
         self.socket.setsockopt(zmq.RECONNECT_IVL_MAX, 5000)
+        # send heartbeat to keep subscriber connections active
+        self.socket.setsockopt(zmq.HEARTBEAT_IVL, 20 * 1000)

in cylc/cylc-flow#7355

Tested manually.. Didn't break anything