Include active broken-connection/irresponsive-workflow checks

[dwsutherland]

closes #857

There were two problems:

• DB restarts don't change the CYLC_WORKFLOW_UUID, and this was the criteria for reestablishing the UIS workflow.
• old contact files of inactive workflows left an old subscription/workflow in the UIS.

These are addressed here by using PID as an alternate criteria, and testing the PUB tcp listener of workflows (which will help clear old contact files and/or reestablish subscriptions)

Update:

I have an idea for another check, however, it would need some back compat added.. It would essentially check both the REQ/RES PUB/SUB are still connected/working.

I would create a new endpoint on the cylc-flow side that would take a REQ, and pass it back via the PUB. We would then pick that up via the existing UIS SUB, and record the timestamp. This would happen periodically (say every ~5min?), and any workflow with a last recorded time older than some threshold (say 11min) would be disconnected/unregistered/purged, and re-registered on the next scan (if available).

Why? Because it's good to check the existing connections (not just create new ones), and also another problem I've seen:

• Some workflows that are inactive for large periods of time (i.e. ones that run every month) stop receiving updates, as if the connections have timed out or failed in periods of inactivity..

Although I could investigate whether ZeroMQ has some kind of default inactivity timeout on connections, I think this will act as a check for any other connection disruptions.

I will crack on with it.

Tasks:

• Add criteria checks (PID, HOST) in addition to UUID.
• Check both the REQ/RES PUB/SUB are still connected/working.

Sibling PR cylc/cylc-flow#7355

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[dwsutherland]

Will sort tests in morning .. I could just change socket testing from testing PUB to REQ/RES server

[dwsutherland]

Pushed a related fix .. The workflow manager wasn't disconnecting the workflow and cleaning it up properly when the workflow stopped, only the data-store manager was disconnecting properly.

This effected a fix I had in..

[dwsutherland]

I've amended the description, and added tasks (see above). These are reduced to:

• Add criteria checks (PID, HOST) in addition to UUID.
• Check both the REQ/RES PUB/SUB are still connected/working.

And I've removed the socket testing (as that's covered by the REQ/RES/PUB/SUB test) and the direct contact file deletion..
This PR now has a cylc-flow sibling: cylc/cylc-flow#7355

I've testing the ping pong:

But am yet to add actual integration tests.. Will work on that this afternoon.

[dwsutherland]

This is to ensure that the workflow manager actually calls it's _disconnect method, and cleans up the workflow record on it's side.. (the _disconnect wasn't been called at all, so the self.workflows[w_id].get('req_client') was not being set to None)

[oliver-sanders]

I don't understand this change, stopped workflows should already be getting detected.

Could you give me an example where this is a problem to help me understand and review?

[dwsutherland]

_disconnect_workflow can be called from multiple places, so only wanted @log_call to activate once (when the workflow manager cleans up it's records and calls it's own _disconnect)

[oliver-sanders]

Hi, I raised the question of proactive crashed workflow detection at our end...

• I don't think we can make use of the detect_old_contact_file check at the UIS end, or at least can't right now without a rethink.
• We have some half-baked plans to change approach:
  • Mark crashed workflows: contact: detect crashed workflows cylc-flow#4858
  • Workflow monitoring service: service for workflow lifecycle management #781
  • Which put together would handle this situation so the UIS doesn't have to.
  • These are chalked up for one of the prioritisation discussions that we've not got around to 😢
• To proactively detect crashed workflows in the UIS (right now):
  • We could consider a ping approach (you beat me to it!).
  • This would be unable to differentiate between workflow crashed and workflow slow to respond / hanging.
  • So we can't declare the workflow stopped and remove the contact file.
  • But we can log an error event for the workflow (which would appear in the GUI) communicating this.
  • The message could potentially tell the user to run cylc scan --ping to poke at the workflows.
  • With more work, we could potentially give the user a button to request the UIS to do this for them (which should be fine as it wouldn't be required for functionality).

This is a very different issue to #845, and would require a cylc-flow interface, so we should split the #845 bugfix (target 1.9.x) from the proactive crash detection (target 1.10.0).

[oliver-sanders]

@dwsutherland

I have extracted the part of this PR which addresses #845 into PR #856 so we can get it into 1.9.3 quickly - will try to get that released today as there are Cylc Review fixes we need.

I've chalked up a new issue to cover the proactive detection of crashed workflows that the remainder of the PR addresses - #856

There are some vague ideas around this, but we haven't managed to give it much time yet. One idea outlined was to get cylc-flow to touch it's own contact file with each main-loop iteration. This would allow us to tell how long it has been since the main-loop turned over which would separate network issues from scheduler issues. It still wouldn't be enough for us to tell if the scheduler has crashed (it could just be hanging for some perverse reason), but it would be enough for us to log an error for the user to pick up.

The UIS might want to do something more networky if there are scenarios where disconnecting/reconnecting to the same scheduler instance might be advantageous. E.g, if there is some unrecoverable ZMQ or networking issue and the connection needs to be reset?

[dwsutherland]

I don't think we can make use of the detect_old_contact_file check at the UIS end, or at least can't right now without a rethink.

I already removed this part from this PR (and the socket test).

This would be unable to differentiate between workflow crashed and workflow slow to respond / hanging.

The UIS might want to do something more networky if there are scenarios where disconnecting/reconnecting to the same scheduler instance might be advantageous. E.g, if there is some unrecoverable ZMQ or networking issue and the connection needs to be reset?

This is the primary purpose of the ping/pong, to detect and refresh/reconnect after possible networking/ZMQ issues.. Not to detect crashed workflows, perhaps that can be another effort.

If the workflow doesn't respond after two ping/pong attempts, then it is reset.. (set to ~3min30)...

I have extracted the part of this PR which addresses #845 into PR #856 so we can get it into 1.9.3 quickly - will try to get that released today as there are Cylc Review fixes we need.

Given my above responses, can we not just get this in now.
The ping/pong is very simple (just send and test that the workflow RES and PUB), back-compat is already accounted for too.

This is a very different issue to #845, and would require a cylc-flow interface

Also, the ping/pong alone would address #845 , because the restarted workflow means the existing-UIS/old connections don't work anymore... So it would reestablish the workflow at the UIS.

Another thing, it's important for a (above mentioned) problem found at ESNZ, so also want this in ASAP.

[oliver-sanders]

This ping solution requires a cylc-flow release so we can't get it in and released today.

I think there are some crossed wires here. The issue in #845 is about the contact file changing between scans, the ping-pong mechanism cannot detect this (it's checking connection timeout not scanning the contact file) but your other change to the workflows mgr does. So I've cherry-picked this change for quick merge and release.

Another thing, it's important for a (above mentioned) problem found at ESNZ, so also want this in ASAP.

I don't think this issue requires anything more than #856 to fix, here's my test to reproduce:

• Start GUI.
• Start workflow.
• Refresh UIS scan -> workflow appears in GUI sidebar
• kill -9 the workflow.
• cylc scan --ping
• Refresh UIS scan -> workflow changes to stopped, data store disconnects

Example GUI log:

[I 2026-06-30 09:52:51.686 CylcUIServer] [data-store] register_workflow('~oliver.sanders/foo/run1', True)
[I 2026-06-30 09:52:51.703 CylcUIServer] [data-store] connect_workflow('~oliver.sanders/foo/run1', <dict>)
[D 2026-06-30 09:53:12.296 ServerApp] 200 POST /cylc/graphql (ba9b516b6cb94071b85cd08789d4fd26@127.0.0.1) 3.43ms
[D 2026-06-30 09:53:28.411 ServerApp] 200 POST /cylc/graphql (ba9b516b6cb94071b85cd08789d4fd26@127.0.0.1) 5.77ms
[I 2026-06-30 09:53:28.417 CylcUIServer] [data-store] disconnect_workflow('~oliver.sanders/foo/run1')

The detects the removal of the contact file and disconnects the workflow as expected.

We can prove that it's working correctly by restarting the workflow and requesting a UIS scan - the UIS will then successfully reconnect.

(note this is a subtly different scenario to #845 as the workflow is detected as stopped in one scan and restarted in another)

This is the primary purpose of the ping/pong, to detect and refresh/reconnect after possible networking/ZMQ issues.. Not to detect crashed workflows, perhaps that can be another effort.

I've not yet heard of issues with ZMQ connections, have you got any details on this or approaches to reproduce or methods to test?

I've opened a stub issue for this problem, please add any details you can: #857

A ping-pong seems like a reasonable solution to this problem. Out of interest, why did you go down the route of implementing a bespoke low-level interface for this rather than using the basic GraphQL request interface used by cylc scan --ping?

[oliver-sanders]

The /active state transition will cause:

1. disconnect
2. unregister
3. register
4. connect

We want the disconnect/reconnect, but we don't want the unregister/register (this yields pruned/added deltas). The workflow hasn't been deleted (unregistered), we're just re-attempting the connection (which may well fail).

Note that if the connection does fail, it will activate the detect_old_contact_file logic via the timeout_handler, so this is equivalent to automatically running detect_old_contact_file in a loop 🤦.

[dwsutherland]

Rebased on top of #856, will add a way of addressing the ~"irresponsive workflow connection reset loop" soon, and possibly change/include to account for the ZMQ options in #857

[dwsutherland]

Making some progress...

Have added handling for irresponsive workflows, it will now mark the workflow as such then send and apply a status update delta, which will be undone by either a reconnect or on receiving a "pong" (which is now the status of the workflow)..

This is useful if the problem is beyond ZeroMQ socket options/handling, perhaps a network outage and/or irresponsive workflow for any other reason...

Which can be tested/seen/reproduced with:

(uis) sutherlander@cortex-hyper:cylc-uiserver$ git diff
diff --git a/cylc/uiserver/app.py b/cylc/uiserver/app.py
index 9abb52a..705ad25 100644
--- a/cylc/uiserver/app.py
+++ b/cylc/uiserver/app.py
@@ -348,7 +348,7 @@ class CylcUIServer(ExtensionApp):
             This involves sending workflows a REQ/RES request which prompts
             a PUB response captured by our SUB.
         ''',
-        default_value=180.0
+        default_value=15.0
     )
     max_workers = Int(
         config=True,
diff --git a/cylc/uiserver/data_store_mgr.py b/cylc/uiserver/data_store_mgr.py
index cddd712..0117a7f 100644
--- a/cylc/uiserver/data_store_mgr.py
+++ b/cylc/uiserver/data_store_mgr.py
@@ -36,6 +36,7 @@ from concurrent.futures import ThreadPoolExecutor
 from copy import deepcopy
 import json
 from pathlib import Path
+import random
 import time
 from typing import TYPE_CHECKING, Dict, Optional, Set, cast
 
@@ -300,9 +301,13 @@ class DataStoreMgr:
             topic == 'ping'
             and w_id in self.workflows_mgr.workflows
         ):
+            if random.randint(1,10):
+                return
+            print('PONG')
             self.workflows_mgr.workflows[w_id]['pubsub_time'] = time.time()
             # workflow is now clearly responsive
             if w_id in self.workflows_mgr.irresponsive:
+                print('RESPONSIVE')
                 # Bring status up to date
                 status_data = json.loads(delta.decode('utf-8'))
                 self.create_status_delta(
diff --git a/cylc/uiserver/workflows_mgr.py b/cylc/uiserver/workflows_mgr.py
index ab04254..ba76b7f 100644
--- a/cylc/uiserver/workflows_mgr.py
+++ b/cylc/uiserver/workflows_mgr.py
@@ -160,7 +160,7 @@ class WorkflowsManager:  # noqa: SIM119
 
         # Connection checker threshold, twice the check interval and buffer
         # to allow for two check attempts.
-        self.conn_threshold = uiserver.connections_check_interval * 2 + 30
+        self.conn_threshold = uiserver.connections_check_interval * 2 + 5
         # Naughty bin of irresponsive workflows
         self.irresponsive = set()
 
@@ -288,12 +288,13 @@ class WorkflowsManager:  # noqa: SIM119
                     status='irresponsive',
                     status_msg='Workflow is not responding to the UI Server.',
                 )
+                print('IRRESPONSIVE')
                 # using `~` to indicate disconnect/reconnect only
                 # if successful, the DSMgr will overwrite the above delta.
-                if wid in active_before:
-                    yield (wid, '~active', 'active', flow)
-                else:
-                    yield (wid, '~inactive', 'active', flow)
+                #if wid in active_before:
+                #    yield (wid, '~active', 'active', flow)
+                #else:
+                #    yield (wid, '~inactive', 'active', flow)
 
             else:
                 # this flow is running

(ignore the websocket closed warnings, I just closed a tab while a delta was being sent or something)

Will tidy up a bit more next week (and add socket options ZeroMQ side of things)