(strong contender for the worst Cylc terminology to date, apologies)
When tasks are removed from the graph by reload or restart, any instances of those tasks in the active window are "orphaned" from their families.
The current strategy (inherited from Cylc 7) is to have the root family "adopt" orphans (although this appears only to happen for reload, not restart, and is only reflected in one of several WorkflowConfig interfaces leading to internal inconsistencies).
These orphans then persist until they leave the pool naturally, or are manually cleaned up by the user, after which, the workflow runs cleanly according to the new config.
However, these orphans can sometimes trip users up as an unexpected consequence of performing a graph change. They may lead to workflow stall and have caused a fair few bugs, most recently #7196 & #7197.
The current arrangement may be considered a hangover of SoS logic where orphans made more sense. In the SoD world, we could simplify the situation by removing orphans rather than persisting them. The user has said they no longer want these tasks, perhaps the simplest solution is just to get rid of them.
Orphan Tasks Example
For example, this workflow has two chains of tasks a => b => c and d => e => f. The second chain may be toggled on/off.
The workflow will toggle the chain off during the runtime of 1/e:
#!Jinja2
{% set whatevs = whatevs | default(True) %}
[scheduler]
allow implicit tasks = True
[scheduling]
cycling mode = integer
initial cycle point = 1
runahead limit = P2
[[graph]]
P1 = """
b[-P1] => b
a => b => c
{% if whatevs %}
e[-P1] => e
d => e => f
b => e
{% endif %}
"""
[runtime]
[[root]]
script = sleep 2
[[e]]
script = """
if [[ $CYLC_TASK_CYCLE_POINT -eq 1 ]]; then
cylc stop --now --now --max-polls=5 --interval=1 "$CYLC_WORKFLOW_ID"
cylc play -s 'whatevs=False' "$CYLC_WORKFLOW_ID"
fi
"""The result of this (post 1/e completion) is three orphan tasks, all in the waiting states:
The Problem
Predictability:
The orphan tasks you get is the result of the graph state at the time the reload was performed, it's not predictable what the result of the graph change will be in this regard. As a result, it is tricky for the user to postulate over what orphan tasks might be left, might consequently run, or require removal.
Downstream Issues:
Orphan tasks may lead to workflow stall as they may be left with partially-satisfied prerequisites which can never be satisfied as the result of the graph change.
Bugs:
Orphan tasks are an edge case, they have no corresponding entry in WorkflowConfig.taskdefs, they aren't included in WorkflowConfig.runtime['descendants'], their graph edges are severed, their inheritance is disrupted. All of these internal inconsistencies may lead to bugs.
E.g ATM:
- Orphans not visible in the GUI (because they no longer exist in the config).
- Orphans can't be operated on via cycle point or glob (because the no longer belong to the
root family, except in one WorkflowConfig interface).
- Orphans can't be operated on even when an explicit task ID is provided because they are not valid at any cycle point, so ID matching filters them out.
- Even if we were able to operate on them, the remove, set and trigger operations would likely fail anyway due to missing taskdef entries.
- I think there's a bug in broadcasts too as families loose their inheritance causing them to disassociate from family broadcasts that previously targetted them.
(see #7196, #7197)
The Solution
When tasks are orphaned by reload or restart, cylc remove them. This will kill any active orphans (putting them into the failed state) and clean up any waiting tasks which are no longer needed (incl partially-satisfied waiting tasks).
Questions
- Remove waiting orphans?
- Kill and remove non-waiting orphans (preparing, submitted, running, in-complete).
- Note, if we do not do this, then we need to fix all the bugs so that users can still interact with these tasks.
- If we were to do this for 8.7.0, what would we do about the issues on 8.6.x?
(strong contender for the worst Cylc terminology to date, apologies)
When tasks are removed from the graph by reload or restart, any instances of those tasks in the active window are "orphaned" from their families.
The current strategy (inherited from Cylc 7) is to have the
rootfamily "adopt" orphans (although this appears only to happen for reload, not restart, and is only reflected in one of severalWorkflowConfiginterfaces leading to internal inconsistencies).These orphans then persist until they leave the pool naturally, or are manually cleaned up by the user, after which, the workflow runs cleanly according to the new config.
However, these orphans can sometimes trip users up as an unexpected consequence of performing a graph change. They may lead to workflow stall and have caused a fair few bugs, most recently #7196 & #7197.
The current arrangement may be considered a hangover of SoS logic where orphans made more sense. In the SoD world, we could simplify the situation by removing orphans rather than persisting them. The user has said they no longer want these tasks, perhaps the simplest solution is just to get rid of them.
Orphan Tasks Example
For example, this workflow has two chains of tasks
a => b => candd => e => f. The second chain may be toggled on/off.The workflow will toggle the chain off during the runtime of
1/e:The result of this (post
1/ecompletion) is three orphan tasks, all in the waiting states:2/e3/e4/dThe Problem
Predictability:
The orphan tasks you get is the result of the graph state at the time the reload was performed, it's not predictable what the result of the graph change will be in this regard. As a result, it is tricky for the user to postulate over what orphan tasks might be left, might consequently run, or require removal.
Downstream Issues:
Orphan tasks may lead to workflow stall as they may be left with partially-satisfied prerequisites which can never be satisfied as the result of the graph change.
Bugs:
Orphan tasks are an edge case, they have no corresponding entry in
WorkflowConfig.taskdefs, they aren't included inWorkflowConfig.runtime['descendants'], their graph edges are severed, their inheritance is disrupted. All of these internal inconsistencies may lead to bugs.E.g ATM:
rootfamily, except in oneWorkflowConfiginterface).(see #7196, #7197)
The Solution
When tasks are orphaned by reload or restart,
cylc removethem. This will kill any active orphans (putting them into the failed state) and clean up any waiting tasks which are no longer needed (incl partially-satisfied waiting tasks).Questions