uis subscription data sync info (UIS-597)

[dwsutherland]

sibling of cylc/cylc-uiserver#597

Have included testing some of the changes, however the parts not tested are used by cylc-uiserver not cylc-flow.

This end is to:

• hook into resolvers to extract which workflows a subscription/query is targeting
• adds an API endpoint and store stub

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests included (limited), as functionality is only used UIS side.
• CHANGES.md entry included if this is a change that can affect users
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

Reminder (largely for myself): This PR is pending an upstream implementation in cylc-uiserver.

Drafting for now as this code may change in relation to upstream requirements.

[oliver-sanders]

(unassigning this to re-focus my review queue, feel free to re-assign when undrafted)

[dwsutherland]

Don't think the lint failure is relevant:

Exception: HTTP Error 429: Too Many Requests

[oliver-sanders]

As I understand it:

• When a subscription is started, or a query is issued, the timer field is set.
• Once the timeout has elapsed, the subscription will be recalculated.
• (When a subscription is stopped, we should also set the timer, but I'm not sure we currently are?)

Just wondering if it might be easier to add hook methods, e.g, query_issues, subscription_started, subscription_stopped to avoid having to set and track these timeouts, e.g:

def  query_issued(self, w_id):
    self.loop.call_later(TIMEOUT, self.recalculate_subscription, w_id)

def  subscription_stopped(self, w_id):
    self.loop.call_later(TIMEOUT, self.recalculate_subscription, w_id)

[oliver-sanders]

This is the same as the existing function, just with the addition of the await self.data_store_mgr.set_query_sync_levels(set(w_ids)) bit.

Consequently, we have to duplicate the if is_sub: ...; else logic to toggle between these two implementations for all uses below.

I think we can simplify things by merging the two implementations like so:

async def get_flow_data_from_ids(self, data_store, native_ids, is_sub):
    w_ids = {Tokens(w_id).workflow_id for w_id in native_ids}
    if is_sub:
        await self.data_store_mgr.set_query_sync_levels(w_ids)
    return [data_store[w_id] for w_id in w_ids if w_id in data_store]

Then pass trough a bool flag to toggle:

flow_data = get_flow_data_from_ids(
    self.data_store_mgr.data,
    nat_ids,
    bool(args['delta_store'])
)

[dwsutherland]

Will have to check if it needs to be order preserving (i.e. for tests), but yeah maybe a list would work the same for query sync level setting.

[dwsutherland]

Although there may be some simplification not thought of, both calls also pass in a different store: self.delta_store[args['sub_id']] and self.data_store_mgr.data according to args['delta_store']..

And although args['delta_store'] implies is_sub, some subs resolve from the main store self.data_store_mgr.data .. So bool(args['delta_store']) can be False while is_sub is True.

(I did attempt simplification for a while.. but got stuck on this compication)

[dwsutherland]

As I understand it:

• When a subscription is started, or a query is issued, the timer field is set.
• Once the timeout has elapsed, the subscription will be recalculated.
• (When a subscription is stopped, we should also set the timer, but I'm not sure we currently are?)

Off the top of my head, there's no timer on subscriptions (stays up for as long as the subscription is active), however, the query timer trumps the subscription.. So once the query timer is spent, it falls back to the subscription..

The query data types are added to the level for the duration of the timer, and then the subscription topics are determined from this, both when the query is made and when the timer finishes..

[dwsutherland]

Back compatibility in:
cylc/cylc-uiserver#597 (comment)