It should be possible to perform incremental updates to large/complex variable values without blowing the entire variable value away

[jswalden]

youtube-live has all sorts of variables named broadcast_${broadcastID}_{datum} and similar, for datum=health/state and for all broadcasts that have been fetched. It also has unfinished_${i}_{datum}, for datum=state/id/name/shortName/health/etc. (The numbers of broadcast IDs, and of unfinished broadcasts to expose, depend on configuration options.)

The module presently, therefore, exposes a huge mess of flat variables. Conveniently, in some cases where it needs a refresh of the state of a single broadcast, it can therefore update only the few variables that pertain to that broadcast.

When variable values can be JSONValue, it'd be saner to have type YouTubeVariables = { broadcasts: Record<BroadcastID, { health, state }>; unfinishedBroadcasts: { state; id; name; shortName; health }[] };. But then you lose ability to do narrow updating of only the data pertinent to one broadcast -- you'd have to update all of broadcasts (and possibly unfinishedBroadcasts) if you had fresh info for one broadcast in both.

(It is difficult to stick with the existing mess of flat variables, as TypeScript doesn't have good understanding of template string-keyed fields of varying type within the same object -- ultimately degrading key type to string and value type to a union of all property value types.)

I could imagine a registry-style API where you can specify paths into a config store (initial leaf being the name of a variable) and only update those specific paths, without throwing out the rest of the system. But I haven't thought at too much length about all this, and there might well be a better/cleaner way.

[jswalden]

I've done a decent bit of experimentation around this using something similar to type-fest's Path utility. (It's possible that utility would substitute for what I wrote, with some trouble. Not sure.) Or, similar to the JSON Pointer concept, but more flavored to what TypeScript can let you do than a raw string format like that.

As such things go, it's reasonably straightforward to imagine something like a variable-update message specifying a variable ID, an array of components indicating how to descend into the variable value, and a value to replace that selected subtree in the existing variable. (A components array of [] would replace the entire variable.)

The difficulty starts popping up when you start considering how subtrees that are type unions get handled. Imagine replacing a subtree within a nullable field somewhere, for example:

type VariablesConfig = {
    extra: {
        field: [string];
    } | null;
};

and then think about what components like ['extra', 'field', 0] would mean -- fail out if some intermediate stage is unavailable or a primitive, I guess?

Or consider a field of two distinct types, yet somehow a nested reference within the intersection of them can still be named -- for setting you'd want to require the intersection of those types, but if they don't intersect, it might just be impossible to actually update just that subtree.

type VariablesConfig = {
    either: {
        x: number
    } | {
        x: string
    };
};

Consider ['either', 'x'] which for setting purposes would have to resolve to number & string to satisfy both halves of the union, but that basically means that field is unaddressable (in type-safe fashion, at least). Maybe it's acceptable to say the user can only update bigger components that would contain such unions' complexities within them -- so you could usefully use ['either'] and just accept that you'd have to replace it over all, not just a narrow field in it. But this might be unpleasant for users to accept.

Whether or not unions, nullable complex types, etc. would be common enough for this all to founder for these reasons from the module developer perspective, I dunno. But it's enough for me to stop noodling on the partial work I did on the problem, at least.

[Julusian]

I'm not sure on the type safety aspect, but I would think that using the json-patch (rfc6902) format would be best.
I suppose maybe there is a question of whether this should be done to each variable value, or to the whole object?

This isnt very high in my priorities currently, but a plan would make it easier to drop into

[jswalden]

json-patch would be perfect if variable values were no more refined than simply JsonValue. But because of more-refined types existing, and TypeScript fundamentally involving complete type erasure at runtime, a JSON patch would run into the same kinds of type safety considerations around unions, unfortunately. (json-patch 'test' operations can't even help here: they're not type tests, they're exact value tests only, and exact value tests would only help for tagged-union object values basically.)

(I did some reading in the space and stumbled across json-patch, json-pointer, json-path. I think json-patch actually showed up earlier when I was looking at the manifest.json tweaks for #198 and #199, til I realized just having incorporation by reference required no more than basic JSON schema use to achieve.)

I was hoping I could jump-start the work by doing some reasonable portion of it, perhaps even all of it 🙂 but these typing considerations on my intuition about a first approach at an interface suggests a completely different approach is necessary. Or perhaps one that just doesn't generalize, like say allowing add/set/remove operations on the values of object-valued variables to the union of all the object's value types, or push/pop/remove/delete actions on array variables-valued variables?

[jswalden]

I guess the other option would be something like using a library that can generate a JSON Schema from a type -- I think there is in fact such a thing, if my recollections of my research hold -- then you apply the requested change and check whether it still conforms to the schema. But we're getting kind of absurdly engineered by the time we're doing something like, seems to me.

[jswalden]

Or perhaps one that just doesn't generalize, like say allowing add/set/remove operations on the values of object-valued variables to the union of all the object's value types, or push/pop/remove/delete actions on array variables-valued variables?

At least in theory I have patches completed to implement this. But work to update youtube-live to the 2.0 API isn't complete yet (updating presets and getting them making use of templates and all is a bear, I probably should be reading API definitions rather than trying to bull my way through just tweaking what was already defined), so I can't yet say how close it is to working.