Add workers

[mohagan9]

Describe the feature request
Run jobs async via bull queue (like automations).

Instead of having a visual diagram, just have a JavaScript block which allows you to write some code.
There should be binding inputs (like any other query).

Additionally there should be a screen showing active jobs with filter on status:

• Error
• Queued ?
• Running (with percentage - more below)
• Complete

Existing queries should be selectable as functions with their bindings exposed as inputs. E.g.

// There is a SQL query called "Search Rows"
// It has bindings `text` and `limit`
search_rows({ text: "hello", limit: 5 }) 

Library usage should be heavily restricted, requiring existing queries instead of native fetch for example.

Additionally, there should be a way to emit progress via a helper function.
In code you can manually add progress markers between 0 and 1.
That way when helper.emitProgress(x) is called you will be able to see the percentage progress of the job in a list of jobs.

[mohagan9]

Instead of having an emitProgress, it may be better to implement checkpoints instead, which could support progress.

e.g.

checkpoint.save(state, progress)

I'll need to look into the likes of temporal to see how they do this, without having to add additional infra. The lightest out of the box approach is DBOS: https://github.com/dbos-inc/dbos-transact-ts

That still involves adding Postgres which I really don't want to for the sake of keeping memory usage down.

It may involve creating my own basic approach that saves checkpoint docs to CouchDB, but that'll require figuring out how to manage state variables.

e.g. consider the worker:

let x = 0 
for (x < 10) {
  // do thing
  checkpoint.save({ x }, x/10)
  x++
}

In this scenario if there is an error half way, how does one continue here? I assume I just need to turn the whole thing into a function.

type State = {
  x: number
}
function worker(state: State) {
  let x = state.x
  for (x < 10) {
    // do thing
    checkpoint.save(state, x/10)
  }
}
return worker(state)

I would need a way to allow the user to dynamically define the State typing. And the worker function would need to be a mandatory interface, with the state automatically fetched and injected.