Metrics-related bug fixes

[psilabs-dev]

Problem 1: The metrics leak that can span a couple months (resulting in a reported 100gb memory on grafana in a machine with only 96gb ram) is traced back to worker stats not being cleaned up on exit.

E.g.:

• worker X, Y reported memory usage of 1mb, total memory usage 2mb
• worker X dies, but 1mb usage still lives in redis. total memory usage is still 2mb (even though actual usage is just Y)
• rinse and repeat with hundreds of dying workers, you get a load of ghost ram usage

Any gauge metric reported by an exited worker poisons the actual metrics cache, because complete cleanup happens only on LRR start (which there is a reason why cleanup doesn't happen during LRR runtime).

Consider why non-gauge metrics do not need to be cleaned up when a worker exits though.

Problem 2: Redis stores the metrics of dead workers, and the way we count total active workers is which PIDs are in the metrics cache. What that actually counts is all workers, dead or alive.

These two issues are addressed by an architectural tweak:

• worker metrics are now an explicit lifecycle problem, when mojo reaps children processes, their corresponding PID is used to clean up gauge metrics.
• mojo lifecycle also includes active worker registration and unregistration.

Additionally, the endpoints recorded by metrics are 3-months out of date, so this is updated (maybe Utils/Metrics.pm might do some parse-and-build of openapi.yaml so that we can dynamically handle this problem...).

In draft: todo pending on any non-mojo process that still needs lifecycle management.