Skip to content

Slurm client version compatibility in the sidecar image #128

Description

@matbun

The interlink-sidecar-slurm image ships Slurm client tools (sbatch, squeue, scancel, sinfo)
from AlmaLinux 9 default packages (currently Slurm 22.05). This version is incompatible with
newer Slurm clusters due to wire protocol changes between major versions.

For example, when the sidecar (22.05) connects to a Slurm 25.11 cluster (e.g. deployed via the
Slinky slurm-operator), sbatch fails with:

sbatch: error: Batch job submission failed: Insane message length

Additionally, the auth/slurm plugin (introduced in Slurm 23.11) and the JSON serializer
(serializer_json.so) are not built by default because the RPM spec requires --with jwt
and --with slurmrestd flags along with libjwt-devel and json-c-devel build dependencies.

Problem

  • The sidecar image is tightly coupled to a specific Slurm version but does not document or
    test which cluster versions it supports.
  • Users targeting newer Slurm clusters (23.11+, 24.x, 25.x) must build custom images, which
    is not straightforward (requires source builds with specific flags).
  • There is no CI to verify compatibility across Slurm versions.

Proposal

  • Publish sidecar images for multiple Slurm versions (e.g. interlink-sidecar-slurm:slurm-24.05,
    interlink-sidecar-slurm:slurm-25.11) or parametrize the Dockerfile with an ARG SLURM_VERSION.
  • Build Slurm from source with --with jwt --with slurmrestd to include auth_slurm.so,
    auth_jwt.so, and serializer_json.so.
  • Add integration tests that verify sbatch, sinfo, and job submission against different
    Slurm cluster versions.
  • Document supported Slurm versions in the README.

I can help with this, if needed. Let's just agree on a plan.
Some templating for flexible SLURM version already introduced here: matbun@e04d9d3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions