fix: GPUReset job creation failures and invalid partial drains

[natherz97]

Summary

This PR fixes 2 issues that can cause GPU resets to fail.

1. Ensure that GPU reset job names are valid RFC 1123 subdomain names. Depending on the node name, it was possible for the job name truncation logic in gpu_reset_controller.go to generate an invalid job name. Specifically, it was possible for the truncation to result in the the character sequence ".-" which cannot be included in an RFC 1123 subdomain name. This would result in the controller continually failing to create the GPU reset job. This phase of a GPU reset does not have a timeout and the CRD would never make it to a terminal state. Example node name and derived job name:

   a. Node name: maintenance-ip-10-1-115-210.us-east-2.compute.internal-6fca3f52-abcd1234
   b. Old job name: maintenance-ip-10-1-115-210.us-east-2.-2bba20f7-reset-job
   c. New job name: maintenance-ip-10-1-115-210.us-east-2-2bba20f7-reset-job

2. Ensure that partial drains do not skip pods stuck terminating or NotReady pods. Previously, full and partial drains would skip draining any pods which have a terminal status (failed/succeeded), are stuck terminating, or are NotReady. A pod which is stuck terminating or NotReady may still have running processes with GPU contexts. While reboot remediations would not be blocked on these processes, it's possible that a pod in this state could cause a GPU reset to fail. As a result, we will ensure that pod objects which are targeted by a partial drain are removed or in a terminal status prior to considering the drain as completed.

This sequence is especially problematic with COMPONENT_RESET remediations which are triggered from the gpu-health-monitor. We observed the following sequence:

1. An XID 48 was detected by the syslog-health-monitor and a DCGM_FR_VOLATILE_DBE_DETECTED error was detected by the gpu-health-monitor both requiring a reset against the same GPU
2. A pod was using all 4 GPUs on the given node which blocked the 2 GPU reset events in the node-drainer
3. The pod container requesting the GPUs restarted causing the pod to go NotReady
4. The node-drainer detected that the partial drain could be completed because the given pod was NotReady for 5 minutes
5. The 2 GPU reset events were de-duplicated into a single GPUReset CRD which failed its nvidia-smi command with the error "1 device is currently being used by one or more other processes".
6. A fallback reboot event was triggered which was blocked on a full drain. Since there were multiple pods, the single NotReady pod did not allow the full drain to succeed.
7. The pod went back to Ready status
8. As a side effect of the GPU reset, the nvidia-dcgm and gpu-health-monitor pods are recycled. This caused the gpu-health-monitor to fire another health event needing a GPU reset
9. The node-drainer detected that a previous drain had already completed against the given GPU and allowed the drain to be skipped even though the drain would've been blocked it we had re-evaluated (and would've seen the pod was Ready).
10. The follow-up GPU reset failed again due to the same "1 device is currently being used by one or more other processes".
11. We continually repeated steps 8-10 until the full node-drain completed.

Alternative mitigations would be to either:

1. Don't allow partial drains to be skipped even if a partial drain completed during the same quarantine session against the same GPU.
2. Add de-duplication to the gpu-health-monitor. This issue isn't unique to GPU resets and we could be stuck in reboot loops if a given fault requires RESTART_VM but the reboot doesn't resolve the underlying DCGM fault.
3. Only allow 1 GPUReset per quarantine session per GPU per node.

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📚 Documentation
• 🔧 Refactoring
• 🔨 Build/CI

Component(s) Affected

• Core Services
• Documentation/CI
• Fault Management
• Health Monitors
• Janitor
• Other: ____________

Testing

• Tests pass locally
• Manual testing completed
• No breaking changes (or documented)

Checklist

• Self-review completed
• Documentation updated (if needed)
• Ready for review

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-pull-request-1500.docs.buildwithfern.com/nvsentinel

[github-actions]

Merging this branch will increase overall coverage

Impacted Packages	Coverage Δ	🤖
github.com/nvidia/nvsentinel/janitor/pkg/controller	13.05% (+0.06%)	👍
github.com/nvidia/nvsentinel/node-drainer/pkg/informers	22.32% (+0.23%)	👍
github.com/nvidia/nvsentinel/node-drainer/pkg/reconciler	28.53% (ø)

---

Coverage by file

Changed files (no unit tests)

Changed File	Coverage Δ	Total	Covered	Missed	🤖
github.com/nvidia/nvsentinel/janitor/pkg/controller/gpureset_controller.go	13.01% (+0.14%)	5773	751 (+8)	5022 (-8)	👍
github.com/nvidia/nvsentinel/node-drainer/pkg/informers/informers.go	22.32% (+0.23%)	1752	391 (+4)	1361 (-4)	👍

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

• github.com/nvidia/nvsentinel/janitor/pkg/controller/gpureset_controller_test.go
• github.com/nvidia/nvsentinel/node-drainer/pkg/reconciler/reconciler_integration_test.go

[github-actions]

Merging this branch will increase overall coverage

Impacted Packages	Coverage Δ	🤖
github.com/nvidia/nvsentinel/janitor/pkg/controller	13.07% (+0.08%)	👍
github.com/nvidia/nvsentinel/node-drainer/pkg/informers	22.32% (+0.23%)	👍
github.com/nvidia/nvsentinel/node-drainer/pkg/reconciler	28.53% (ø)

---

Coverage by file

Changed files (no unit tests)

Changed File	Coverage Δ	Total	Covered	Missed	🤖
github.com/nvidia/nvsentinel/janitor/pkg/controller/gpureset_controller.go	13.04% (+0.17%)	5773	753 (+10)	5020 (-10)	👍
github.com/nvidia/nvsentinel/node-drainer/pkg/informers/informers.go	22.32% (+0.23%)	1752	391 (+4)	1361 (-4)	👍

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

• github.com/nvidia/nvsentinel/janitor/pkg/controller/gpureset_controller_test.go
• github.com/nvidia/nvsentinel/node-drainer/pkg/reconciler/reconciler_integration_test.go