Skip to content

Retry failed remote deletion requests before dropping pods from the virtual-kubelet cache #540

Description

@Bianco95

Provider.DeletePod() currently sends the remote /delete request asynchronously and immediately removes the pod from p.pods.
If the request fails because the Slurm plugin or network is temporarily unavailable:

If the request fails because the Slurm plugin or network is temporarily unavailable:

  • the error is only logged;
  • the pod is no longer tracked by the virtual kubelet;
  • the status loop stops reporting it;
  • no later retry is attempted;
  • the corresponding Slurm job may continue running indefinitely

Deletion should be made reliable by either waiting for the remote deletion to succeed before removing the pod from p.pods, or by adding a durable retry mechanism. The pod should remain eligible for cleanup retries until the remote side confirms deletion.

Please add tests covering transient /delete failures and successful retry on a later attempt.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions