RecoveryPolicy=None still deletes failed pods with no recreation, contradicting the documented behavior

[0xjasoncao]

Description

kthena defines NoneRestartPolicy as:

▎ NoneRestartPolicy will follow the same behavior as the default pod or deployment.

That is, when RecoveryPolicy is set to None, a failed pod should be restarted in place by the kubelet according to the pod's own
restartPolicy, just like a normal Deployment. The controller should not intervene.

Problem

The actual behavior contradicts the docs: with RecoveryPolicy=None, the controller still deletes the failed pod, and nothing recreates it
afterward — so the pod is gone for good and the replica count can no longer be maintained.

Root-cause chain:

1. Pod fails / container restarts → updatePod enters handleErrorPod.
2. handleErrorPod unconditionally starts go c.handlePodAfterGraceTime(ms, errPod)
   (pkg/model-serving-controller/controller/model_serving_controller.go).
3. When RestartGracePeriodSeconds is not set (or is 0), handlePodAfterGraceTime deletes the pod immediately
4. The pod is deleted → deletePod is triggered → handleDeletedPod, whose switch ms.Spec.RecoveryPolicy has only ServingGroupRecreate and
   RoleRecreate cases — there is no None branch, so None falls through to the default and does nothing.
5. Result: the pod is deleted but never recreated, contradicting the None semantics.

Steps to reproduce the issue

example:

  apiVersion: workload.serving.volcano.sh/v1alpha1
  kind: ModelServing
  metadata:
    name: busybox-recovery-none
    namespace: kthena-test
  spec:
    replicas: 1
    schedulerName: default-scheduler
    recoveryPolicy: None
    template:
      roles:
        - name: worker
          replicas: 3
          workerReplicas: 0
          entryTemplate:
            metadata:
              labels:
                app: busybox-recovery-none
            spec:
              restartPolicy: Always
              containers:
                - name: busybox
                  image: ghcr.io/containerd/busybox:1.36
                  env:
                    - name: POD_NAME
                      valueFrom:
                        fieldRef:
                          fieldPath: metadata.name
                  command:
                    - sh
                    - -c
                    - |
                      idx=$(echo "$POD_NAME" | awk -F- '{print $(NF-1)}')
                      case "$idx" in
                        0) sleep 10 ;;
                        *) sleep 3600 ;;
                      esac
                  resources:
                    requests:
                      cpu: 50m
                      memory: 16Mi
                    limits:
                      cpu: 100m
                      memory: 32Mi

Steps & observations:

1. Create the ModelServing above. All 3 worker pods become Running; Role / ServingGroup status is Running.
2. About 10s later, worker-0's container exits. The kubelet restarts it in place once (restartPolicy: Always); the controller detects the
   failure and calls handleErrorPod.
3. Because no grace period is set, handlePodAfterGraceTime deletes the pod immediately.
4. The deletion triggers handleDeletedPod; RecoveryPolicy=None has no matching case, so the pod is never recreated.
5. worker-1 and worker-2 stay Running long-term as controls.

Evidence and production path

Logs

  modelServing=kthena-test/busybox-recovery-none (0 -> 1)
  I0820 11:25:57.356993       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleCreating" message="Role worker/worker-0 in ServingGroup
  busybox-recovery-none-0 is now Creating"
  I0820 11:25:57.358988       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleCreating" message="Role worker/worker-1 in ServingGroup
  busybox-recovery-none-0 is now Creating"
  I0820 11:25:57.361119       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleCreating" message="Role worker/worker-2 in ServingGroup
  busybox-recovery-none-0 is now Creating"
  I0820 11:25:58.672673       1 model_serving_controller.go:1429] Update role worker/worker-0 status to Running
  I0820 11:25:58.672780       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleRunning" message="Role worker/worker-0 in ServingGroup
  busybox-recovery-none-0 is now Running"
  I0820 11:25:58.732770       1 model_serving_controller.go:1429] Update role worker/worker-1 status to Running
  I0820 11:25:58.732877       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleRunning" message="Role worker/worker-1 in ServingGroup
  busybox-recovery-none-0 is now Running"
  I0820 11:25:58.950607       1 model_serving_controller.go:1429] Update role worker/worker-2 status to Running
  I0820 11:25:58.950639       1 model_serving_controller.go:1448] Update ServingGroup busybox-recovery-none-0 status to Running
  I0820 11:25:58.950913       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleRunning" message="Role worker/worker-2 in ServingGroup
  busybox-recovery-none-0 is now Running"
  I0820 11:26:08.985045       1 model_serving_controller.go:1448] Update ServingGroup busybox-recovery-none-0 status to Running
  I0820 11:26:19.848662       1 model_serving_controller.go:1479] update role worker/worker-0 to Creating when pod fails
  I0820 11:26:19.848692       1 model_serving_controller.go:1493] update ServingGroup busybox-recovery-none-0 to processing when pod fails
  I0820 11:26:19.849011       1 event.go:389] "Event occurred" object="kthena-test/busybox-recovery-none" fieldPath="" kind="ModelServing"
  apiVersion="workload.serving.volcano.sh/v1alpha1" type="Normal" reason="RoleCreating" message="Role worker/worker-0 in ServingGroup
  busybox-recovery-none-0 is now Creating"
  I0820 11:26:19.851055       1 model_serving_controller.go:1538] busybox-recovery-none-0-worker-0-0 been deleted without grace time

Describe the results you received and expected

With RecoveryPolicy=None, the controller should not delete the failed pod; the kubelet should restart the container in place per the pod's restartPolicy.

What version of Kthena are you using?

V1.0.0

Any other relevant information

No response

[chenhuiluo]

/assign