Description
A source-level review found that ModelServing controller worker goroutines do not recover from unexpected panics. In particular, pod processing can index pod.OwnerReferences[0] without first verifying that the pod has an owner reference.
If a pod without owner references reaches this path, the controller panics. Because the worker goroutine does not have panic recovery, the panic can terminate the ModelServing controller process and cause a controller crash loop.
This was found through source review and has not been reproduced in a live cluster yet.
Steps to reproduce the issue
1.Run the ModelServing controller.
2.Have a pod processed by the ModelServing controller with an empty OwnerReferences list.
3.The pod-processing worker accesses:
4.Since OwnerReferences is empty, this causes an index out of range panic.
5.The worker does not recover the panic.
6.The ModelServing controller process terminates and is restarted by Kubernetes.
Evidence and production path
Pod processing
↓
ModelServing controller worker goroutine
↓
pod.OwnerReferences[0]
↓
OwnerReferences is empty
↓
index out of range panic
↓
no panic recovery in worker goroutine
↓
ModelServing controller process terminates
↓
controller crash/restart
Pods without owner references are possible, so the controller should not assume OwnerReferences[0] always exists.
This was identified through source review. A live-cluster reproduction and panic stack trace have not been obtained yet.
Describe the results you received and expected
Actual result
A pod without owner references can trigger an index out of range panic in the ModelServing controller. The panic is not recovered by the worker goroutine and can terminate the controller process.
###vExpected result
The controller should safely handle pods without owner references, for example by skipping the pod and logging an appropriate message, without crashing the controller.
Unexpected panics in worker goroutines should also be recovered so that a single malformed or unexpected pod cannot terminate the entire controller.
What version of Kthena are you using?
main / exact commit to be filled in when reproduced.
Any other relevant information
No response
Description
A source-level review found that ModelServing controller worker goroutines do not recover from unexpected panics. In particular, pod processing can index pod.OwnerReferences[0] without first verifying that the pod has an owner reference.
If a pod without owner references reaches this path, the controller panics. Because the worker goroutine does not have panic recovery, the panic can terminate the ModelServing controller process and cause a controller crash loop.
This was found through source review and has not been reproduced in a live cluster yet.
Steps to reproduce the issue
1.Run the ModelServing controller.
2.Have a pod processed by the ModelServing controller with an empty OwnerReferences list.
3.The pod-processing worker accesses:
4.Since OwnerReferences is empty, this causes an index out of range panic.
5.The worker does not recover the panic.
6.The ModelServing controller process terminates and is restarted by Kubernetes.
Evidence and production path
Pod processing
↓
ModelServing controller worker goroutine
↓
pod.OwnerReferences[0]
↓
OwnerReferences is empty
↓
index out of range panic
↓
no panic recovery in worker goroutine
↓
ModelServing controller process terminates
↓
controller crash/restart
Pods without owner references are possible, so the controller should not assume OwnerReferences[0] always exists.
This was identified through source review. A live-cluster reproduction and panic stack trace have not been obtained yet.
Describe the results you received and expected
Actual result
A pod without owner references can trigger an index out of range panic in the ModelServing controller. The panic is not recovered by the worker goroutine and can terminate the controller process.
###vExpected result
The controller should safely handle pods without owner references, for example by skipping the pod and logging an appropriate message, without crashing the controller.
Unexpected panics in worker goroutines should also be recovered so that a single malformed or unexpected pod cannot terminate the entire controller.
What version of Kthena are you using?
main / exact commit to be filled in when reproduced.
Any other relevant information
No response