Since upgrading to k8s 1.32 the cluster finds itself in an autoscaling loop.
Testing is unstable and continues to add and remove nodes.
Production is stable, but on 4 nodes even though it's not needed.
Log output from the autoscaler suggests that Prometheus not being able to bind it's persistent volumes might be the problem.
"ip-10-0-2-176.eu-west-1.compute.internal" pod="monitoring/prometheus-prometheus-0" err="no matching NodeSelectorTerms"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036483 1 klogx.go:87] failed to find place for monitoring/prometheus-prometheus-0: can't schedule pod monitoring/prometheus-prometheus-0: couldn't fin
d a matching Node with passing predicates
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036505 1 filter_out_schedulable.go:122] 0 pods marked as unschedulable can be scheduled.
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036524 1 filter_out_schedulable.go:85] No schedulable pods
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036540 1 filter_out_daemon_sets.go:47] Filtered out 0 daemon set pods, 1 unschedulable pods left
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036583 1 klogx.go:87] Pod monitoring/prometheus-prometheus-0 is unschedulable
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036607 1 orchestrator.go:110] Upcoming 0 nodes
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036706 1 binder.go:812] "Could not get a CSINode object for the node" logger="Filter.VolumeBinding" node="template-node-for-eks-main-pool-202404040739
44333300000004-40c75406-5214-3296-49ff-82658c2c2a18-5051851628976474691" err="csinode.storage.k8s.io \"template-node-for-eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18-505185162897
6474691\" not found"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036740 1 binder.go:838] "All bound volumes for pod match with node" logger="Filter.VolumeBinding" pod="monitoring/prometheus-prometheus-0" node="templ
ate-node-for-eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18-5051851628976474691"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.036966 1 binder.go:812] "Could not get a CSINode object for the node" logger="Filter.VolumeBinding" node="template-node-for-eks-main-pool-202404040739
44333300000004-40c75406-5214-3296-49ff-82658c2c2a18-5051851628976474691-e-0" err="csinode.storage.k8s.io \"template-node-for-eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18-50518516
28976474691-e-0\" not found"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.037005 1 binder.go:838] "All bound volumes for pod match with node" logger="Filter.VolumeBinding" pod="monitoring/prometheus-prometheus-0" node="templ
ate-node-for-eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18-5051851628976474691-e-0"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.039844 1 orchestrator.go:185] Best option to resize: eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.039857 1 orchestrator.go:189] Estimated 1 nodes needed in eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.039875 1 orchestrator.go:254] Final scale-up plan: [{eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18 3->4 (max: 5)}]
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.039901 1 executor.go:166] Scale-up: setting group eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18 size to 4
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.039928 1 auto_scaling_groups.go:267] Setting asg eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18 size to 4
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.040098 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"
03be89ef-d018-4e77-95c0-d164cd292820", APIVersion:"v1", ResourceVersion:"388143342", FieldPath:""}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group eks-main-pool-20240404073944333300000004-40c754
06-5214-3296-49ff-82658c2c2a18 size to 4 instead of 3 (max: 5)
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.165281 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"
03be89ef-d018-4e77-95c0-d164cd292820", APIVersion:"v1", ResourceVersion:"388143342", FieldPath:""}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: group eks-main-pool-20240404073944333300000004-40c75406-5214-
3296-49ff-82658c2c2a18 size set to 4 instead of 3 (max: 5)
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.166865 1 eventing_scale_up_processor.go:47] Skipping event processing for unschedulable pods since there is a ScaleUp attempt this loop
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:38.172989 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Pod", Namespace:"monitoring", Name:"prometheus-prometheus-0", UID:"96522357-
a2b2-4742-8cb6-59e8226f4e51", APIVersion:"v1", ResourceVersion:"388142361", FieldPath:""}): type: 'Normal' reason: 'TriggeredScaleUp' pod triggered scale-up: [{eks-main-pool-20240404073944333300000004-40c75406-5
214-3296-49ff-82658c2c2a18 3->4 (max: 5)}]
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:42.649249 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:42.649290 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 14.48µs
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:48.183358 1 static_autoscaler.go:274] Starting main loop
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:48.185317 1 aws_manager.go:188] Found multiple availability zones for ASG "eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18"
; using eu-west-1b for failure-domain.beta.kubernetes.io/zone label
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:48.185565 1 filter_out_schedulable.go:65] Filtering out schedulables
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:48.185832 1 binder.go:832] "PersistentVolume and node mismatch for pod" logger="Filter.VolumeBinding" PV="pvc-d80961bd-a1ec-44c0-b199-5c7603143a3e" node=
"ip-10-0-1-204.eu-west-1.compute.internal" pod="monitoring/prometheus-prometheus-0" err="no matching NodeSelectorTerms"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:48.185931 1 binder.go:832] "PersistentVolume and node mismatch for pod" logger="Filter.VolumeBinding" PV="pvc-d80961bd-a1ec-44c0-b199-5c7603143a3e" node=
"ip-10-0-2-176.eu-west-1.compute.internal" pod="monitoring/prometheus-prometheus-0" err="no matching NodeSelectorTerms"
aws-cluster-autoscaler-64544d6dfd-75pq2 I0812 13:11:48.185982 1 binder.go:812] "Could not get a CSINode object for the node" logger="Filter.VolumeBinding" node="template-node-for-eks-main-pool-202404040739
44333300000004-40c75406-5214-3296-49ff-82658c2c2a18-4187874590498881149-upcoming-0" err="csinode.storage.k8s.io \"template-node-for-eks-main-pool-20240404073944333300000004-40c75406-5214-3296-49ff-82658c2c2a18-4
187874590498881149-upcoming-0\" not found"
Since upgrading to k8s 1.32 the cluster finds itself in an autoscaling loop.
Testing is unstable and continues to add and remove nodes.
Production is stable, but on 4 nodes even though it's not needed.
Log output from the autoscaler suggests that Prometheus not being able to bind it's persistent volumes might be the problem.