Skip to content

Add initial K8s web HPA. - #5951

Open
simi wants to merge 1 commit into
masterfrom
hpa
Open

Add initial K8s web HPA.#5951
simi wants to merge 1 commit into
masterfrom
hpa

Conversation

@simi

@simi simi commented Sep 6, 2025

Copy link
Copy Markdown
Contributor
  • not super sure about the thresholds
  • should pair nicely with EKS node auto scaling

@codecov

codecov Bot commented Sep 6, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.21%. Comparing base (ca56873) to head (86b4654).
⚠️ Report is 658 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5951   +/-   ##
=======================================
  Coverage   97.21%   97.21%           
=======================================
  Files         473      473           
  Lines        9764     9764           
=======================================
  Hits         9492     9492           
  Misses        272      272           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@simi
simi marked this pull request as ready for review September 6, 2025 10:18

@colby-swandale colby-swandale left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I think we need to take a closer look at usage metrics we have in Datadog to see if we're actually hitting 100% CPU. I had some metrics from Kubernetes on my second monitor last week, and I don't think I ever saw the CPU get that high.
  • If this change is merged, a quick heads-up that it's almost certainly going to be replaced with a new policy based on queue time (https://judoscale.com/blog/request-queue-time) that I've been building up #5945.

annotations:
shipit.shopify.io/restart: 'true'
spec:
replicas: <%= environment == 'production' ? 3 : 2 %>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifying a replica and HPA policy are exclusive, they will try to overwrite one another

stabilizationWindowSeconds: 60
policies:
- type: Percent
value: 100

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are our web pods hitting 100% CPU? Can you check with Datadog to review what the ideal CPU usage percentage would be for scaling up?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our web pods are not hitting 100+% often, but it has happened a few times in the past month 👇

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants