Skip to content

feat(core): add graceful shutdown and concurrency safety - #18

Open
hajnalmt wants to merge 4 commits into
volcano-sh:masterfrom
hajnalmt:feat/code-quality
Open

feat(core): add graceful shutdown and concurrency safety#18
hajnalmt wants to merge 4 commits into
volcano-sh:masterfrom
hajnalmt:feat/code-quality

Conversation

@hajnalmt

@hajnalmt hajnalmt commented May 7, 2026

Copy link
Copy Markdown
Member

Summary

  • Implements graceful shutdown via signal.NotifyContext (SIGTERM/SIGINT)
  • Adds sync.RWMutex protection for package-level mutable state (numaMap, config)
  • Extracts hardcoded environment variable name into a named constant
  • Improves error messages for missing configuration

Stacked PR

This PR is stacked on top of PR #16. The relevant commit for review is b77e504.

Related Issues

Design

Graceful shutdown

The main loop now uses signal.NotifyContext to catch SIGTERM and SIGINT. When a signal is received:

  1. The context is cancelled
  2. The ticker is stopped (via defer tick.Stop())
  3. klog is flushed (via defer klog.Flush())
  4. A shutdown message is logged before the process exits

This is critical for Kubernetes where pods receive SIGTERM before being forcefully killed. Without signal handling, the process would be mid-operation when killed, potentially leaving stale state.

Concurrency safety

Package-level mutable state is now protected with sync.RWMutex:

  • numaMap in framework.go — holds registered topology info providers and their latest state
  • config in kubeletconfig.go — holds kubelet policy and reservation data

Write operations (RegisterNumaType, TopoInfoUpdate, TryUpdatingResourceReservation) take a write lock. Read operations (GetPolicy, GetResReserved, GetAllResAllocatableInfo, GetCpusDetail) take a read lock.

While the current implementation is single-goroutine, this makes the code safe for future concurrency (e.g., adding an HTTP health endpoint on a separate goroutine) and prevents subtle bugs.

Constants

The MY_NODE_NAME environment variable string is extracted into a const envNodeName in both main.go and update.go, eliminating magic strings and improving error messages.

AI Assistance Disclosure

This PR was developed with AI guidance. The author reviewed, directed, and validated all changes carefully.

Signals come, the daemon hears,
no more dying mid-career.
Mutex guards the shared terrain,
future threads may run in vain—
but never race, and never clash,
for locks prevent the data crash.

Closes

hajnalmt added 4 commits May 7, 2026 11:48
Remove the k8s.io/kubernetes internal dependency (anti-pattern) and the
cadvisor dependency entirely. Replace them with minimal local packages
that read directly from sysfs/procfs and parse kubelet state files as
simple JSON.

Key changes:
- Go 1.15 -> 1.25, k8s.io/* v0.19.6 -> v0.35.0
- klog v1 -> klog/v2 v2.130.1
- volcano.sh/apis updated to v1.14.2
- Remove 25-entry replace block (now only 1 for volcano.sh/apis)
- Add pkg/internal/{cpuset,checkpoint,capacity,eviction} as lightweight
  replacements for k8s.io/kubernetes internals
- Fix unchecked error in checkpoint unmarshal
- Fix "scoket_id" typo
- Fix shadowed builtin `cap` variable
- Replace all deprecated ioutil usage with os package

Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
- Reject eviction threshold percentages outside 0-100 range
- Reject reversed CPU ranges (end < start) in capacity parser
- Validate single CPU values are numeric
- Lower noisy klog.Infof to V(4) in reservation calculation
- Add table-driven unit tests for cpuset, checkpoint, capacity, and
  eviction packages covering edge cases and invalid inputs

Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Rewrite the build system for modern Go development:
- Makefile: add lint (golangci-lint), test (-race), and clean targets;
  decouple image build from local binary via multi-stage Docker
- Dockerfile: multi-stage build (golang:1.25 builder + distroless/static
  nonroot runtime) replacing the bare alpine image
- .golangci.yml: golangci-lint v2 config with errcheck, govet,
  ineffassign, staticcheck, unused, and revive enabled
- GitHub Actions CI: lint, test, and build jobs on push/PR
- pkg/version: new package for build-time metadata via ldflags

Fix lint violations caught by the new tooling:
- main.go: for { select { case <-tick.C } } -> for range tick.C
- machineinfo: rename cap variable to avoid shadowing builtin
- numatopo: getNumaNodeCpuCap -> getNumaNodeCPUCap,
  getCoreIDSocketIDForCpu -> getCoreIDSocketIDForCPU
- Add package comments to all packages

Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Implement signal handling with signal.NotifyContext so the exporter
shuts down cleanly on SIGTERM/SIGINT. This is critical for Kubernetes
pods which receive SIGTERM before being killed.

Add sync.RWMutex protection for the package-level numaMap and config
variables. While the current implementation is single-goroutine, this
makes the code safe for future concurrency and prevents subtle bugs.

Extract the MY_NODE_NAME environment variable into a named constant
and improve error messages for missing configuration.

Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
@volcano-sh-bot
volcano-sh-bot requested review from Thor-wl and wpeng102 May 7, 2026 11:28
@volcano-sh-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign k82cn for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several structural improvements to the resource-exporter, including the addition of internal packages for capacity, checkpoint, cpuset, and eviction logic. It also enhances the build process via Makefile updates, adds graceful shutdown handling with signal notifications, and ensures thread-safety for shared state by introducing mutexes. The code review identified an opportunity to improve the robustness of the countCPUs function by providing more descriptive error handling for malformed input strings, which has been acknowledged as a valid improvement.

Comment on lines +70 to +88
if len(parts) == 1 {
if _, err := strconv.Atoi(strings.TrimSpace(parts[0])); err != nil {
return 0, err
}
count++
} else if len(parts) == 2 {
start, err := strconv.Atoi(strings.TrimSpace(parts[0]))
if err != nil {
return 0, err
}
end, err := strconv.Atoi(strings.TrimSpace(parts[1]))
if err != nil {
return 0, err
}
if end < start {
return 0, fmt.Errorf("invalid CPU range: start %d > end %d", start, end)
}
count += end - start + 1
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation of countCPUs silently ignores invalid range parts, such as 0-1-2 in a string like 0-1-2,3. This could lead to an incorrect CPU count without any indication of an error. It would be more robust to return an error for such malformed range parts. Using a switch statement can make the logic clearer and handle invalid formats explicitly. This change also improves error messages for better diagnostics.

		switch len(parts) {
		case 1:
			if _, err := strconv.Atoi(strings.TrimSpace(parts[0])); err != nil {
				return 0, fmt.Errorf("invalid CPU ID in range part %q: %w", r, err)
			}
			count++
		case 2:
			start, err := strconv.Atoi(strings.TrimSpace(parts[0]))
			if err != nil {
				return 0, fmt.Errorf("invalid start CPU ID in range part %q: %w", r, err)
			}
			end, err := strconv.Atoi(strings.TrimSpace(parts[1]))
			if err != nil {
				return 0, fmt.Errorf("invalid end CPU ID in range part %q: %w", r, err)
			}
			if end < start {
				return 0, fmt.Errorf("invalid CPU range: start %d > end %d in %q", start, end, r)
			}
			count += end - start + 1
		default:
			return 0, fmt.Errorf("invalid CPU range format: %q", r)
		}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve code quality: graceful shutdown and concurrency safety

2 participants