⚡ hotreload

A reliable Go hot-reload tool featuring a TUI, Prometheus metrics, structured logging, crash-loop detection, and Alertmanager integration.

---

Overview

hotreload watches your Go projects for file changes, rebuilds automatically, and manages process restarts. Unlike standard watchers, it is built with observability and reliability in mind.

Save a file → watch hotreload cancel the in-flight build, recompile, and restart your server — all in under a second.

---

Demo & Proof of Work

The repository includes a purpose-built testserver to demonstrate hotreload's capabilities, particularly its resilience and crash recovery.

Run the demo with chaos mode enabled:

hotreload --build "go build -o tmp/testserver ./testserver" --exec "./tmp/testserver --chaos" --root ./testserver

What to expect:

1. The TUI launches, showing a ● RUNNING state.
2. The server exposes several routes on :8080 (e.g., /status for live stats, /slow for simulated latency).
3. Chaos Mode: A background worker randomly injects panics or fatal exits every 15–45 seconds.
4. Recovery: You will see the TUI instantly transition to ● CRASHED, increment its internal counters, and auto-recover back to ● RUNNING.
5. Metrics: Check http://localhost:2112/metrics to see the crash loops and restart counts actively tracked in Prometheus format.

(Try hitting http://localhost:8080/bomb in your browser to trigger an instant panic and watch hotreload catch it).

---

Features

Feature	Details
🔁 Instant Reload	Recursive directory watcher with debounced triggers (configurable, default 300 ms)
⚡ Context Cancellation	In-flight builds are killed the moment a new file event arrives
🛡️ Build-Error Preservation	Old server stays alive when a build fails — a syntax error never takes you down
🔄 Atomic Binary Replacement	New binary is built to a temp file, then os.Renamed into place — no half-written state
🧠 Crash Loop Detection	Exponential back-off (1s → 2s → 4s → … → 60s) + Alertmanager webhook
📊 Prometheus Metrics	Custom registry with 6 metrics — builds, durations, restarts, crash loops, file events
📝 Structured JSON Logs	Every event emits a typed slog field — Loki/Grafana-compatible out of the box
🎛️ Rich TUI	Bubbletea dashboard with 5 states: BUILDING, RUNNING, CRASHED, BACK-OFF, BUILD_ERR
🔒 Lockfile	PID file prevents two instances watching the same directory — with actionable kill instructions
📡 Signal Handling	SIGTERM/SIGINT = graceful shutdown, SIGHUP = config reload without restart
📄 YAML Config	hotreload.yaml — no flags needed in CI or team repos
🧪 Dry-run Mode	--dry-run prints resolved config and exits — no processes started
🏷️ Version Info	hotreload version shows version, commit, and build date from ldflags
🚪 Exit Codes	0 = clean, 1 = build fail, 2 = config error, 3 = runtime — scripts and CI depend on these
📦 Pre-built Binaries	GoReleaser publishes Linux/macOS/arm64 binaries on every Git tag

---

Architecture

hotreload is a pipeline. Each stage is independently cancellable.

FileSystem → Watcher → Debouncer → Orchestrator ──► Builder ──► Runner
                                        │                          │
                                   (cancel prev)          (crash detector)
                                        │                          │
                                   Prometheus ◄── metrics ─────────┘
                                   Alertmanager ◄─ webhook (crash loop)

---

Installation

Pre-built binary (recommended)

# Linux amd64
curl -L https://github.com/mihir-dixit2k27/hotreload/releases/latest/download/hotreload_Linux_x86_64.tar.gz | tar xz
sudo mv hotreload /usr/local/bin/

From source

go install github.com/mihir-dixit2k27/hotreload@latest

Build locally

git clone https://github.com/mihir-dixit2k27/hotreload
cd hotreload
make install

---

Quick Start

hotreload --build "go build -o tmp/server ./cmd/server" --exec "./tmp/server"

That's it. hotreload watches your project, rebuilds on every save, and keeps the old server running if the build fails.

See the full CLI reference below for all flags.

---

Config File (hotreload.yaml)

Drop a hotreload.yaml in your project root — no flags needed day-to-day.
Priority: CLI flags > hotreload.yaml > built-in defaults.

root: ./
build: "go build -o tmp/server ./cmd/server"
exec: "./tmp/server"

exts: [".go", ".mod", ".html", ".tmpl"]

ignore:
  - .git
  - node_modules
  - "*.test.go"
  - "*.pb.go"
  - tmp/

debounce_ms: 300
metrics: true
metrics_port: 2112
log_json: false          # true for Loki / containers
tui: true

crash_loop:
  threshold: 5           # crashes in window before back-off
  window_seconds: 30
  max_backoff: 60        # cap on exponential growth

alertmanager:
  webhook_url: ""        # set to http://localhost:9093 to enable

See hotreload.yaml.example for a fully documented template.

---

Observability

Prometheus Metrics

Metric	Type	Labels	Description
hotreload_builds_total	Counter	status="success|failure"	Total builds by outcome
hotreload_build_duration_seconds	Histogram	—	Build time distribution in seconds
hotreload_process_restarts_total	Counter	—	Total server process restarts
hotreload_crash_loop_backoff_seconds	Gauge	—	Current back-off duration in seconds (0 = healthy)
hotreload_file_events_total	Counter	—	Raw file-change events received
hotreload_debounced_events_total	Counter	—	Debounced build triggers fired

Structured Log Events

Every event carries a typed event field so you can filter in Loki without regex. Enable with --log-json=true.

{"level":"INFO","event":"startup","root":"./","build_cmd":"go build -o tmp/main ."}
{"level":"INFO","event":"file_change","file":"handlers/user.go","op":"WRITE"}
{"level":"INFO","event":"build_start","build_num":3,"build_cmd":"go build -o tmp/main ."}
{"level":"INFO","event":"build_success","build_num":3,"elapsed_ms":801}
{"level":"INFO","event":"server_start","run_cmd":"./tmp/main"}
{"level":"WARN","event":"server_crash","run_cmd":"./tmp/main","run_duration_ms":12}
{"level":"WARN","event":"crash_loop_detected","crashes_in_window":3,"backoff_seconds":5}

---

Design Decisions

• Process Groups: When a backend server starts, it may spawn child processes. Sending SIGKILL to only the parent PID orphans children, causing "port already in use" on restart. By assigning the server to its own process group and signalling the whole group, all descendants are cleaned up cleanly.
• Context Cancellation: exec.CommandContext delegates cancellation straight to the OS, instantly killing the compiler and freeing CPU for the next build — no polling loop required.
• Build-Error Preservation: Some tools kill your running process before attempting the new build — meaning a syntax error takes your local server down. hotreload only replaces the binary after confirming the new build succeeded.
• Custom Prometheus Registry: Using prometheus.NewRegistry() instead of the default global registry means hotreload's metrics don't pollute the default registry if the package is ever embedded as a library.
• Atomic Binary Replacement: Building to a temp file and calling os.Rename makes the swap atomic at the filesystem level. The old binary is never in a half-written state.
• PID Verification: The lockfile verifies /proc/<pid>/comm on POSIX systems to guard against PID recycling false positives, preventing duplicate watchers on the same directory.
• Exponential Back-off: A fixed retry delay wastes time on transient errors and hammers the CPU on persistent ones. Exponential back-off (1s → 60s) adapts automatically.

---

Compared to Alternatives

	hotreload	air	nodemon
Language	Go	Go	Node.js / any
Build-error preservation	✅ old server stays alive	❌ kills before build	❌ N/A
Atomic binary replacement	✅ os.Rename	❌	❌
TUI dashboard	✅ Bubbletea (5 states)	✅ Basic color output	❌
Prometheus /metrics	✅ 6 metrics	❌	❌
Structured JSON logs	✅ slog + typed event	❌ plaintext	❌ plaintext
Alertmanager webhook	✅ on crash loop	❌	❌
Crash loop back-off	✅ exponential 1s → 60s	❌	❌
SIGHUP config reload	✅ no restart needed	❌	❌
Lockfile (duplicate prevention)	✅ PID file	❌	❌
Config file	✅ hotreload.yaml	✅ .air.toml	✅ nodemon.json

---

CLI Reference

Usage:
  hotreload [flags]
  hotreload version

Flags:
  --config            Path to config file (default: hotreload.yaml)
  --root              Root directory to watch (default: .)
  --build             Build command (default: "go build -o tmp/main .")
  --exec              Run command (default: "./tmp/main")
  --exts              Comma-separated file extensions to watch
  --debounce-ms       Debounce window in ms, 50–5000 (default: 300)
  --tui               Enable TUI dashboard (default: true)
  --log-json          Emit JSON logs (default: false)
  --metrics           Enable Prometheus /metrics endpoint (default: true)
  --metrics-port      Prometheus metrics port, 1024–65535 (default: 2112)
  --alertmanager-url  Alertmanager base URL for crash-loop alerts
  --dry-run           Print resolved config and exit without starting
  -h, --help          Show help

Subcommands:
  version             Print version, commit hash, and build date

Signals:
  SIGINT / SIGTERM    Graceful shutdown (kills server process group)
  SIGHUP              Reload hotreload.yaml without restarting

---

Project Structure

cmd/hotreload/main.go       → CLI entrypoint, flag parsing, signal handling
internal/
  config/                    → YAML loading, flag merging, validation
  watcher/                   → Recursive fsnotify watcher + ignore rules
  debouncer/                 → Channel-based event coalescing
  builder/                   → exec.CommandContext + atomic os.Rename
  runner/                    → Process group management (POSIX + Windows stubs)
  orchestrator/              → Build-error preservation pipeline
  crashloop/                 → Sliding window detector + exponential back-off
  metrics/                   → Custom Prometheus registry, 6 metrics
  alertmanager/              → Alertmanager v2 webhook on crash loop
  lockfile/                  → PID file with stale-lock detection
  tui/                       → Bubbletea TUI with 5 states
testserver/                  → Dummy Go server with Chaos Mode demo

---

Development

# Run all tests with race detector
make test

# Run golangci-lint
make lint

# Build binary to ./bin/hotreload
make build

# Run demo (testserver with TUI)
make demo

---

License

MIT © Mihir Dixit