production diagnostics

Pre-Alpha. This page describes behavior that may change.

Symptom-based troubleshooting using Ze's built-in diagnostic commands. All commands work on gokrazy appliances without external Linux tools: no ss, no dmesg, no lsof, no dig, no tcpdump, no mtr.

Quick reference

Symptom	First command
BGP session won't establish	show tcp-check <peer-ip> 179
Path/routing issue	show traceroute <dest> or monitor traceroute <dest>
BGP session flapping	show system kernel-log level warning count 50
High CPU	show system profile cpu duration 10s
Memory leak	show system profile heap
FD exhaustion	show system file-descriptors summary
Goroutine leak	show system goroutines summary
DNS failure	show dns lookup <name> type A
Latency/reachability	monitor ping <target>
Process killed	show system kernel-log level err
Route/link/addr changes	monitor system netlink all
Packet-level debugging	show capture interface eth0 tcp port 179 count 10 format text

Diagnostic commands

TCP connectivity probe

show tcp-check <peer-ip> 179

If "refused": peer is not listening. If "timeout": firewall or routing issue.

Path tracing

show traceroute <peer-ip>
monitor traceroute <target>
monitor traceroute <target> | log | origin

show traceroute runs a single trace. monitor traceroute is an mtr-style continuous trace with per-hop loss and latency statistics. | log | origin annotates each hop with ASN names. | log | resolve adds reverse DNS hostnames.

Continuous ping

monitor ping <target>
monitor ping <target> interval 500ms

Shows live stats: Sent, Recv, Loss%, Last, Min, Avg, Max, StDev. Use | log for scrollback output suitable for correlation with other events.

Kernel log inspection

show system kernel-log level warning count 50
show system kernel-log level err

Replaces dmesg. Reads from /dev/kmsg on Linux.

CPU and memory profiling

show system profile cpu duration 10s
show system profile heap

Returns base64-encoded pprof data. Decode with go tool pprof.

Goroutine inspection

show system goroutines summary
show system goroutines blocked
show system goroutines full

A steadily increasing count indicates a leak. Large numbers stuck in "chan receive" or "select" with the same stack suggest leaked goroutines.

File descriptor usage

show system file-descriptors summary
show system file-descriptors detail

If total is close to soft-limit, the process is near exhaustion.

Socket state

show system sockets tcp state ESTABLISHED port 179
show system sockets

Replaces ss. Shows TCP/UDP socket state.

DNS resolution and cache

show dns lookup <hostname> type A
show dns cache stats
show dns cache list
show dns cache record <hostname>
clear dns cache

Cache listing shows all entries with remaining TTL. Selective delete and full flush are available.

Live packet capture

show capture interface eth0 tcp port 179 count 10 format text
show capture interface eth0 tcp port 179 duration 10s format pcap

Replaces tcpdump. Uses AF_PACKET with BPF filters. Output in text or pcap format.

Netlink monitoring

monitor system netlink all
monitor system netlink link
monitor system netlink route

Streams live kernel events as JSON. Replaces ip monitor. Useful for observing route, link, and address changes in real time.

Process memory map

show system memory
show system memory-map

Reads from /proc/self/status. Compare VmRSS with Go heap-in-use.

System metrics

show system cpu
show system conntrack

Platform detection

show system platform reports the runtime platform type and capability flags.

show system platform
show system platform | json

Detected platforms: gokrazy, systemd, container, plain-linux, darwin.

Capability flags: read-only-root, perm-available, systemd-available, gokrazy-update-socket, gokrazy-ui-available, reboot-allowed, persistent-storage-writable, fd-limit-soft-current, fd-limit-hard-max, fd-limit-raisable.

The same platform information feeds ze doctor checks (e.g. gokrazy /perm writability) and is included in ze support archives as the platform module.

Platform notes

Commands that read /proc (sockets, kernel-log, file-descriptors, memory-map) are Linux-only. On other platforms they return "not available on this platform". The remaining commands (tcp-check, traceroute, goroutines, dns, profile) work on all platforms but traceroute requires CAP_NET_RAW.

See also

• Health Checks for the health registry and report bus.
• Monitoring for live event streams and Prometheus.
• Command Reference for the full command list.

Adapted from main/docs/guide/production-diagnostics.md.