demos: standalone system-services-monitor demo

demos/system-services-monitor/Dockerfile

@@ -0,0 +1,21 @@
+FROM python:3.13-slim
+
+# nsenter is in util-linux (already in slim), but ensure it's available
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    util-linux \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY config.py metrics.py monitor.py ./
+COPY checks/ ./checks/
+
+# nsenter requires root to enter host namespaces.
+# The DaemonSet securityContext controls the actual privilege level.
+
+EXPOSE 9101
+
+ENTRYPOINT ["python", "monitor.py"]

demos/system-services-monitor/README.md

@@ -0,0 +1,76 @@
+# System Service Monitor — Standalone Demo
+
+A standalone DaemonSet companion to NVSentinel that catches GPU infrastructure failures invisible to DCGM-based monitoring.
+
+**Related issue:** [#883 - NVSentinel not detecting fabric health on H100s](https://github.com/NVIDIA/NVSentinel/issues/883)
+
+## Problem
+
+NVIDIA Fabric Manager can fail and stay broken for weeks undetected. NVSentinel's existing monitors (DCGM-based, syslog-based) miss it because individual GPUs appear healthy to DCGM even when Fabric Manager is down. This tool fills the gap with service-level health checks.
+
+**Requirements:** Kubernetes cluster with GPU nodes, Prometheus Operator
+
+## What It Monitors
+
+| # | Check | What It Catches | Method |
+|---|-------|-----------------|--------|
+| 1 | **Fabric Manager Service** | FM not running, flapping, error state | `nsenter` + `systemctl` |
+| 2 | **Critical GPU Services** | persistenced dead | `nsenter` + `systemctl` |
+| 3 | **Per-GPU Fabric State** | FM_NOT_STARTED, FM_REGISTRATION_STUCK, FM_FABRIC_ERROR | `nsenter` + `nvidia-smi` |
+| 4 | **CUDA Validation** | Context failures, memory errors | PyTorch subprocess |
+
+## Quick Start
+
+```bash
+# Build
+docker build -t system-services-monitor:latest .
+
+# Deploy (assumes nvsentinel namespace exists)
+kubectl apply -f k8s/rbac.yaml
+kubectl apply -f k8s/configmap.yaml
+kubectl apply -f k8s/daemonset.yaml
+kubectl apply -f k8s/servicemonitor.yaml
+
+# Verify
+kubectl get ds -n nvsentinel system-services-monitor
+
+# Port-forward to a specific node's pod
+NODE=<node-name>
+POD=$(kubectl get pod -n nvsentinel -o wide --field-selector spec.nodeName=${NODE} \
+  -l app=system-services-monitor -o jsonpath='{.items[0].metadata.name}')
+kubectl port-forward -n nvsentinel pod/${POD} 9101:9101
+curl -s localhost:9101/metrics | grep fabric_manager_up
+```
+
+## Metrics
+
+Exposed on port 9101. Key metrics:
+
+| Metric | Description |
+|--------|-------------|
+| `fabric_manager_up` | Fabric Manager running (1/0) |
+| `gpu_node_health_up` | Overall node health (1/0) |
+| `nvidia_service_up` | Per-service status |
+| `fabric_state_healthy` | Per-GPU fabric state (1/0) |
+
+## Alert Rules
+
+The ServiceMonitor includes PrometheusRule with alerts:
+- `FabricManagerDown` (critical, 5m)
+- `FabricManagerFlapping` (warning, 5m)
+- `FabricStateUnhealthy` (critical, 5m) -- per-GPU fabric orchestration failure
+- `GPUServiceDown` (critical, 3m)
+- `CUDAValidationFailed` (critical, 5m)
+
+## Validated On
+
+- 2x P4d.24xlarge (8x A100-SXM4-40GB each) -- Amazon Linux 2023, EKS 1.32
+- All check categories produce correct metrics
+
+## Configuration
+
+All settings via ConfigMap environment variables. See `k8s/configmap.yaml`.
+
+## Relationship to NVSentinel
+
+This is a **standalone companion tool** that exposes Prometheus metrics and alerts. It does not integrate with NVSentinel's gRPC event pipeline or remediation workflow. See the native `health-monitors/system-services-monitor/` for an integrated version that emits HealthEvents to platform-connector.

demos/system-services-monitor/checks/__init__.py

@@ -0,0 +1,15 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Health check modules for GPU Node Health Validator."""

demos/system-services-monitor/checks/cuda_validation.py

@@ -0,0 +1,104 @@
+"""Check 5: CUDA validation — context creation and memory test.
+
+Runs a minimal CUDA test on each GPU: allocate memory, write a pattern,
+read back, verify. Optional P2P test copies data between GPU pairs.
+This check runs at a slower cadence (default 10 minutes) since it
+consumes GPU resources.
+"""
+
+import logging
+import subprocess
+import sys
+import textwrap
+from dataclasses import dataclass, field
+from typing import List, Optional
+
+logger = logging.getLogger(__name__)
+
+# Inline Python script executed as a subprocess so that a PyTorch import
+# failure doesn't crash the main monitor process.
+_CUDA_TEST_SCRIPT = textwrap.dedent("""\
+    import sys
+    import json
+
+    results = {"passed": True, "gpu_count": 0, "errors": []}
+
+    try:
+        import torch
+    except ImportError:
+        results["errors"].append("PyTorch not available")
+        results["passed"] = False
+        print(json.dumps(results))
+        sys.exit(0)
+
+    gpu_count = torch.cuda.device_count()
+    results["gpu_count"] = gpu_count
+
+    if gpu_count == 0:
+        results["errors"].append("No CUDA devices found")
+        results["passed"] = False
+        print(json.dumps(results))
+        sys.exit(0)
+
+    for i in range(gpu_count):
+        try:
+            torch.cuda.set_device(i)
+            # Allocate, write, read back, verify
+            t = torch.randn(1024, device="cuda")
+            assert t.sum().isfinite(), f"GPU {i}: non-finite sum"
+            del t
+            torch.cuda.empty_cache()
+        except Exception as e:
+            results["errors"].append(f"GPU {i}: {e}")
+            results["passed"] = False
+
+    print(json.dumps(results))
+""")
+
+
+@dataclass
+class CUDAValidationResult:
+    """Result of CUDA validation across all GPUs."""
+    passed: bool
+    gpu_count: int = 0
+    errors: List[str] = field(default_factory=list)
+    error: Optional[str] = None  # check-level error (couldn't run at all)
+
+
+class CUDAValidator:
+    """Validates CUDA context creation and memory on each GPU."""
+
+    def check(self) -> CUDAValidationResult:
+        """Run CUDA validation script as a subprocess."""
+        try:
+            result = subprocess.run(
+                [sys.executable, "-c", _CUDA_TEST_SCRIPT],
+                capture_output=True,
+                text=True,
+                timeout=120,  # generous timeout for multi-GPU test
+            )
+
+            if result.returncode != 0:
+                return CUDAValidationResult(
+                    passed=False,
+                    error=f"CUDA test script failed: {result.stderr.strip()}",
+                )
+
+            import json
+            data = json.loads(result.stdout.strip())
+            return CUDAValidationResult(
+                passed=data.get("passed", False),
+                gpu_count=data.get("gpu_count", 0),
+                errors=data.get("errors", []),
+            )
+
+        except subprocess.TimeoutExpired:
+            return CUDAValidationResult(
+                passed=False,
+                error="CUDA validation timed out",
+            )
+        except Exception as e:
+            return CUDAValidationResult(
+                passed=False,
+                error=str(e),
+            )