feat(dev): add local model serving tool

[binaryaaron]

Summary

• Add a source-tree tool for compiling and managing local vLLM inference services from pinned TOML profiles.
• Support Hugging Face generation models and NVIDIA GLiNER/GLiNER2 through the pinned external vLLM Factory integration.
• Preserve pure compilation, digest-protected plans, capability probes, reconnectable receipts, PID-reuse protection, and bounded process-group cleanup.
• Ship seven profiles: NVIDIA GLiNER, GLiNER2, TinyLlama, GPT-OSS 20B and 120B, Qwen3 30B A3B Instruct, and NVIDIA Nemotron 3.5 Lightning.
• Add a local-model deployment guide and a checked-in NVIDIA GPU container image for running the same lifecycle in a container.

Simplified scope

The tool owns one deployment domain: a managed local-process vLLM server.

• Intent schema v2 has four sections: task, model, vllm, and local.
• Commands are compile, launch, probe, inspect, and cancel.
• Factory-backed detection and stock vLLM generation share the same compiler and lifecycle.
• Docker compiler/runtime branches, native Transformers GLiNER serving, and Hugging Face cache discovery were removed because no shipped profile used them.
• The supplied container runs the unchanged local-process lifecycle within a GPU container boundary; it does not add a second compiler path.
• The simplification removed 988 production lines from the hoster and reduced its dedicated test surface from 1,705 to 852 lines while retaining behavioral coverage for secrets, plan integrity, probes, launch failure cleanup, PID identity, TERM/KILL cleanup, LoRA, Factory constraints, and every shipped profile.

The tool remains under tools/ and is not part of the distributable Anonymizer package. It does not attach to existing endpoints or manage remote compute.

Validation

• make test: 1,226 passed.
• Focused inference-host suite: 35 passed, including vLLM Factory integration tests.
• make check: formatting, lint, type checks, lock consistency, and copyright checks passed.
• make docs-build: strict build passed.
• All seven v2 profiles compiled through the CLI.
• Built tools/inference_service.Dockerfile as nemo-anonymizer-local-models:pr212; the packaged CLI exposed all five lifecycle commands and the vLLM Factory compatibility doctor passed.
• A100 dogfood co-hosted pinned local GPT-OSS 20B and NVIDIA GLiNER services. The managed probes passed for chat completions and for dynamic labels, offsets, and scores.
• The full Anonymizer pipeline processed two local records with zero failed records, detected and redacted four PII entities, and preserved the no-PII record unchanged.
• Both services were inspected as running, canceled through their receipts with cleanup_complete: true, and left the A100 at 0 MiB used.
• Earlier branch validation also ran the 50-record repository benchmark with zero failed model requests or pipeline records and live-probed GPT-OSS 120B on the A100.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Post-rebase A100 dogfood completed on HEAD 1dd931ea8b7cf558c22b3cd2c2916d746492ca19 using the managed compiled plans and local endpoints for nvidia/gliner-pii plus pinned nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (33268dc8a6da85a56be2b12241453e4e1237bbe1).

• Full benchmark: 50 records across biographies redact+evaluate and legal hash+evaluate
• Both cases completed on their first attempt
• Model requests: 200 total, 200 successful, 0 failed
• Pipeline errors: 0 stage, 0 NDD workflow, 0 model workflow
• Replacement completeness: 0 missing final entities and 0 missing final values
• Managed shutdown: both services terminated with cleanup_complete: true
• Final A100 state: 0 MiB used

The analyzer reproduced the seven previously verified measurement false positives (1 biographies, 6 legal). No leak-metric changes are included in this hoster PR.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added pinned GPT-OSS 120B, GPT-OSS 20B, and Qwen3 30B A3B Instruct profiles on rebased HEAD 9d9536b990738e409e2ea335dd7d4b051dcd9c74. GPT-OSS 120B loaded successfully on the A100 through vLLM 0.27.1; the managed capability probe passed after accounting for reasoning-token output, and cleanup returned GPU use to 0 MiB. Fresh validation: 1,260 local tests passed, make check passed, strict docs build passed, and GitHub tests pass on Python 3.11, 3.12, and 3.13. NVSkills live evaluation is running.

[binaryaaron]

Medium hoster simplification is now on ccfedc8a4aeeb353d90bfc7692b4cc10a2c2c887.

• Removed Docker, native Transformers GLiNER, and cache-discovery branches.
• Compacted profiles to intent schema v2 with task, model, vllm, and local sections.
• Kept compile/launch/probe/inspect/cancel, plan digests, receipts, secrets, PID identity checks, bounded cleanup, LoRA, vLLM controls, and Factory-backed detection.
• Net hoster reduction: 988 production lines. Dedicated tests: 1,705 to 852 lines.
• Local validation: 1,226 tests passed, 35 focused inference-host tests passed, make check passed, and strict docs passed.
• A100 dogfood: local GPT-OSS 20B plus NVIDIA GLiNER passed managed probes; a two-record Anonymizer run found and redacted four PII entities with zero failed records; both receipt-owned services canceled cleanly; final GPU use was 0 MiB.

The implementation began as an Engineer Arc candidate. Parent review rejected its over-aggressive test reduction, restored lifecycle-focused behavioral coverage, removed a stray compiler branch, and reran all validation before push.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added the local-model deployment docs and checked-in GPU container image in commit 0034b8f. Validation: full suite 1,226 passed; make check passed; strict docs build passed; Docker image build passed; packaged CLI and vLLM Factory compatibility doctor passed inside the image.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Modern Python boundary refactor is now on HEAD ff40e2b.

• Closed FactoryPlugin typing now flows through intent, compiler, runtime, Factory integration, and adapter boundaries.
• Exhaustive match/assert_never dispatch covers owned task, environment, and plugin unions.
• Internal multi-field products are frozen slotted dataclasses; Factory metadata is one immutable exhaustive specification.
• CommandSpec owns redacted rendering, secret-source inventory, and strict execution-time resolution.
• HTTP client ownership is context-managed, compiler argument construction is more declarative, and obsolete delegation/test-loader seams are removed.
• CLI translation remains narrow while preserving exit 125 for known filesystem, transport, integrity, runtime, validation, TOML, and empty-source-revision failures.

A Terra implementation and independent-review cycle found and resolved plugin-spec duplication, secret ownership, positional chunk tuples, and stale delegation seams. Primary-agent review then caught the empty-source-revision exit regression and the remaining manual HTTP ownership flag before push.

Validation on the exact signed commit chain:

• focused inference-service tests: 36 passed, 3 skipped
• make check: passed, including ty
• make test: 1,227 passed, 3 skipped
• all four commits have DCO signoff and valid repository ED25519 signatures

No bundled skill files changed, so no new /nvskills-ci request is needed.