Skip to content

README.md

Latest commit

 

History

History
829 lines (648 loc) · 47.2 KB

README.md

File metadata and controls

829 lines (648 loc) · 47.2 KB
Raw

FreeAI

version tests python cuda release docker coverage

The world's most complete self-hosted AI workstation.
Local models. Autonomous agents. Full SDLC automation.
One-line spec → full project. One URL → all clients.

FreeAI Dashboard FreeAI UI

5-Minute Quick Start — pick one:

# Bare metal (recommended, auto-detects 4090/Z790 and configures)
curl -fsSL https://raw.githubusercontent.com/ProjectZeroDays/FreeAI_Ubuntu-AI-Inference-Workstation/main/hardware/install-stack.sh | bash
# Or: SETUP_MODE=autonomous sudo ./hardware/install-stack.sh

# Docker (any host with NVIDIA Docker)
docker compose up -d --build

# Live ISO (no install)
# Download freeaios-amd64.iso from Releases → boot → Install FreeAI

Full docs: https://projectzerodays.github.io/FreeAI_Ubuntu-AI-Inference-Workstation/
Try before you clone: MOCK_LLM=1 runs the entire stack without a GPU (make test is fully offline).

Production-grade, self-hosted AI inference workstation stack: GGUF coder models on NVIDIA GPUs, a task-classifying model router with fallback chains, a multi-agent REST layer, a workflow engine, autonomous SDLC agents that turn a one-line spec into a packaged project, a presets/settings control plane, an AI power optimizer, self-healing watchdogs, and a full XFCE + VNC remote desktop. Deployable bare-metal, via Docker Compose, Kubernetes, cloud GPU providers, or as a Live ISO.

Table of Contents

1. Overview

The stack answers one question: how do I run capable coding models on my own GPU, 24/7, with agents that actually ship work - and reach it from anywhere?

  • One URL for every client. The FreeAI Router classifies each prompt (with a confidence score), routes to the best healthy backend, falls back automatically, caches repeats, and blocks repetition-loop garbage before it reaches you.
  • Agents with teeth. The Agent API exposes scaffolding/refactor/debug/analysis/chat personas with temperature profiles and session memory. The Workflow Engine chains them into pipelines. The Autonomous SDLC layer goes further: plan -> code -> verify with real compilers/tests -> fix -> review -> document -> package a tarball.
  • Ops that run themselves. Watchdogs restart dead services in seconds. An AI resource optimizer watches GPU temperature/utilization and shifts between performance/balanced/eco power profiles. Daily cleanup rotates logs and prunes old workspaces; weekly backups snapshot your config.
  • Everything is one settings file. The dashboard writes config/runtime-settings.json; the optimizer, autonomous API, router, and llama launcher all consume it on their own cadence. Change once, propagates everywhere.
  • Runs where you want. Same code on bare-metal Ubuntu, Docker Compose (split or all-in-one), Kubernetes, Vast.ai/RunPod/Lambda/Hetzner, or a bootable Live ISO.

2. Screenshots

FreeAIOS GRUB Boot Menu
Boot menu
Rendered preview of the ISO's boot menu (GRUB) — live/build-live.sh — Install / Try Live / Rescue
freeai-cli freeai-cli providers + live test
CLI CLI providers
Real --help output: 14 subcommands Real providers listing + provider-test openai
FreeAI Dashboard - active load
Dashboard active
74% util, alerts panel, service badges, settings + presets
FreeAI UI UI in use (refactor via moe-13b)
FreeAI UI UI output
Model presets + agent picker + prompt console Router response: model_used, task_type, confidence, elapsed_ms
Workflow Designer External Providers panel
Designer Providers panel
3-step pipeline (architecture -> codegen -> tests) w/ step config 21+ hosted APIs as backends: KEYED/NO KEY badges, fallback flags, Test pings
FreeAI UI - 8-model roster picker
8-model roster picker
Dashboard - model shelf
Model shelf
Full Dashboard (every panel)
Dashboard full
Alerts, GPU, services, clients, providers, runs, files, settings, model shelf
Dashboard - timed idle window
Dashboard idle
Eco enforced (6% util, 198W/2400MHz), idle banner w/ auto-restore countdown
FreeToken Desktop Console
FreeToken Desktop
Console UI — edge runtime on RTX 30/40/50

Dashboard shots use sample telemetry; on a live box the same panels stream real nvidia-smi data, router metrics, and idle-window state. FreeToken desktop image from FlashML-org/FreeToken (Apache-2.0).

3. Feature Set Breakdown

Subsystem Highlights
Router (:8010) Keyword classifier w/ confidence score - fallback chain across the roster - degenerate-output (repetition loop) detection w/ automatic retry - LRU response cache (X-Cache: HIT/MISS) - per-client token-bucket rate limiting (429) - optional X-API-Key auth - /metrics (counts, per-task/model, avg latency) - mock mode (MOCK_LLM=1) for GPU-less dev
Agent API (:8020) project / refactor / debug / analyze / orchestrate / chat endpoints - profiles: strict (t0.0) balanced (t0.2) creative (t0.8) verbose (4096 tok) minimal (512 tok) - session memory w/ inspect + clear - error envelopes - call counters
Workflow Engine (:8040) registry-based pipelines - sequential + parallel steps - 3-attempt retry per step - missing-dependency validation - JSONL audit log - export/import definitions - inline execution - 4 shipped templates
SDLC Runs panel dashboard view of autonomous runs: status badges, run IDs, specs, 15s refresh
Autonomous SDLC (:8050) 7-phase lifecycle (plan/coding/testing/fixing/documenting/packaging) - real verification: compileall, pytest->unittest, node --check inside sandboxed workspace - static placeholder scan fallback - artifact tarball download - run cancel - concurrency cap
Presets & Settings 4 recommended presets + named custom presets (CRUD) - timed idle window w/ auto-restore (survives restarts) - one runtime-settings.json consumed live by 5 services
Dashboard (:8030) GPU util/mem/temp/power/clock + Chart.js history - alerts (services down, thermal, util) - service UP/DOWN badges - settings panel - preset picker - idle countdown banner - model shelf (registry vs disk + free GB) - router metrics - SSE live updates - security headers
Optimizer + Tune performance/balanced/eco power modes w/ hysteresis + 10-min cooldown - gpu-power-tune.sh power cap + clock lock (-10..20C) - nvidia-persistenced enablement
Self-healing supervisor 10s loop - health agent 30s - recovery agent 15s - systemd units w/ auto-restart
Maintenance daily cleanup timer (log rotation 25MB x5, workspace pruning) - weekly backup timer (config/registry/manifests + run manifests, keep 10, restore mode)
Parallel hot models second resident shard (--profile llama2, per-GPU CUDA_VISIBLE_DEVICES) - /admin/hot-models roster + /admin/model-switch zero-reload swap
RAG Qdrant sidecar :6333 + ingest watcher (--profile rag) - MiniLM 384-dim embeddings, deterministic hash fallback for GPU-less CI
Evals golden-task harness (evals/golden_tasks.json) - run_eval.py scores router answers vs expectations incl. reviewer-model pass
Tooling freeai-cli (14 subcommands) - Makefile - MkDocs site - auto-docs generator - GitHub Actions CI (py/bash/js/json gates) + docker publish + release bundling
Desktop XFCE + TigerVNC + noVNC (compose --profile desktop)

Why FreeAI vs Alternatives

Feature FreeAI LM Studio OpenWebUI Ollama vLLM LocalAI TextGen WebUI
Multi-backend (llama.cpp + vLLM + FreeToken MoE 290B+) ✔ ✖ ✖ ✖ ✔ ✖ ✖
Router w/ fallback chains + degenerate retry + cache ✔ ✖ ✖ ✖ ✖ ✖ ✖
Autonomous SDLC (plan→code→test→fix→doc→package) ✔ ✖ ✖ ✖ ✖ ✖ ✖
Workflow engine + designer ✔ ✖ ✖ ✖ ✖ ✖ ✖
Full desktop OS (XFCE+VNC/noVNC) + Live ISO ✔ ✖ ✖ ✖ ✖ ✖ ✖
GPU optimizer (eco/balanced/perf) + watchdogs ✔ ✖ ✖ ✖ ✖ ✖ ✖
300+ providers (OpenAI/Anthropic/Google/OpenRouter/venice/agnes/zen/HF) ✔ ✖ ✖ ✖ ✖ ✖ ✖
Red/Blue/Purple autonomous teaming ✔ ✖ ✖ ✖ ✖ ✖ ✖
One-command bare-metal/cloud/K8s/ISO deploy ✔ ✖ ✖ ✖ ✖ ✖ ✖

FreeAI is the only stack that ships all of the above in one repo.

Use Cases

  • One-line spec → shippable project: freeai.py auto-start "Build a FastAPI notes service with tests" --watch
  • Multi-model fallback: Local 9B Q6_K → cloud GPT-4o → Venice uncensored → FreeToken 290B MoE, auto
  • 24/7 workstation: Full remote XFCE desktop via noVNC (6080) + dashboard (8030) from anywhere (TailScale/Cloudflare)
  • Enterprise RAG: Qdrant + ingest watcher + top-K cache for codebase/docs automation
  • Autonomous teaming: Red attacks, Blue hardens, Purple validates and bridges to JIRA-ready tickets

Beginner Mode

New to self-hosting? Start here:

# 1. One model, one chat — no router complexity
docker compose up -d llama dashboard
# Open http://localhost:8030 → Chat with freeai/qwen3.6-12b-heretic

# 2. When ready, enable the full stack:
docker compose --profile rag --profile desktop up -d
python scripts/autoconfigure.py --autonomous  # auto-tunes for your GPU

4. Architecture

FreeAI UI (ui/)        Workflow Designer (workflow/ui/)      freeai-cli
        |                        |                               |
        v                        v                               v
   Agent API (:8020) <---- Workflow Engine (:8040)      Autonomous SDLC (:8050)
        |                        |                          plan->code->verify
        v                        v                          ->fix->document->package
   Router (:8010) <---------------------------+
   classify -> confidence -> fallback chain
   cache / rate-limit / auth / metrics
        |
        v
   llama.cpp (:9001 GGUF CUDA)        vLLM (:9002, optional profile)

Dashboard (:8030) polls nvidia-smi + service ports + router metrics,
raises alerts, writes runtime-settings.json consumed by all services.

Chat request flow: client -> POST /route -> classifier scores task -> fallback chain tries backends in order -> degenerate check (repetition loop? try next) -> cache store -> response w/ model_used, task_type, confidence, elapsed_ms.

Autonomous run flow: spec -> plan JSON -> per-task code blocks (=== FILE: path ===) -> sandboxed writes -> verifier (shell tools or static scan) -> fix loop fed real errors -> reviewer verdict -> docs generation -> tarball artifact -> download via API/CLI.

Settings propagation: dashboard writes config/runtime-settings.json -> optimizer reacts within 60s, autonomous API per-request, router at restart, llama launcher via config/llama.env on "Save + restart llama". See docs/ARCHITECTURE.md for the full table.

5. Ports and Services

Port Service Exposed by default
8010 FreeAI Router (/route, /models, /metrics, /health) LAN only (UFW blocks)
8020 Agent REST API LAN only
8030 Dashboard UI + /api/* Yes (UFW allow)
8040 Workflow Engine LAN only
8050 Autonomous SDLC API Yes (UFW allow)
8011 Router WebSocket (ws://:8011/ws/route token streaming) LAN only
9001 llama.cpp server (--jinja) localhost/tailnet only
9002 vLLM (optional profile) LAN only
8888 JupyterLab (--profile jupyter / clients-provision) LAN only
3000 / 5000 OpenCode / ZCode (clients-provision) LAN only
8443 Caddy TLS gateway (--profile tls) - dashboard + basic-auth /auto/* optional public
9100 FreeToken edge MoE (--profile freetoken) - 290B+ on RTX 30/40/50 auto-fallback when healthy
9600 LoLLMs chat UI (--profile lollms) LAN only
6333/6334 Qdrant RAG vector DB (--profile rag) LAN only
5901 / 6080 VNC / noVNC desktop via --profile desktop

All ports overridable: ROUTER_PORT, AGENT_API_PORT, DASHBOARD_PORT, WORKFLOW_PORT, AUTONOMOUS_PORT, LLAMA_PORT, VLLM_PORT.

6. Getting Started

Fastest paths (details in Section 8):

Bare metal (Ubuntu 24.04 + NVIDIA):

git clone https://github.com/ProjectZeroDays/FreeAI_Ubuntu-AI-Inference-Workstation.git
cd FreeAI_Ubuntu-AI-Inference-Workstation
sudo ./hardware/install-stack.sh          # drivers->CUDA->Docker->stack->systemd->UFW
bash models/auto-download-models.sh       # ~15GB of GGUFs, resumable

Docker (split services):

docker compose up -d --build
docker compose --profile vllm up -d       # optional second backend

Docker (all-in-one):

docker compose --profile allinone up -d   # supervisord runs everything in one CUDA container

No GPU? Dev mode:

python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt
MOCK_LLM=1 python3 router/router.py       # canned completions, full API surface

Verify: python3 freeai.py status - open http://localhost:8030.

Deployment readiness matrix

Track Status Entry point Verified by
Installed as a system (bare metal) Ready now sudo ./hardware/install-stack.sh E2E: provisioner chains drivers -> CUDA -> Docker -> llama.cpp CUDA build -> models -> 6 systemd units -> UFW; install.sh --check drift report; make smoke on-box
Docker Compose (split) Ready now docker compose up -d --build (+ vllm / warmup / desktop / jupyter / tls profiles) healthchecks on every service; scripts/smoke-test.sh -> ALL_SYSTEMS_OPERATIONAL
Docker all-in-one Ready now docker compose --profile allinone up -d --build supervisord per-service restarts; CUDA 13 base (driver >= 580)
Kubernetes Ready now kubectl apply -f k8s/ manifests + HPAs shipped; images from CI (see blocker)
Cloud: Lambda / Hetzner / Paperspace Ready now deploy.ps1 -Hostname <ip> or SSH + installer same bare-metal path, drivers preinstalled on these hosts
Cloud: Vast.ai Ready now import vastai/template.json (Portal + Selkies + Guacamole) vastai/onstart.sh -> bundle -> provision -> clients -> desktop
Cloud: RunPod / any Docker host Ready now (build locally) docker compose --profile allinone up -d --build GHCR prebuilt image pending CI (blocker below)
Live ISO (FreeAIOS) Builder ready - compile on any Ubuntu host live/build-live.sh (Subiquity autoinstall remaster) Boot menu: Install FreeAI (wipes disk, stack first-boot) / Try Live / Rescue; needs one build run + network for NVIDIA driver

Known blocker (GitHub-side): the account billing lock stops ALL Actions jobs from starting (no GHCR images, no release bundles, no artifact builds). Workarounds until cleared: docker compose build locally, or SSH + install-stack.sh. Everything else above works today; clearing billing at github.com -> Settings -> Billing turns on docker-publish, release, and auto-release with zero repo changes.

7. Hardware Requirements

Tier GPU VRAM RAM Storage What runs
Floor 8 GB (RTX 3060 Ti / 4060) 32 GB 500 GB SSD Subset of roster Q4_K (9B-class), short ctx, 1-2 agents
Recommended 16 GB (RTX 4070 Ti SUPER / 4080) 64 GB DDR5-6000 1 TB OS + 2 TB models Full 8-model roster Q6_K on disk w/ instant hot-swap (/admin/model-switch; --profile llama2 keeps a 2nd shard resident), full SDLC loops 24/7
Headroom 24 GB (RTX 4090 / 3090) 96-128 GB +4 TB models Larger coders + MTP speculative decoding, vLLM coexistence

Verified parts list (MPN/ASIN): hardware/PARTS-LIST.md - assembly: hardware/BUILD.md - build-vs-cloud economics: hardware/LOCAL-DEPLOY.md.

8. Install and Deploy Handbook

8.1 Bare Metal Provisioner

hardware/install-stack.sh is idempotent and chains: base packages -> NVIDIA driver (570-server) -> CUDA toolkit (nvcc for source builds) -> Docker -> stack venv + llama.cpp CUDA build -> model downloads -> systemd units (core, watchdogs, gpu-tune, optimizer, cleanup + backup timers) -> UFW (22/8030/8050 only) -> unattended security upgrades + NTP.

sudo ./hardware/install-stack.sh                 # full provisioning
NO_START=1 sudo ./hardware/install-stack.sh      # install without starting

Reboot once after driver install, then systemctl status freeai-stack.

8.2 Docker Compose Profiles

Command Starts
docker compose up -d --build llama, router, agents, workflow, autonomous, dashboard
--profile vllm + vLLM :9002 (prefix caching on)
--profile allinone single supervisord CUDA container, every service
--profile warmup one-shot GPU warmup after healthy
--profile desktop + XFCE/VNC/noVNC
--profile jupyter + JupyterLab :8888
--profile tls + Caddy TLS gateway :8443
--profile lollms + LoLLMs chat UI :9600 (points at router)
--profile freetoken + FreeToken edge MoE engine :9100 (290B+ models on consumer GPUs)

Config via .env (copy .env.example). Every core service has a compose healthcheck; warmup waits on router + llama health.

8.3 Kubernetes

kubectl apply -f k8s/namespace.yml
kubectl apply -f k8s/models-pvc.yml     # model storage first
kubectl apply -f k8s/

GPU nodeSelector + tolerations (llama/vLLM), HPAs on router/agents/workflow (CPU 70%, 1-4). Images published to GHCR by the docker-publish workflow on tags.

8.4 GPU Providers

Provider Path
Vast.ai template env PROVISIONING_SCRIPT=<release bundle URL>; onstart fetch + run pipeline
RunPod Docker template from GHCR all-in-one image; mount /models
Lambda / Paperspace bare Ubuntu + sudo ./hardware/install-stack.sh
Hetzner GPU / OVH same as Lambda; UFW included
AWS g5/g6, Azure NC, GCP G2 (spot) Terraform module (roadmap); eco optimizer shines here

Details: docs/DEPLOYMENT-PLANS.md - fallback modes (local router -> cloud inference, or full cloud stack) + spot resilience + firewall/fail2ban hardening: docs/CLOUD-FALLBACK.md.

8.5 Live ISO

live/build-live.sh builds FreeAIOS via live-build: GRUB menu - Try Live (RAM) / Install to disk (autoinstall) / Rescue shell - bundled NVIDIA drivers, GPU-detect first boot (MOCK fallback in VMs), optional casper persistence. Plan: docs/DEPLOYMENT-PLANS.md Track C.

9. Tool Handbook

9.1 FreeAI Router

  • Classification: keyword hits -> (task_type, confidence): full_project / refactor / analysis / general_code (0.5).
  • Fallback chain: per-task primary + alternates; per-agent overrides via AGENT_MODEL_OVERRIDES JSON or config.
  • Degenerate guard: tail-period repetition detector; loops trigger next backend (X-Coherence-Retries header).
  • Cache: LRU 128, key = task+agent+prompt hash.
  • Rate limit: token bucket per IP (60 cap, 60/min refill).
  • Auth: ROUTER_API_KEY set -> X-API-Key required (except /health).
  • llama sampling backstop: --repeat-penalty 1.05 --repeat-last-n 64 server defaults.

9.2 Agent API

Personas: project, refactor, debug, analyze, orchestrate, chat. Profiles: strict/balanced/creative/verbose/minimal. Session memory: 20 turns x 100 sessions (LRU).

curl -X POST localhost:8020/agent/chat -H "Content-Type: application/json" \
  -d '{"message":"Design a rate limiter","session_id":"s1"}'
curl localhost:8020/memory/s1 && curl localhost:8020/profiles

9.3 Workflow Engine

Registry: project_pipeline, full_build, api_build, microservice_build. Steps declare consumes/produces; validation flags dangling deps; 3-attempt retry per step; JSONL audit at logs/workflow-audit.jsonl. Designer at workflow/ui/designer.html exports JSON -> POST /workflow/run-inline; export via GET /workflow/export/{name}.

9.4 Autonomous SDLC Agents

Lifecycle: queued -> planning -> coding -> testing <-> fixing -> reviewing -> documenting -> packaging -> done | failed | cancelled.

  • Workspace workspaces/<run_id>/: traversal/absolute/drive paths rejected, 512KB/file cap.
  • Verification: real commands in-workspace when shell enabled (ENABLE_SHELL_TOOLS=1 + per-run enable_shell); else static placeholder scan.
  • Fix loop: up to 3 rounds fed actual compiler/test output.
  • Artifact: _artifact.tar.gz via API or freeai.py auto-fetch.
  • Guard: max_concurrent_runs -> 429 over cap.

9.5 Resource Optimizer and GPU Tune

Samples nvidia-smi every 60s (3-sample window, hysteresis, 10-min cooldown):

Mode Power Clock Trigger
performance stock stock util >= 85 pct and temp <= 75C
balanced 240W 2520MHz steady state
eco 200W 2400MHz temp >= 82C or util <= 10 pct

Publishes config/runtime-state.json (dashboard shows live mode). Manual override: uncheck AI auto-management, pick mode, Save. Undervolt profile: hardware/gpu-power-tune.sh apply|reset|status (power cap + clock lock, -10..20C).

9.6 Dashboard

Panels: Alerts (services down, thermal, util thresholds) - GPU stats with Chart.js utilization history - Services UP/DOWN badges - Settings (preset dropdown, custom save/delete, idle timer, auto-management checkbox, power/clock caps, llama sampling, concurrency cap) - Model shelf (registry vs on-disk GGUFs + free GB) - router metrics embed - SSE /api/events pushes settings changes to every open tab.

9.7 freeai-cli

python3 freeai.py status                       # health + router metrics
python3 freeai.py models                       # roster
python3 freeai.py route "Build an API" --profile strict
python3 freeai.py workflows && python3 freeai.py run full_build --context {"spec":"..."}
python3 freeai.py auto-start "FastAPI notes service" --watch 20
python3 freeai.py auto-fetch <run_id> -o out.tar.gz
python3 freeai.py presets                      # recommended + custom
python3 freeai.py preset "Silent Eco"          # apply
python3 freeai.py preset "Idle (timed)" --idle 45
python3 freeai.py settings get auto_management
python3 freeai.py settings set max_concurrent_runs 2

9.8 Watchdogs and systemd Units

Unit Role
freeai-stack.service start.sh: all services, Restart=on-failure
freeai-agents.service health-agent (30s) + recovery-agent (15s) via run-watchdogs.sh
gpu-tune.service applies eco power/clock at boot, resets on stop
resource-optimizer.service the AI optimizer loop
freeai-cleanup.timer daily: rotate logs 25MB x5, prune workspaces >7d
freeai-backup.timer weekly: config/registry/manifests snapshot, keep 10

9.9 Backup and Cleanup Maintenance

bash scripts/backup.sh                # snapshot now (backups/backup-TS.tar.gz)
bash scripts/backup.sh list
bash scripts/backup.sh restore backups/backup-XXXX.tar.gz
WORKSPACE_RETENTION_DAYS=14 bash scripts/cleanup.sh

Backups cover config/, registry/, manifest/, VERSION and every workspaces/*/​_run.json run manifest.

9.10 External AI Providers

Bridge turns 21+ hosted APIs into router backends with the same response shape as local GGUFs. Three adapters: openai (most hosts), anthropic (native messages), gemini (generateContent).

export OPENAI_API_KEY=sk-... && export GROQ_API_KEY=gsk_...
curl -X POST localhost:8010/route -H "Content-Type: application/json" \
  -d '{"prompt":"Design a rate limiter","model":"openai/gpt-4o-mini"}'
  • Explicit model: "provider/model" wins over the local chain
  • fallback: true providers (config/providers.json) become last-resort after all local GGUFs - GPU down, hosted models take over
  • Dashboard External AI Providers panel: keyed badges + live Test pings; CLI: freeai.py providers / provider-test groq
  • Streaming: openai-style = true SSE; anthropic/gemini = single-frame

Full per-provider setup (env vars, model slugs, custom/Azure/vLLM entries, cost guardrails): docs/PROVIDERS.md.

Local FreeToken (edge MoE) in 30s: docker compose --profile freetoken up -d -> curl http://localhost:9100/v1/models -> model: "freetoken/deepseek-ai/DeepSeek-V4-Flash" routes automatically as fallback when healthy. Native: uv pip install "freetoken[accel]" && freetoken serve --model deepseek-ai/DeepSeek-V4-Flash. See detailed steps in docs/PROVIDERS.md#FreeToken-Local-Setup.

10. Configuration Reference

Layering: defaults < config/config.json < config/providers.json (external backends) < config/runtime-settings.json < environment variables.

config/config.json

Section Keys
router port, api_key, rate_limit_capacity, rate_limit_refill_per_min, cache_enabled, cache_size, backend_timeout_s, mock_llm, model_overrides
agents port, default_profile, memory_max_turns
workflow port, audit_log, step_retries, retry_delay_s
dashboard port, gpu_temp_alert_c (85), gpu_util_alert_pct (90)

config/runtime-settings.json (dashboard Settings panel)

Key Default Consumed by
auto_management true optimizer (checkbox: AI owns power modes)
forced_mode balanced optimizer when auto off
power_limit_w / locked_clock_mhz 240 / 2520 optimizer + gpu tune
eco_power_w / eco_clock_mhz 200 / 2400 idle + eco windows
repeat_penalty / repeat_last_n 1.05 / 64 llama launcher (via llama.env + restart)
llama_ctx 4096 llama launcher (restart)
max_concurrent_runs 3 autonomous API (live, 429 over cap)
idle - timed idle window state + restore snapshot

Key environment variables

Env Used by Default
ROUTER_API_KEY router auth empty (off)
RATE_LIMIT_CAPACITY / _REFILL_PER_MIN router 60 / 60
CACHE_ENABLED / CACHE_SIZE router true / 128
MOCK_LLM router false
LLAMA_BASE router http://localhost:9001
AGENT_MODEL_OVERRIDES switcher {}
AGENT_API workflow + autonomous http://localhost:8020
ENABLE_SHELL_TOOLS autonomous false
MAX_FIX_ROUNDS / SHELL_TIMEOUT_S autonomous 3 / 120
OPTIMIZER_INTERVAL_S / COOLDOWN_S optimizer 60 / 600
GPU_ID / GPU_POWER_LIMIT_W / GPU_LOCKED_CLOCK_MHZ tune 0 / 240 / 2520
WORKSPACES_DIR / MAX_FILE_BYTES autonomous repo/workspaces / 524288

11. API Reference

Condensed; full curl examples in docs/API.md.

Router :8010

Method Path Notes
GET /health liveness + mock flag
GET /models roster: name/role/strengths/endpoint
POST /route {prompt, max_tokens?, temperature?, agent?} -> {model_used, task_type, confidence, elapsed_ms, response}; headers X-Cache, X-Coherence-Retries
GET /metrics counters, per-task/model, latency_avg_ms

Agent API :8020

Method Path
POST /agent/project {spec, profile?, session_id?}
POST /agent/refactor {code, language?, goals?}
POST /agent/debug {code, error, language?}
POST /agent/analyze {context, question}
POST /agent/orchestrate {prompt, agent_hint?}
POST /agent/chat {message, session_id}
GET/DELETE /memory/{session_id}
GET /profiles, /metrics, /health

Workflow Engine :8040

Method Path
GET /workflows, /health
POST /workflow/run {workflow, context, strict_validation?}
POST /workflow/run-inline {definition}
GET /workflow/export/{name}
POST /workflow/validate {steps:[...]}

Autonomous SDLC :8050

Method Path
POST /auto/start {spec, profile?, max_tasks?, enable_shell?} -> run_id (429 over cap)
GET /auto/runs, /auto/runs/{id}
POST /auto/runs/{id}/cancel
GET /auto/runs/{id}/artifact (tar.gz)
POST /auto/runs/{id}/shell {command} (guarded)

Dashboard :8030

Method Path
GET /api/status (gpu, services, alerts, power_mode, router_metrics, version)
GET/POST /api/settings ; POST /api/settings/llama-restart
GET/POST/DELETE /api/presets[/{name}] ; POST /api/presets/{name}/apply {duration_min?}
GET /api/models-status ; GET /api/runs (autonomous runs proxy) ; GET /api/events (SSE)

12. Model Management

Roster (registry/registry.json):

Key GGUF Role
qwen3.6-12b Qwen3.6 12B IQ Ultra Heretic Uncensored Thinking (GGUF) Primary coder - architecture, full projects, CI/CD
claude-code-9b CodeClawd - Qwen3.5 9B Claude Code (empero-ai) Code specialist - SFT on Claude Code + Codex agent traces, <think> + <tool_call>
qwythos-v2 Qwythos 9B v2 (empero-ai) Reasoning primary - Claude-trace CoT, looping fixed (FTPO), 1M context, vision, MTP
qwythos-9b Qwythos 9B Claude Mythos 5 1M (empero-ai) Reasoning (v1 fallback) - 1M context, vision, function calling
qwable-9b Qwable 9B Claude Fable 5 (empero-ai) General assistant - Claude Fable 5 + GPT-5.5 terminal-agent distill, multimodal
qwen3.5-thinking Qwen3.5 9B Claude 4.6 HighIQ THINKING Heretic (mradermacher i1) Reasoning fallback - thinking variant, imatrix quants
qwen3.5-9b Qwen3.5 9B Claude HighIQ Heretic Uncensored (GGUF) Legacy fallback
moe-13b L3.1 MOE 2x8B DeepSeek DeepHermes e32 Abliterated 13.7B (GGUF) Fast coder - refactor, debug, patch

Reasoning models (qwythos-*, claude-code-9b, qwen3.5-thinking): replies open with a <think> block; the router clamps temperature to each model's floor (0.6) automatically. Qwythos v2 fixes v1's repetition loops at the model level (FTPO). MTP speculative decoding: DOWNLOAD_MTP=1 + LLAMA_EXTRA_ARGS="--spec-type draft-mtp --spec-draft-n-max 6". 1M-context models: raise LLAMA_CTX (KV memory grows linearly).

openNemo-9B (abliterated / Claude-Opus-4.6-distill): safetensors-only (Nemotron-H hybrid Mamba2). Convert locally with bash scripts/convert-hf.sh empero-ai/openNemo-9B-abliterated Q4_K_M, then add a registry entry.

Note: llama.cpp serves one hot model per process;

13. Custom Integrations

  • OpenCode / JCode / MimoCode: manifests ship in mimocode/ (opencode.json, jcode.json, mimocode.json + *-models.json). Point the provider base_url at the router (:8010) or llama direct (:9001) - model ids match the registry. Recommended host analysis + MCP plan: docs/CODEX-INTEGRATION.md.
  • Hermes CLI: hermes config set model.provider custom; model.base_url https://<host>:8010; model.api_key <ROUTER_API_KEY>.
  • Any OpenAI-ish client: llama.cpp (:9001) and vLLM (:9002) speak /v1/chat/completions natively.
  • MCP (roadmap): server wrapper over /route + /agent/* + /workflow so Codex/OpenCode-class clients consume the stack natively.
  • Webhooks (idea): POST /auto/start from GitHub Issues label; run-status webhook on done/failed.
  • Custom agents: subclass pattern in agents/ - a function that builds a prompt + calls call_router(); expose via agents/api.py and add a workflow Step.

14. Remote Access

./hardware/setup-remote-access.sh tailscale    # private mesh, --ssh enabled
./hardware/setup-remote-access.sh cloudflare   # cloudflared + tunnel steps
./hardware/setup-remote-access.sh both

UFW opens only 22/8030/8050. Router (8010) and llama (9001) stay off the public internet by design; reach them via tailnet. noVNC desktop: compose --profile desktop, port 6080.

Public HTTPS: --profile tls serves https://<host>:8443 (internal cert by default). For a real domain with automatic ACME certs: set FREEAI_DOMAIN + ACME_EMAIL in .env, swap the caddy volume to docker/Caddyfile.public, and publish 80/443. The autonomous API is reachable through the gateway behind basic auth (AUTOAUTH_USER/AUTOAUTH_HASH - bcrypt via docker run --rm caddy caddy hash-password); dashboard settings writes are blocked at the gateway (LAN/tailnet only).

15. Testing and Validation

make test     # 88-test offline pytest suite
make lint     # bash -n + py_compile + node --check + json.tool over tracked files
python evals/run_eval.py   # golden-task eval sweep (needs router up; MOCK_LLM=1 works)

Suite map: router unit (classifier/switcher/cache/limiter) - router API via Flask test client (mock backend) - coherence (degenerate detector + retry) - agents (profiles/memory/metrics via TestClient) - workflow (validation/retries/definitions) - autonomous SDLC (full lifecycle, fix loop, cancellation, sandbox safety) - optimizer (mode decisions) - presets (CRUD/apply/idle expiry/cap). Golden-task evals live outside pytest: they score real router output quality per task class.

CI (.github/workflows): ci.yml (compile/syntax/JSON gates) - workflow-ci.yml (offline smoke) - docs.yml (auto-generate workflows.json) - docker-publish.yml (5 images to GHCR on tags) - release.yml (source bundle on tags) - local-build.yml (image tarball artifacts) - iso-build.yml (FreeAIOS ISO remaster on dispatch/tags; ISO artifact + <2GiB split parts on release).

Note: if Actions shows failures reading "account is locked due to a billing issue", no job starts at all (even local-build.yml) - that is GitHub-side billing; resolve at github.com -> Settings -> Billing and every workflow resumes automatically with zero repo changes.

16. Performance Tuning Guide

  1. Power first: gpu-tune 240W/2520MHz loses ~3-5% throughput for -10..20C; let the optimizer ride performance only under real load.
  2. Context: LLAMA_CTX up to 16-32K if RAM allows (KV cache grows linearly); pair with llama.cpp KV quant flags via LLAMA_EXTRA_ARGS.
  3. Offload: N_GPU_LAYERS=80 default; drop for smaller cards.
  4. Speculative decoding: opt-in via LLAMA_EXTRA_ARGS=--model-draft <small.gguf> --draft-max 16 - validate output coherence before keeping.
  5. Concurrency: max_concurrent_runs caps GPU thrash; 2-3 is the sweet spot on 16GB.
  6. Cache: leave CACHE_ENABLED on; prompts with identical task+agent+text return instantly.
  7. vLLM: enable only when you need HF-hosted models concurrently - it owns VRAM; prefix caching is on.

16b. Context Window Presets (Auto-Tuning Profiles)

Three preset levels tie context size to VRAM budget, KV-cache strategy, GPU layers, router weights, hot-model pool, SDLC concurrency, and RAG chunking. Machine-readable below: a settings-dashboard client (or AI agent) can parse this block and apply every parameter programmatically via POST /api/settings + LLAMA_EXTRA_ARGS.

context_presets:

  # --------------------------------
  # LEVEL 1 - 16K CONTEXT (stable + fast)
  # --------------------------------
  level_1_16k:
    description: "Stable + fast preset for 9B models on RTX 4090."
    ctx_size: 16384
    gpu_layers: 40
    kv_cache: "full_vram"
    batch_size: 64
    offload: false
    recommended_models: [qwen3.5-9b, qwen3.5-thinking-9b, claude-code-9b, qwythos-9b, qwythos-v2, qwable-9b]
    router_adjustments: {reasoning_weight: 0.5, coding_weight: 0.7, heavy_model_weight: 0.0}
    rag: {chunk_size: 768, chunk_overlap: 96, top_k: 8}
    sdlc: {max_parallel_agents: 4}

  # --------------------------------
  # LEVEL 2 - 32K CONTEXT (high capacity)
  # --------------------------------
  level_2_32k:
    description: "High-capacity preset for large documents, RAG, and long coding sessions."
    ctx_size: 32768
    gpu_layers: 36
    kv_cache: "expanded_vram"
    batch_size: 48
    offload: false
    recommended_models: [qwen3.5-9b, qwen3.5-thinking-9b, qwen3.6-12b, moe-13b]
    router_adjustments: {reasoning_weight: 0.6, coding_weight: 0.6, heavy_model_weight: 0.2}
    rag: {chunk_size: 1024, chunk_overlap: 128, top_k: 10}
    sdlc: {max_parallel_agents: 3}

  # --------------------------------
  # LEVEL 3 - 64K CONTEXT (experimental)
  # --------------------------------
  level_3_64k:
    description: "Ultra-long contexts. Reduced GPU layers for stability; hybrid KV offload to RAM."
    ctx_size: 65536
    gpu_layers: 28
    kv_cache: "hybrid_offload"
    batch_size: 16
    offload: true
    recommended_models: [qwen3.5-thinking-9b, moe-13b]
    router_adjustments: {reasoning_weight: 0.7, coding_weight: 0.4, heavy_model_weight: 0.4}
    rag: {chunk_size: 1536, chunk_overlap: 192, top_k: 12}
    sdlc: {max_parallel_agents: 2}

llama_cpp_flags:
  level_1_16k: ["--ctx-size 16384", "--gpu-layers 40", "--flash-attn", "--batch-size 64"]
  level_2_32k: ["--ctx-size 32768", "--gpu-layers 36", "--flash-attn", "--batch-size 48"]
  level_3_64k: ["--ctx-size 65536", "--gpu-layers 28", "--flash-attn", "--batch-size 16", "--no-kv-offload"]

hot_model_pool:
  level_1_16k: [qwen3.5-9b, qwen3.5-thinking-9b, claude-code-9b]
  level_2_32k: [qwen3.5-9b, qwen3.5-thinking-9b, moe-13b]
  level_3_64k: [qwen3.5-thinking-9b, moe-13b]

router_weights:
  level_1_16k: {code: 0.7, reasoning: 0.5, heavy: 0.0}
  level_2_32k: {code: 0.6, reasoning: 0.6, heavy: 0.2}
  level_3_64k: {code: 0.4, reasoning: 0.7, heavy: 0.4}

agent_personas:
  level_1_16k: {coder: {max_tokens: 2048}, architect: {max_tokens: 4096}, reviewer: {max_tokens: 2048}}
  level_2_32k: {coder: {max_tokens: 4096}, architect: {max_tokens: 8192}, reviewer: {max_tokens: 4096}}
  level_3_64k: {coder: {max_tokens: 8192}, architect: {max_tokens: 16384}, reviewer: {max_tokens: 8192}}

rag_config:
  level_1_16k: {chunk_size: 768, chunk_overlap: 96, top_k: 8}
  level_2_32k: {chunk_size: 1024, chunk_overlap: 128, top_k: 10}
  level_3_64k: {chunk_size: 1536, chunk_overlap: 192, top_k: 12}

Model ids match registry/registry.json (§12). moe-13b is the L3.1 2×8B MoE — Mixtral-class routing, 2 experts active per token. Apply a level by: settings panel (LLAMA_CTX + max_concurrent_runs)

  • LLAMA_EXTRA_ARGS flags above + registry role weights; the 1M-ctx Qwythos models ignore these ceilings (raise LLAMA_CTX directly).

17. Troubleshooting

Symptom Fix
llama-server not found run ./install.sh; binary at llama.cpp/build/bin/
CUDA build skipped install CUDA toolkit (nvcc on PATH), re-run installer
Download aborts: disk preflight needs size+10GB; free space or move models dir
Port already in use at start stop the other stack, or ALLOW_PORT_REUSE=1
Router 502 all backends unhealthy; check logs/llama.log, :9001/health
401 from router ROUTER_API_KEY set - send X-API-Key
429 rate limited raise RATE_LIMIT_CAPACITY/_REFILL_PER_MIN
Repetition loops in output router auto-retries next model; if persistent lower temperature, verify --jinja active, refresh llama.cpp (make update-llama)
Settings changed, router unchanged rate/cache/timeout apply on router restart
Idle window stuck resource-optimizer service down? systemctl status resource-optimizer
Workflow step fails 3x inspect logs/workflow-audit.jsonl for agent + error
Dashboard zeros nvidia-smi missing/no GPU in container
Actions startup failures: billing github.com -> Settings -> Billing (account lock), then re-run

More: docs/TROUBLESHOOTING.md.

18. Security Notes

  • Only 22/8030/8050 exposed by UFW; router + model servers private/tailnet.
  • Router API keys optional but recommended off-LAN: ROUTER_API_KEY + X-API-Key.
  • Rate limiting per-IP on every route; health endpoint exempt.
  • Autonomous sandbox: path-traversal/absolute/drive-letter rejection, 512KB file cap, shell double-gated (server env + per-run flag), command timeouts, output caps.
  • Dashboard: security headers (nosniff, frame-deny, referrer-policy); settings writes validated + bounded.
  • Secrets: ROUTER_API_KEY lives in .env / systemd env - never committed (gitignored); CI scans for stray secrets.
  • Generated code runs in workspace sandbox; treat artifacts as untrusted until reviewed.

19. Roadmap and Future Implementations

Full matrix: ROADMAP.md. Headliners:

  • Codex-class integration epic: MCP server wrapper, approval profiles (suggest/auto/full-auto), diff-based surgical edits, OS-sandbox runner (bwrap/nspawn, network-off), git-native runs - see docs/CODEX-INTEGRATION.md + docs/GAP-ANALYSIS-CODEX.md
  • Distribution: all-inone GHCR image in CI, FreeAIOS Live ISO v0.1 (build script shipped), provider kits (RunPod template, spot-cloud Terraform)
  • Ops: Prometheus exporter, off-site backup sync (rclone), dashboard auth on write endpoints
  • Agents: prompt-template library, run artifacts panel, two-model review gate, token accounting
  • Models: performance scoring, registry UI

20. Documentation Index

Doc Contents
docs/API.md full endpoint reference + curl
docs/ARCHITECTURE.md diagrams, request flows, settings interconnection
docs/MODEL-SWITCHING.md classifier -> chain -> overrides tuning guide
docs/AUTONOMOUS-AGENTS.md SDLC lifecycle, safety model, API/CLI
docs/DEPLOYMENT.md bare-metal / compose / k8s / profiles / dev mode
docs/TROUBLESHOOTING.md symptom -> fix table
docs/ENHANCEMENT-PLAN.md shipped vs planned matrix
docs/CODEX-INTEGRATION.md OpenCode vs JCode host choice, Codex feature port map
docs/GAP-ANALYSIS-CODEX.md capability gap matrix vs Codex
docs/DEPLOYMENT-PLANS.md Live ISO / all-in-one / provider rollout plans
docs/CLOUD-FALLBACK.md Vast/RunPod/Lambda/Hetzner/Paperspace/AWS/Azure/GCP - Mode A (cloud inference) + Mode B (full cloud) + spot resilience + hardening
docs/OPTIMIZATION-AUDIT.md high-impact scale/reliability audit: logging, config, watchdogs, model lifecycle, networking, RAG
docs/BUILD-SHEET.md workstation build (i9-14900KF/RTX 4090/128GB DDR5), GPU tier + model performance tables, power envelope
docs/FIRST-BOOT-GUIDE.md 10-step bring-up: BIOS -> Ubuntu -> CUDA -> installer -> registry -> services -> dashboard -> noVNC
hardware/PARTS-LIST.md verified workstation SKUs
hardware/BUILD.md assembly + Ubuntu install walkthrough
hardware/LOCAL-DEPLOY.md min requirements, build-vs-cloud economics
CHANGELOG.md release notes

21. Help and FAQ

  • Where do I ask? GitHub Issues on this repo - include freeai.py status output and relevant logs/ *.log tail.
  • No GPU, can I try? Yes: MOCK_LLM=1 runs the entire API surface with canned completions; pytest is fully offline.
  • Is my model garbage or is the stack? Check /metrics degenerate_skips - if climbing, the model loops; router already retries the next backend. Verify --jinja and quant tier first.
  • Windows? Development-friendly (tests run anywhere python does); serving targets Linux + NVIDIA. WSL2 works for API-only dev with MOCK_LLM.
  • Multiple GPUs? Point registry entries at per-GPU llama instances (different LLAMA_PORT + CUDA_VISIBLE_DEVICES); the fallback chain becomes a pool.
  • Upgrade llama.cpp? make update-llama (safe, rebuilds only).

22. Contributing and License

PRs welcome: keep the CI gates green (make lint && make test), match the existing ASCII-doc style, add tests for new behavior.

Ideas with the highest leverage right now (see ROADMAP): MCP server, approval profiles, diff edits, sandbox runner, Prometheus exporter.

License: MIT - see LICENSE.

23. Acknowledgments

FreeAI (the router, UI, workflow engine, and autonomous SDLC layer in this repo) is an original project — it is not a fork. It stands on these upstream projects:

Project Role here
llama.cpp GGUF inference engine — CUDA build, llama-server behind the router
vLLM High-throughput coexisting backend (:9002, prefix caching)
Qdrant Vector DB for the RAG sidecar (--profile rag)
LoLLMs Optional chat frontend (--profile lollms)
FreeToken Edge MoE serving engine (--profile freetoken) — Apache-2.0
TigerVNC / noVNC Desktop remote access layer
OpenCode Coding-client integration target (:3000)

Model credits: empero-ai (CodeClawd, Qwythos, Qwable distills), mradermacher (imatrix quants), Qwen team (base models). Quantization sanity depends on their upload hygiene — thank them.

The OLED dashboard/UI styling takes visual inspiration from FreeToken's console aesthetic; all code in this repo is original MIT-licensed work.