An offline-first AI assistant built on BitNet — runs on a R2,500 Raspberry Pi with no internet, no API costs, and no load shedding excuses.
Named after my daughter, born March 2026. Built so that wherever she grows up, the people around her have access to the same quality of information as anyone else.
Charlie is a fine-tuned BitNet-compatible model and inference framework designed to run completely offline on consumer hardware. It uses Microsoft's 1-bit ternary weight architecture via bitnet.cpp to achieve useful AI capability on hardware that costs a fraction of what cloud AI requires.
Model size: ~400MB
RAM required: ~1.1GB
Hardware: Core i5, Raspberry Pi 4/5 (8GB), any x86/ARM CPU
Internet: Not required
API costs: Zero
GPU: Not required
Charlie runs at ~16 tokens/second on a Core i5 and ~8-11 tokens/second on a Raspberry Pi 5. That is fast enough for natural conversation.
Most AI assistants assume you have:
- Fast, reliable internet
- A credit card for API billing in USD
- Hardware powerful enough to run large models
In South Africa — and most of the world — none of those assumptions hold reliably. Load shedding cuts power for 4-12 hours a day. Rural connectivity is unreliable or nonexistent. Cloud AI costs are denominated in dollars that most people don't have.
Charlie assumes none of those things. She runs on a device that costs R2,500, works during Stage 6 load shedding on a battery bank, and can be distributed on a USB stick.
| Component | Status |
|---|---|
| Base model (Falcon-Edge 1B, tool calling fine-tune) | ✅ v1 training complete |
| bitnet.cpp inference | ✅ Working on Core i5 WSL2 |
| Tool router (pre-LLM token optimisation) | ✅ Designed, testing |
| Markdown knowledge base (zero dependencies) | ✅ Built |
| Web learner (SearXNG + Crawl4AI → offline KB) | ✅ Built |
| Bootstrap primitives (safe self-improvement) | ✅ Designed |
| FastAPI OpenAI-compatible server | 🔧 In progress |
| Raspberry Pi deployment | 🔧 In progress |
| isiZulu language support | 📋 Planned v2 |
Charlie is built around four separating concerns:
SOUL.md → personality and tone (runtime, no retraining)
MODE.md → online/offline/community mode (runtime)
skills/ → capabilities (training time)
tools/ → what Charlie can do (runtime, MCP)
Instead of injecting all tool schemas into every prompt (OpenClaw-style token bloat), Charlie uses a pre-LLM router that selects only the 2-3 tools relevant to the current prompt:
User prompt
↓
Tool Router (keyword + semantic, <5ms)
↓
Inject only relevant tool schemas (~160 tokens vs ~800 tokens)
↓
Charlie LLM (3,800+ tokens available for conversation)
On a 4096 token context window this saves ~640 tokens per request — the difference between Charlie losing context after 5 exchanges and maintaining a full conversation.
Skills are composable training ingredients, not runtime plugins:
charlie/
├── skills/
│ ├── tool_calling/ ← when and how to call tools
│ ├── restraint/ ← when NOT to call tools
│ └── bootstrap/ ← how to propose new skills
├── tools/
│ ├── offline/ ← always available
│ │ ├── rag_search ← local markdown knowledge base
│ │ ├── calculator ← safe maths
│ │ ├── load_shedding ← Eskom schedule lookup
│ │ └── local_calendar
│ └── online/ ← available when connected
│ ├── searxng_search
│ ├── gmail (MCP)
│ ├── google_calendar (MCP)
│ └── web_learn ← SearXNG + Crawl4AI → saves to KB
├── serve/
│ ├── tool_router.py
│ ├── knowledge_base.py
│ ├── web_learner.py
│ └── bootstrap_primitives.py
├── SOUL.md
└── MODE.md
Charlie's knowledge base is plain markdown files. No vector database. No indexing pipeline. No dependencies.
# A community health worker adds local knowledge:
nano knowledge/local/local_clinic.md
# Charlie picks it up on next restartBM25 search with title and tag boosting. Runs on a Pi Zero. Anyone who can type can contribute.
When Charlie cannot accomplish a task she uses four MCP-wrapped primitive tools to research and propose new skills:
mcp_searxng → research topics
mcp_requests → fetch documentation
mcp_bash → read files, run scripts (allowlisted commands only)
write_file → save proposals to skills/proposed/ only
The safety model: Charlie proposes. Humans approve. Nothing auto-deploys. All proposals go to skills/proposed/ and require explicit human review before moving to skills/active/.
Charlie v1 is fine-tuned from tiiuae/Falcon-E-1B-Base (bfloat16 revision) using LoRA on tool-calling datasets.
| Validation run (v0) | Full run (v1) | |
|---|---|---|
| Dataset | 6,500 examples | 22,500 examples |
| Epochs | 1 | 2 |
| Max sequence length | 256 tokens | 384 tokens |
| Final loss | 0.856 | ~0.40 (estimated) |
| Smoke test | 1/3 | TBD |
HuggingFace:
- Merged model: jeff3c/charlie-falcon-edge-tool-calling-v1
- LoRA adapter: jeff3c/charlie-falcon-edge-adapter-v1
Eight walls have been hit and documented:
- Microsoft BitNet is not directly fine-tunable — quantized weights cannot store gradients. Use Falcon-Edge bfloat16 revision instead.
- Falcon-Edge revision confusion —
prequantized= inference only,bfloat16= training. This is not documented clearly anywhere. - Tokenizer mismatch causes loss of 9.8 — base model revisions ship bare tokenizers with no special tokens. Use the instruct tokenizer.
<tool_call>must be a single token — multi-token tool call tags are unreliable at 1B scale. Confirmed: token ID 21 for Falcon-Edge instruct tokenizer.- BitnetLinear + PEFT incompatible — LoRA cannot attach to custom quantized layers. Apply LoRA before
replace_linear_with_bitnet_linear(). - fp16 + BitnetLinear = grad scaler crash — BitnetLinear produces fp16 gradients that PyTorch's AMP scaler refuses to unscale.
- 256 token sequences truncate tool call closing tags — minimum 384 tokens needed to capture complete tool call format.
- Kaggle draft sessions don't run unattended — use "Save & Run All (Commit)" for overnight training runs.
Full findings: docs/findings.md
# Install bitnet.cpp
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
# Fix known build issue
sed -i 's/int8_t \* y_col = y + col \* by;/const int8_t * y_col = y + col * by;/' \
src/ggml-bitnet-mad.cpp
# Install clang and build
sudo apt install -y clang cmake build-essential
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
# Download Charlie v1
huggingface-cli download jeff3c/charlie-falcon-edge-tool-calling-v1 \
--local-dir ~/models/charlie-v1/
# Convert to GGUF
python convert-hf-to-gguf-bitnet.py \
~/models/charlie-v1/ \
--outtype i2_s \
--outfile ~/models/charlie-v1.gguf
# Run
python run_inference.py \
-m ~/models/charlie-v1.gguf \
-p "What meetings do I have tomorrow?" \
-cnv -t 2# Requires Kaggle account (free) or any GPU with 16GB VRAM
# Training script: finetune/kaggle_charlie_v1_full.py
# Key config:
BASE_MODEL = "tiiuae/Falcon-E-1B-Base"
MODEL_REVISION = "bfloat16" # NOT prequantized
TOKENIZER_MODEL = "tiiuae/Falcon-E-1B-Instruct" # NOT base
TOKENIZER_REVISION = "main"See finetune/README.md for the complete guide.
Edit SOUL.md to change personality without retraining:
# SOUL.md
name: Charlie
tone:
style: friendly # friendly | professional | clinical | encouraging
verbosity: concise
warmth: high
communication:
use_local_references: true # knows what load shedding is
use_humour: occasionallyEdit MODE.md to switch between online and offline modes:
# MODE.md
active_mode: offline_personal # offline_community | offline_personal | connected_personalDrop a markdown file in knowledge/:
nano knowledge/health/local_clinic.md# Manguzi Community Clinic
Open Monday to Friday, 7:30am to 4:30pm.
Services: antenatal care, TB treatment, immunisations.
Emergency: 035 592 0021
#health #clinic #localCharlie finds it on next restart. No indexing. No pipeline. Just a file.
v1 (current)
- Tool calling fine-tune on Falcon-Edge 1B
- bitnet.cpp inference on CPU
- Markdown knowledge base
- Tool router
v2
- FastAPI OpenAI-compatible server
- Raspberry Pi deployment guide
- Bootstrap skill (propose new capabilities)
- Restraint fine-tune (when NOT to call tools)
- Web learner (learn from internet, store offline)
v3
- isiZulu language support
- SMS interface (R200 USB GSM modem)
- Community kiosk deployment guide
- Request advice skill (consult frontier models, cache forever)
Charlie is built for a context most AI projects ignore. If you want to help:
- Add knowledge — drop a markdown file in
knowledge/for your domain - Add a language — isiZulu, Afrikaans, Sesotho data contributions welcome
- Test on hardware — results on Raspberry Pi, old laptops, ARM devices
- Add a skill — see
skills/tool_calling/SKILL.mdfor the format - Report findings — undocumented walls you hit are valuable contributions
All contributions go through skills/proposed/ — the same process Charlie uses for bootstrap. Humans review before anything merges.
- bitnet.cpp — Microsoft's 1-bit inference engine
- Falcon-Edge — TII's BitNet-compatible model family
- HuggingFace Transformers + PEFT + TRL
- Crawl4AI — web scraping to markdown
- SearXNG — self-hosted privacy-respecting search
Built with Claude (Anthropic) as a collaborative pair programmer over an extended development session. The architecture decisions, mission, and persistence through 8 consecutive build failures were human. The code generation and research assistance were AI. Both mattered.
MIT — do whatever you want with it, just keep it accessible.
Charlie is named after my daughter, who will be born in April 2026. Built so that the communities around her have access to the same information as anyone else.