Add RAG-backed AI agents

.env.local.example

@@ -24,10 +24,13 @@ SUPABASE_SERVICE_ROLE_KEY=your-service-role-key
 # key — users have to re-save their WhatsApp settings to reconnect.
 ENCRYPTION_KEY=your-64-char-hex-key-here
 
-# Meta App Secret (Meta for Developers → App Settings → Basic).
-# Verifies the HMAC-SHA256 signature on every inbound webhook POST.
-# Required — without it the webhook rejects every request.
-META_APP_SECRET=your-meta-app-secret
+# Z-API instance credentials are saved per account in Settings > WhatsApp
+# and encrypted in the database. Optional values below are only for the
+# local readiness smoke check (`npm run zapi:check`); the app does not
+# use them for normal sending.
+# ZAPI_INSTANCE_ID=your-zapi-instance-id
+# ZAPI_INSTANCE_TOKEN=your-zapi-instance-token
+# ZAPI_CLIENT_TOKEN=your-zapi-client-token
 
 # ============================================================
 # RECOMMENDED — safe defaults exist but you'll want to set these.
@@ -42,9 +45,6 @@ META_APP_SECRET=your-meta-app-secret
 # or a background worker that has no incoming request.
 NEXT_PUBLIC_SITE_URL=https://crm.example.com
 
-# Default language locale (e.g. en)
-NEXT_PUBLIC_APP_LOCALE=en
-
 # ============================================================
 # OPTIONAL — only needed if you use the feature.
 # ============================================================
@@ -81,38 +81,21 @@ NEXT_PUBLIC_APP_LOCALE=en
 # See docs/automations-and-cron.md.
 # AUTOMATION_CRON_SECRET=generate-a-long-random-string
 
-# Meta App ID (Meta for Developers → App Settings → Basic). Required to
-# create/edit message templates with an IMAGE header: Meta only accepts a
-# Resumable-Upload media handle (not a plain URL) as the header sample, and
-# that upload is app-scoped. Without it, image-header template submission
-# returns a clear error; text/body-only templates are unaffected. Pair
-# with META_APP_SECRET.
-# META_APP_ID=your-meta-app-id
-
-# When "true", POST /api/whatsapp/templates/submit skips the Meta call
-# and stores the row with a synthetic `dry-run-<uuid>` meta_template_id.
-# Set this in CI and local development so you can exercise the full
-# template UI without a real WABA. Leave unset (or "false") in prod.
-# WHATSAPP_TEMPLATES_DRY_RUN=true
+# Distributed rate limiting (recommended for multi-instance deploys).
+# When these are set, src/lib/rate-limit.ts uses Upstash Redis over
+# REST with a fixed-window counter shared across all app instances.
+# When unset, the app falls back to the in-memory per-process limiter.
+# UPSTASH_REDIS_REST_URL=https://your-instance.upstash.io
+# UPSTASH_REDIS_REST_TOKEN=your-upstash-rest-token
 
-# ------------------------------------------------------------------
-# AI reply assistant (optional)
-# ------------------------------------------------------------------
-# The AI assistant is bring-your-own-key: each account pastes its own
-# OpenAI or Anthropic key under Settings → AI Assistant. The key is
-# stored AES-256-GCM-encrypted with ENCRYPTION_KEY (above) — there is
-# NO global provider key env var, and nothing here is required for the
-# feature to work. The two vars below only tune behaviour.
-#
-# The AI knowledge base (migration 030) uses Postgres full-text search
-# out of the box. Optional semantic search needs the `pgvector`
-# extension — migration 030 runs `CREATE EXTENSION IF NOT EXISTS vector`
-# (Supabase has it available) — plus a per-account embeddings key set in
-# Settings → AI Assistant. Still no env var required.
+# Optional rate-limit store tuning.
+# Timeout for the Upstash REST call in milliseconds. Default: 1500
+# RATE_LIMIT_STORE_TIMEOUT_MS=1500
 
-# Per-call timeout for provider requests, in milliseconds. Default 30000.
-# AI_REQUEST_TIMEOUT_MS=30000
+# Fallback if the external store errors or times out:
+# - memory (default): keep the old in-process limiter behavior
+# - deny: fail closed with 429 until the store recovers
+# RATE_LIMIT_FALLBACK_MODE=memory
 
-# How many recent text messages of a conversation to send the model as
-# context (draft + auto-reply). Default 20.
-# AI_CONTEXT_MESSAGE_LIMIT=20
+# Optional Redis key prefix for the shared buckets.
+# RATE_LIMIT_KEY_PREFIX=wacrm:rate-limit:v1

.gitignore

@@ -39,6 +39,9 @@ yarn-error.log*
 # vercel
 .vercel
 
+# git worktrees (subagent-driven-development isolation)
+/.worktrees/
+
 # typescript
 *.tsbuildinfo
 next-env.d.ts

CHANGELOG.md

@@ -9,6 +9,21 @@ Versions follow [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Pre-1.0, `MINOR` bumps cover new modules; `PATCH` bumps cover bug fixes
 and polish.
 
+## [0.9.0] — 2026-07-20
+
+Removes the AI reply assistant from the product.
+
+> **Migration required:** apply `supabase/migrations/037_drop_ai.sql`.
+> This migration is destructive: it drops AI configuration, knowledge
+> base, usage, message badge, and conversation auto-reply data.
+
+### Removed
+
+- **AI Agents and AI reply assistant.** Removed the `/agents` section,
+  `/api/ai/*` routes, inbox draft button, auto-reply handoff banner,
+  AI message badge, provider configuration, knowledge base UI, and the
+  internal AI runtime.
+
 ## [0.8.1] — 2026-07-10
 
 Fixes inbound chats fragmenting into multiple threads for the same

README.md

@@ -33,13 +33,6 @@ clone or fork it to run your own CRM.
 - **No-code automations** — triggers on inbound messages, new
   contacts, keywords, or schedule; conditional branches, waits,
   tags, webhooks. Visual builder.
-- **AI reply assistant** — bring your own OpenAI or Anthropic key
-  (stored encrypted; no per-seat AI fee, your data stays yours).
-  One-click AI-drafted replies in the inbox, plus an optional
-  auto-reply bot with a per-conversation cap and clean human handoff.
-  Add a **knowledge base** (FAQs, policies, product docs) and it
-  answers from your own content — hybrid retrieval (Postgres full-text,
-  or semantic pgvector when an embeddings key is set).
 - **Real-time dashboard** — response times, daily volume, pipeline
   value, cross-module activity feed.
 - **Team accounts** — invite teammates by link, role-based access

docs/ai-rag-agents-implementation.md

@@ -0,0 +1,155 @@
+# RAG, Agents, and Multi-Agent Implementation
+
+## Architecture Summary
+
+The current `wacrm` AI surface should evolve incrementally instead of adding a second AI subsystem. The repository already has an account-scoped BYOK AI configuration, provider-agnostic structured JSON generation, a WhatsApp agent dispatcher, deterministic CRM actions, automation generation, and audit points for AI-originated pipeline moves.
+
+RAG should be restored as a new account-scoped knowledge layer because the previous AI knowledge schema was intentionally removed by `supabase/migrations/037_drop_ai.sql`. Multi-agent should be introduced as a registry of CRM-specific specialist roles behind one coordinator, not as independent long-running bots that can write to the CRM directly.
+
+## Recommended Architecture Style
+
+Use the existing modular Next.js + Supabase architecture:
+
+- Next.js App Router route handlers for dashboard APIs and admin actions.
+- Supabase Postgres with RLS and service-role execution for webhook/background paths.
+- Provider-agnostic AI calls through `src/lib/ai/generate-json.ts`.
+- Deterministic action execution through existing CRM helpers.
+- Account-scoped knowledge retrieval injected into the current single-decision agent flow.
+
+This keeps the product closer to ByteChef and Dify in capability while preserving the smaller operational footprint of the current app.
+
+## Recommended Stack
+
+| Layer | Choice | Reason |
+|---|---|---|
+| AI calls | Existing `generateJson<T>()` + focused provider helpers | Matches the current no-SDK pattern and keeps BYOK simple. |
+| RAG storage | Supabase Postgres, FTS, pgvector | The repo already targets Supabase and previously used this shape successfully. |
+| Ingestion jobs | Postgres job table + route-triggered processing first | Enough for MVP; a durable queue can be added after real volume appears. |
+| Agent orchestration | Coordinator + specialist registry | Gives multi-agent behavior without runaway tool loops. |
+| Tools/actions | Server-side registry of allowed CRM actions | Lets models propose actions while the backend validates and executes. |
+| UI | Settings tabs and existing builder surfaces | Avoids reviving the deleted standalone `/agents` area prematurely. |
+| Observability | `ai_runs`, `ai_tool_calls`, `ai_retrieval_events` | Required to debug cost, quality, retrieval misses, and unsafe decisions. |
+
+## Existing Local Assets
+
+- `src/lib/ai/config.ts`: loads and saves encrypted per-account provider config.
+- `src/lib/ai/generate-json.ts`: provider-agnostic structured-output call.
+- `src/lib/ai/agent-context.ts`: loads recent conversation and linked deal context.
+- `src/lib/ai/agent-decide.ts`: asks the model for one sanitized decision.
+- `src/lib/ai/agent-dispatch.ts`: executes reply, tags, stage move, and handoff.
+- `src/lib/ai/automation-generate.ts`: creates sanitized automation drafts.
+- `src/lib/automations/resources.ts`: loads account-scoped tags, pipelines, and stages.
+- `supabase/migrations/038_ai_agent.sql`: current AI config and stage-move audit schema.
+- `supabase/migrations/039_ai_reply_cap_rpc.sql`: atomic reply-cap claim.
+
+## External Patterns Applied
+
+| Project | Pattern to reuse | Adaptation for `wacrm` |
+|---|---|---|
+| Inkeep Agents | Visual builder plus SDK/config style, subagents, MCP credentials, traces | Store agent definitions as account-scoped config and keep code-first defaults. |
+| LobeHub | Agents as first-class workspace resources, editable memory, plugins | Create named CRM agent roles with editable instructions and knowledge scope. |
+| Multica | Agent lifecycle, runtime state, squads, MCP overlay | Add explicit agent runs and a coordinator, but avoid external runtime daemons for MVP. |
+| Dafthunk | Visual workflows, async execution, run monitoring | Reuse existing Automations/Flows and add AI nodes later. |
+| Dify | RAG pipeline, datasets, model providers, observability | Implement knowledge ingestion, retrieval events, and model-neutral generation. |
+| ByteChef | Agents inside workflows, workflows as tools, guardrails, unified audit | Make CRM workflows callable tools only through deterministic backend actions. |
+
+## Target Components
+
+### Knowledge Base
+
+Account-owned documents, chunks, retrieval RPCs, ingestion jobs, and retrieval events. It should support a lexical path for every account and semantic retrieval when OpenAI-compatible embeddings are available.
+
+### Retrieval Layer
+
+`src/lib/ai/knowledge/retrieve.ts` should combine FTS and optional semantic results, dedupe chunks, normalize scores, and return short cited snippets. The agent prompt receives only bounded snippets, not entire documents.
+
+### Agent Registry
+
+Account-owned roles:
+
+- `coordinator`: chooses which specialist handles an event.
+- `triage`: classifies intent, urgency, and routing.
+- `support`: answers from knowledge snippets.
+- `sales`: qualifies lead and proposes pipeline movement.
+- `retention`: detects churn or complaint risk and recommends handoff.
+- `automation_builder`: drafts automations and later flows.
+
+### Action Registry
+
+The model can only emit action names and arguments. Server code validates every argument against account resources before executing:
+
+- `send_message`
+- `add_tag`
+- `remove_tag`
+- `move_deal_stage`
+- `assign_conversation`
+- `create_deal`
+- `create_automation_draft`
+- `create_followup_task`
+
+### Observability
+
+Add a trace model that can answer:
+
+- Which agent handled the event?
+- Which model and provider ran?
+- Which knowledge chunks were retrieved?
+- Which actions were proposed?
+- Which actions were executed, rejected, or skipped?
+- How many tokens were used?
+- Why did the system hand off to a human?
+
+## Data Flow
+
+```mermaid
+flowchart TD
+  A["Inbound WhatsApp message"] --> B["Store message and acknowledge webhook"]
+  B --> C["dispatchInboundToAgent"]
+  C --> D["Load AI config and CRM context"]
+  D --> E["Retrieve account knowledge"]
+  E --> F["Coordinator chooses specialist"]
+  F --> G["Specialist returns structured decision"]
+  G --> H["Validate IDs and action arguments"]
+  H --> I["Execute allowed CRM actions"]
+  I --> J["Log ai_run, tool calls, retrieval events"]
+```
+
+## Architecture Decisions
+
+| Decision | Reason | Trade-off |
+|---|---|---|
+| Restore RAG as new schema after migration 040 | `037_drop_ai.sql` dropped the old knowledge tables and RPCs. | Requires a fresh migration instead of reusing old tables. |
+| Keep one deterministic executor | Prevents model-generated writes from bypassing tenant and resource validation. | Less flexible than free-form tool loops. |
+| Start with coordinator + specialists | Provides multi-agent routing while bounding cost and complexity. | Specialists are configs and prompts first, not autonomous workers. |
+| Use lexical retrieval even without embeddings | Every account can use knowledge without a second key. | Lexical misses paraphrases more often. |
+| Add semantic retrieval only when embedding config exists | Avoids forcing Anthropic-only accounts into OpenAI embedding spend. | UI must explain degraded retrieval mode. |
+| Log AI runs before full dashboards | Debuggability is needed before analytics UI. | Initial release exposes logs mostly for engineering/admin inspection. |
+
+## Risks and Mitigations
+
+| Risk | Mitigation |
+|---|---|
+| Cross-account knowledge leakage | RLS, account-scoped RPCs, revoked `PUBLIC`, service-role callers always pass `account_id`. |
+| Hallucinated IDs or unsafe writes | Sanitizers blank/drop unknown IDs; action registry validates every argument. |
+| BYOK cost spikes | Existing rate limits plus per-agent max actions, reply caps, and usage logging. |
+| Slow WhatsApp webhook | Keep AI dispatch fire-and-forget and perform ingestion outside webhook response path. |
+| Bad RAG answer | Require citations, log retrieved chunks, hand off when confidence or evidence is missing. |
+| Agent recursion | No recursive agent calls in MVP; coordinator chooses one specialist per event. |
+
+## Implementation Order
+
+1. Add the RAG schema and retrieval RPCs.
+2. Add chunking, embeddings, ingestion, and retrieval modules with unit tests.
+3. Add account knowledge APIs and a small Settings UI.
+4. Inject retrieved knowledge into the current WhatsApp agent decision.
+5. Add `ai_runs`, `ai_tool_calls`, and retrieval logging.
+6. Add an agent registry with fixed specialist defaults.
+7. Add coordinator routing and per-specialist prompts.
+8. Expose safe internal actions as a registry for agent decisions.
+
+## Open Questions
+
+- Should embeddings reuse the account's OpenAI chat key when provider is OpenAI, or require a separate embeddings key?
+- Should knowledge management live under `Settings -> AI agent` first, or get its own `Settings -> Knowledge` section?
+- Should flows be generated by the automation copilot in the same release, or stay out of scope until RAG is proven?
+- Which users may inspect AI run logs: admin-only or agent-plus?