Fresh `npm install -g cavemem` ships broken local embedding provider — backfill stuck at 0%, silent failure

[chiefmojo]

Fresh npm install -g cavemem ships broken local embedding provider — backfill stuck at 0%, silent failure

Summary

A clean npm install -g cavemem (latest published version, 0.2.1) installs successfully and all CLI/hook wiring works, but the local embedding provider (embedding.provider: "local", default) fails at runtime with ERR_MODULE_NOT_FOUND for @xenova/transformers. The worker logs the error but keeps running and reports no obvious failure state — cavemem status shows backfill: 0 / N (0%) indefinitely with no indication why.

Environment

• OS: macOS (Darwin 25.5.0), Apple Silicon
• Node: v22.23.1 (also reproduced on v26.5.0 default install before switching)
• npm: 11.17.0
• cavemem version: 0.2.1 (latest on npm at time of filing)
• Install method: npm install -g cavemem
• IDEs wired: Claude Code, OpenCode, Codex (all three affected identically — same worker/embedder)

Steps to reproduce

npm install -g cavemem
cavemem install                # or --ide opencode / codex
cavemem status
# → backfill: 0 / N (0%)  last batch never

Wait as long as you like — backfill never advances. No error surfaces in cavemem status or cavemem doctor.

Running the worker in the foreground reveals the actual failure:

node "$(npm root -g)/cavemem/dist/index.js" worker run

[cavemem:embed] loading local model Xenova/all-MiniLM-L6-v2
[cavemem worker] embedder unavailable: Local embedding provider requires @xenova/transformers. Install it or set embedding.provider to 'none'. (Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@xenova/transformers' imported from <global-node-modules>/cavemem/dist/chunk-T35V7EPZ.js)
[cavemem worker] listening on http://127.0.0.1:37777 (pid <pid>)

The worker starts, the HTTP listener comes up, cavemem status reports worker: running — but embeddings never happen, and nothing tells the user the embedder itself is dead.

Root cause

@xenova/transformers is not declared in package.json dependencies for the published 0.2.1 package, despite being a hard runtime requirement for the default embedding.provider: "local" path:

npm view cavemem@0.2.1 dependencies

{
  "commander": "^12.1.0",
  "kleur": "^4.1.5",
  "better-sqlite3": "^11.5.0",
  "hono": "^4.6.10",
  "@hono/node-server": "^1.13.7",
  "@modelcontextprotocol/sdk": "^1.0.0"
}

No @xenova/transformers anywhere. So a normal npm install -g cavemem never pulls it in, and the local embedder is dead on arrival for every fresh install.

This looks like a stale npm publish, not a main-branch bug

PR #52 ("fix(embedding): migrate local provider to @huggingface/transformers v3") merged 2026-07-03, swapping the dependency from @xenova/transformers → @huggingface/transformers@3 (security motivated — old xenova pin drags in an onnxruntime-web → onnx-proto → protobufjs chain with 5 advisories).

But the latest version published to npm is 0.2.1, published 2026-05-06 — over a month before #52 merged:

npm view cavemem time --json

{
  "0.1.0": "2026-04-18T00:46:12.169Z",
  "0.1.1": "2026-04-18T01:23:49.235Z",
  "0.1.2": "2026-04-18T02:01:07.028Z",
  "0.1.3": "2026-04-18T11:53:47.694Z",
  "0.2.1": "2026-05-06T12:16:25.400Z"
}

So whatever fix landed in #52 (or whatever dependency management existed before it) has never been published. Anyone installing today via npm install -g cavemem gets a package with a dangling runtime import and no matching dependency entry, for either the old (@xenova/transformers) or new (@huggingface/transformers) package name.

Impact

• Default install path (embedding.provider: "local", the out-of-the-box default) is broken for 100% of fresh installs from npm.
• Failure is silent from the user-facing CLI (cavemem status, cavemem doctor) — no error surfaced, just a backfill count that never moves. Discovering the cause requires manually running the worker in the foreground and reading stderr.
• Semantic search (search.alpha blending BM25 + cosine) silently degrades to BM25-only, since no vectors ever get produced — again with no warning.

Suggested fixes

1. Publish a new npm version that includes whatever dependency (@huggingface/transformers@3 per fix(embedding): migrate local provider to @huggingface/transformers v3 #52, or @xenova/transformers if reverting) the code actually imports.
2. Add the embedding package as a real dependencies entry (not optional) since embedding.provider defaults to "local" — a default code path should not have a missing hard dependency.
3. Surface the failure in cavemem status/cavemem doctor — e.g. flag embedder: unavailable (see logs) instead of just showing a stalled backfill percentage with no diagnostic.

Workaround (confirmed working)

npm install -g @xenova/transformers   # or @huggingface/transformers, matching whatever `dist/chunk-*.js` actually imports on the version you have
cavemem stop
cavemem start
cavemem status   # backfill should now proceed to 100%

Installing the missing package globally (same node_modules root as cavemem itself) lets Node's module resolution find it via directory walk-up, without needing to patch cavemem's own node_modules.

---

Second confirmed symptom of the same root cause: Codex installer writes to the wrong file, ships without hooks — already fixed upstream, but not on npm

This isn't a new bug to file — it's issue #17 ("Codex installer is out of date"), reported 2026-05-14, fixed in commit f2e2f49, closed 2026-07-03. Recording it here because it independently confirms the theory above: the npm package is stuck before a batch of merged fixes, and this is a second, unrelated fix from that same stuck batch.

What #17 reported and how it was fixed

• cavemem install --ide codex wrote MCP config to ~/.codex/config.json. Codex CLI only reads ~/.codex/config.toml — the JSON file is never loaded, so the MCP server registration silently did nothing.
• Codex hooks (capture: SessionStart, UserPromptSubmit, PostToolUse, Stop) were never written at all for the codex installer — only the (wrong-file) MCP block was.
• Maintainer's fix (per closing comment): Codex installer now writes ~/.codex/config.toml with [features] codex_hooks = true (later just hooks per a follow-up comment on the issue, since OpenAI renamed the feature flag) and an [mcp_servers.cavemem] table, plus ~/.codex/hooks.json with the four capture hook entries. Slated for v0.3.0.

Independent reproduction (before finding #17)

Hit this same failure mode on a fresh cavemem install --ide codex run (macOS, cavemem 0.2.1 from npm, Codex CLI 0.144.1), without prior knowledge of #17:

• ~/.codex/config.json contained the MCP block; codex mcp list showed no cavemem entry — confirming codex never reads that file.
• No ~/.codex/hooks.json existed at all post-install.

Root cause (traced independently in codex-rs source)

Confirmed via codex-rs/hooks/src/engine/discovery.rs and codex-rs/config/src/state.rs (ConfigLayerEntry::hooks_config_folder()) that Codex's hook discovery for the user layer resolves to the directory containing config.toml — i.e., it looks for ~/.codex/hooks.json, using the same JSON schema ({"hooks": {"EventName": [{"hooks": [{"type": "command", "command": "..."}]}]}}) as Claude Code plugin hooks. This matches the maintainer's fix description exactly.

Fix applied locally (manual workaround, same shape as the merged fix)

codex mcp add cavemem -- <path-to-node> <path-to-cavemem>/dist/index.js mcp

writes the correct [mcp_servers.cavemem] table into config.toml. Then hand-wrote ~/.codex/hooks.json:

{
  "hooks": {
    "SessionStart": [
      { "hooks": [{ "type": "command", "command": "<node> <cavemem-cli> hook run session-start --ide codex" }] }
    ],
    "UserPromptSubmit": [
      { "hooks": [{ "type": "command", "command": "<node> <cavemem-cli> hook run user-prompt-submit --ide codex" }] }
    ],
    "PostToolUse": [
      { "hooks": [{ "type": "command", "command": "<node> <cavemem-cli> hook run post-tool-use --ide codex" }] }
    ],
    "Stop": [
      { "hooks": [{ "type": "command", "command": "<node> <cavemem-cli> hook run stop --ide codex" }] }
    ]
  }
}

Confirmed working after Codex's one-time hook-trust approval prompt (shown on next interactive session start — do not use --dangerously-bypass-hook-trust to skip this, it's a legitimate safety gate).

Why this belongs in this issue rather than as a new one

v0.3.0 (containing the #17 fix) has apparently never been published to npm either — same as @huggingface/transformers migration (#52) above. npm view cavemem versions still tops out at 0.2.1. The actual ask here is just: cut a release. Everything needed for a working 0.3.0+ appears to already be merged to main; it just needs npm publish.

---

Third confirmed symptom of the same root cause: OpenCode installer writes to the wrong file and ships no capture mechanism — already fixed upstream (PR #34), but not on npm

Also not a new bug — it's PR #34 ("feat: add OpenCode full featured bridge plugin"), merged 2026-07-03 (same day as #52 and the #17 fix commit f2e2f49). Recording it here as a third independent data point for the same conclusion: a batch of fixes landed on main on 2026-07-03 and none of it has reached npm.

What #34 fixed

• The OpenCode installer wrote mcpServers.cavemem into ~/.opencode/config.json. Two problems: (1) OpenCode's real config file is ~/.config/opencode/opencode.jsonc (XDG location, not ~/.opencode/), and (2) even in the right file, the schema is wrong — OpenCode expects a top-level mcp object with {type: "local", command: [...], enabled: true}, not an mcpServers map with separate command/args.
• No capture mechanism existed for OpenCode at all pre-feat: add OpenCode full featured bridge plugin #34 — the installer only ever wrote the (broken) MCP block. PR feat: add OpenCode full featured bridge plugin #34 adds a bundled bridge plugin (apps/cli/src/opencode-bridge.ts, shipped as dist/opencodeBridge.js) that the installer deploys as a symlink into OpenCode's plugin directory, mapping all 5 cavemem hook events to OpenCode's native plugin API (event for session lifecycle, tool.execute.after for tool capture).

Independent reproduction (before finding #34)

Ran cavemem install --ide opencode fresh (macOS, cavemem 0.2.1 from npm, OpenCode CLI 1.17.18) and confirmed the same two failures independently:

• ~/.opencode/config.json contained mcpServers.cavemem; opencode mcp list showed no cavemem entry at all.
• No plugin file existed anywhere OpenCode would load one from (~/.config/opencode/plugins/, .opencode/plugins/). find /opt/homebrew/lib/node_modules/cavemem -iname "*bridge*" turned up nothing — the installed 0.2.1 package doesn't contain an opencodeBridge.js at all, despite the README's install section describing OpenCode as fully capture-capable.

Fix applied locally (manual workaround, same shape as the merged fix)

Corrected ~/.config/opencode/opencode.jsonc:

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "cavemem": {
      "type": "local",
      "command": ["<path-to-node>", "<path-to-cavemem>/dist/index.js", "mcp"],
      "enabled": true
    }
  }
}

Confirmed via opencode mcp list → cavemem connected.

For capture, couldn't use the real upstream bridge verbatim — it imports @cavemem/core, @cavemem/compress, and @cavemem/config, which are internal monorepo packages never published to npm (npm view @cavemem/core → 404). Wrote a minimal equivalent at ~/.config/opencode/plugins/cavemem-bridge.js that shells out to cavemem hook run <event> (matching the same subprocess pattern the other IDEs use via hooks-scripts/*.sh) instead of importing cavemem internals directly:

import { spawn } from "node:child_process"

function runHook(event, payload) {
  const child = spawn("<node>", ["<cavemem-cli>", "hook", "run", event, "--ide", "opencode"], {
    stdio: ["pipe", "ignore", "ignore"],
  })
  child.stdin.write(JSON.stringify(payload))
  child.stdin.end()
}

export const CavememBridge = async ({ directory }) => {
  let sessionId
  return {
    event: async ({ event }) => {
      if (event.type === "session.created") {
        sessionId = event.properties?.info?.id ?? event.properties?.sessionID
        runHook("session-start", { session_id: sessionId, cwd: directory, source: "startup" })
      }
      if (event.type === "session.idle") {
        runHook("stop", { session_id: event.properties?.sessionID ?? sessionId })
      }
    },
    "tool.execute.after": async (input, output) => {
      runHook("post-tool-use", {
        session_id: input.sessionID ?? sessionId,
        tool_name: input.tool,
        tool_input: input.args,
        tool_response: output.output,
      })
    },
  }
}

Verified end-to-end: ran opencode run "list files in current directory", then queried the DB directly and confirmed a new session row (ide: "opencode", correct cwd) and a tool_use observation with the real tool name/input/output (bash, {"command":"ls -la"}, actual directory listing).

Why this belongs in this issue rather than as a new one

Same conclusion as the codex section above: npm view cavemem versions still tops out at 0.2.1, dated 2026-05-06, while three independent fixes (#52 embedding migration, #17/f2e2f49 codex installer, #34 OpenCode bridge) all merged to main on 2026-07-03 and none have shipped. Three for three IDEs tested (Claude Code was the only one that worked out of the box, since it was apparently the reference implementation cavemem's other installers were later brought up to parity with). This is not three bugs — it's one release that never happened.

---

Update: built and verified a local OpenCode bridge, one correction to the analysis above

Re-checked the OpenCode symptom on a second host (Linux, OpenCode CLI 1.17.15) before filing, and want to correct one detail from the #34 writeup above: MCP registration was not actually broken on this host.

opencode debug config shows OpenCode merges config from multiple sources — ~/.config/opencode/opencode.jsonc (XDG) and a global opencode.json living alongside the OpenCode binary install itself (~/.opencode/opencode.json on this install) — rather than reading only the XDG path. The mcp.cavemem block happened to be sitting in the latter, and opencode mcp list reported cavemem connected. So the "wrong file" framing for MCP doesn't hold across all OpenCode versions/install layouts — it may be version- or install-method-dependent. Worth confirming which behavior is current on main before assuming #34's description is still accurate.

The capture-bridge half of #34 was still completely missing, independent of the MCP question — confirmed directly against the cavemem DB: sessions table had 80 claude-code rows, 41 codex rows, 0 opencode rows, despite regular opencode usage on the host. No plugin file existed anywhere OpenCode auto-loads from (~/.config/opencode/plugins/, .opencode/plugins/).

Built a minimal bridge at ~/.config/opencode/plugins/cavemem-bridge.js (global auto-discovery dir — confirmed via OpenCode's docs that plugin in config is npm-package-only, local files are picked up by directory scan, not config entries), shelling out to cavemem hook run <event> --ide opencode the same way the other IDEs' hook scripts do, wired to:

• event hook, session.created / session.idle → session-start / stop
• chat.message hook → user-prompt-submit
• tool.execute.after hook → post-tool-use

Payload field names (session_id, cwd, source, prompt, tool_name, tool_input, tool_response) were checked against the actual handler code in the installed dist/index.js (packages/hooks/src/handlers/*.ts), not guessed.

Verified end-to-end: ran opencode run "list files in current directory using ls", then queried the DB directly — new opencode session row appeared with the correct cwd, a user_prompt observation with the real prompt text, and a tool_use observation with the real tool/input/output (bash, {"command":"ls"}, real listing). Counts went from opencode: 0 to opencode: 1 session captured.

One known gap: the stop hook fires on session.idle but writes nothing, since cavemem's stop handler needs turn_summary/last_assistant_message and OpenCode's session.idle event carries neither — would need an extra API call (or tracking the last assistant message via the event hook's message.updated/message.part.updated events) to fill in. Turn summaries aside, prompt + tool-use capture works.

This doesn't change the core ask (cut a release with what's already on main), just confirms independently that #34's bridge is the right shape and adds the exact field-mapping detail for anyone reproducing it.

[Keesan12]

A silent embedding failure leaves users with the most dangerous memory state: the system appears ready but recall is incomplete.

The UI should say what percentage is indexed, what failed, and whether the agent is operating in a degraded mode. We want MartinLoop context receipts to carry that same honesty.

Would you block queries until backfill finishes or serve them with explicit coverage? I would appreciate your perspective and feedback on our approach: https://github.com/Keesan12/martin-loop