RPC failover: CG-register TooLowAllowance recovery aborts under a broken primary (deposit read bypasses failover)

[Jurij89]

Summary

When a node has multiple chain RPC endpoints configured and the primary endpoint is broken/stale, an on-chain Context Graph auto-registration triggered by POST /api/knowledge-assets/:name/vm/publish can fail permanently with a TooLowAllowance revert — even though the operational wallet has sufficient TRAC and gas. The bug is in the TooLowAllowance → approve → retry recovery path in the chain adapter: it performs a key read (the registration deposit amount) directly against the primary provider instead of through the RPC‑failover facade. With the primary down, that read fails, is swallowed to 0, and the recovery aborts before it ever sends the TRAC approval.

This is a latent bug in the multi‑RPC failover subsystem (#1329 / #1338 / #1336), present on main. It is not related to PR #1265 (verified: packages/chain is byte‑untouched by that PR).

Impact

• Severity: Medium–High. It defeats the purpose of RPC failover for the first publish to a brand‑new CG: a broken primary is supposed to fail over to a backup and still register the CG, but instead CG registration (and therefore the publish) fails hard.
• Funds are not the problem — the wallet had TRAC + xDAI; only the on‑chain allowance to the ContextGraphs contract was 0 and the approve never ran.
• Trigger conditions: ≥2 RPC endpoints configured AND the primary is responsive‑but‑stale / wrong‑chain / flapping (see Caveats) AND the target CG is not yet registered on‑chain (auto‑register path).

Reproduction

1. Configure a node with a primary + ≥1 backup chain RPC (e.g. the base/gnosis multi‑RPC defaults from feat(chain,cli): enable multi-RPC by default — failover, 503/504 parity, observability #1329).
2. Make the primary RPC broken‑but‑responsive (e.g. point it at a stale / wrong‑chain endpoint, or one that errors on contract reads). (Originally reproduced by intentionally breaking the primary in node config.)
3. Ensure the operational wallet has TRAC + gas but 0 allowance to the ContextGraphs contract.
4. Publish a KA to a not‑yet‑registered CG via POST /api/knowledge-assets/:name/vm/publish (auto‑register path).
5. Observe: registration reverts TooLowAllowance; the recovery does not send the approve; the publish fails.
6. Restore the primary RPC → the same publish succeeds.

Live‑node telemetry during the incident: 16 recorded RPC failovers; allowance read as 0 while TRAC/gas were fine.

Root cause

The CG‑register recovery in packages/chain/src/evm-adapter-context-graph.ts:298–321:

if (!isTooLowAllowanceError(err)) throw err;
const ps = this.contracts.parametersStorage as Contract | undefined;     // :301  bound to this.signer → bare primary provider
let deposit = 0n;
try {
  deposit = ps ? await ps.contextGraphRegistrationDeposit() : 0n;        // :304  read via PRIMARY, not the failover facade
} catch {
  deposit = 0n;                                                          // :307  broken-primary error swallowed to 0
}
if (deposit === 0n) throw err;                                           // :308  ABORTS: re-throws TooLowAllowance, never approves
await this.ensureV10ApproveTrac(this.signer, ..., deposit, ..., true);  // :309  the approve — never reached
return submitCreate();                                                   // :316  the retry — never reached

Mechanism A — operative / strongest. ps (parametersStorage) is a Contract bound to this.signer, whose provider is this.primaryProvider (evm-adapter-base.ts:650; contract factory new Contract(addr, abi, this.signer) at :1614 / :1635). This read does not route through the readContract failover facades. When the primary is broken, ps.contextGraphRegistrationDeposit() throws → the catch zeroes deposit → if (deposit === 0n) throw err re‑throws the original TooLowAllowance without ever sending the approve. The recovery is dead‑on‑arrival under a broken primary. This matches the reported incident exactly.

Mechanism B — secondary, would still bite after fixing A. Even if the deposit read succeeds and the approve mines (it does — on a backup, with a real receipt), the failover transport in rpc-failover-client.ts (runAcrossProviders reads + populateAndSign / broadcast / getReceipt writes) each independently iterates endpoints from index 0 with perEndpointRetries = 0 for >1 RPC (evm-adapter-base.ts:627) and no sticky/pinned endpoint carried across calls. The retried submitCreate() re‑estimates gas (an eth_call that runs the allowance check) on a fresh index‑0 selection; if it lands on a lagging backend (or the responsive‑but‑stale primary), allowance reads 0 and the contract reverts TooLowAllowance again. That revert is a decoded CALL_EXCEPTION → isRetryableRpcError = false (evm-adapter-rpc.ts:122) → it propagates, and the recovery retries only once (evm-adapter-context-graph.ts:316) → terminal failure. confirmAllowanceVisible (evm-adapter-base.ts:1184) is best‑effort (≈6 polls, swallows read errors, returns anyway) and gives no cross‑endpoint read‑after‑write guarantee.

Note: isRetryableRpcError only treats transport errors (timeout / 429 / 5xx / network) as retryable — so a primary that responds with stale/wrong data does not trigger failover at all for these reads.

Why restoring the primary "fixes" it

With a healthy primary, every RPC op resolves at index 0 — one consistent endpoint. The deposit read (bound to the primary) succeeds → the approve fires; and the approve, the allowance re‑read, and the retry all hit that same endpoint → the just‑sent approve is immediately visible → the single retry succeeds.

Suggested fix (priority order)

1. Route the deposit read through the failover facade (readContract / readContractWith) instead of the boot‑bound primary‑only Contract handle (evm-adapter-context-graph.ts:301–308). Highest leverage — directly fixes Mechanism A / the reported incident.
2. Pin the whole recovery to one provider: carry the endpoint that mined the approve into the receipt‑wait, the allowance re‑read, and the submitCreate() retry, instead of re‑selecting from index 0 each call. Removes the read‑after‑write divergence (Mechanism B).
3. Confirm the approve receipt + re‑read allowance on the approving endpoint before retrying, treating a still‑zero allowance there as "approve not yet visible" (back off) rather than letting an arbitrary endpoint surface a terminal revert.
4. Optionally: allow a second approve+retry, or make confirmAllowanceVisible blocking on the approving endpoint.
5. Audit other recovery/"helper" reads in the chain adapter for the same bare‑primary‑bypasses‑failover pattern — the publish/update allowance recovery near evm-adapter-base.ts:892 / 933 may share it.

Affected code

• packages/chain/src/evm-adapter-context-graph.ts:298–321 — the recovery + the bare‑primary deposit read
• packages/chain/src/evm-adapter-base.ts — :627 (perEndpointRetries = 0 for >1 RPC), :650 (this.provider = this.primaryProvider), :1096 (ensureV10ApproveTrac), :1184 (confirmAllowanceVisible, best‑effort), :1614 / :1635 (bare‑primary Contract factory)
• packages/chain/src/rpc-failover-client.ts — runAcrossProviders / populateAndSign / broadcast / getReceipt (index‑0 iteration, no sticky endpoint)
• packages/chain/src/evm-adapter-rpc.ts:116–122 — isRetryableRpcError (only transport errors retryable; CALL_EXCEPTION not retryable)

Scope / attribution

• Lives in: the multi‑RPC failover work — feat(chain,cli): enable multi-RPC by default — failover, 503/504 parity, observability #1329 (multi‑RPC defaults) / refactor(chain): extract the RPC-failover subsystem out of EVMChainAdapterBase (#1336) #1338 / Refactor: extract the RPC-failover subsystem out of EVMChainAdapterBase (follow-up to #1335) #1336 (RpcFailoverClient extraction) — plus the standing CG‑register approve‑retry recovery. Present on main.
• NOT related to PR Consolidate the agent publish/share surface onto the canonical KA lifecycle (#1087) #1265 (agent API/tooling consolidation): git diff origin/main <pr-head> -- packages/chain is empty (byte‑identical); the /vm/publish route change in Consolidate the agent publish/share surface onto the canonical KA lifecycle (#1087) #1265 is comment‑only; the diff has zero references to TooLowAllowance / allowance / approve / failover. Consolidate the agent publish/share surface onto the canonical KA lifecycle (#1087) #1265 only routes publishes through /vm/publish, which surfaces this pre‑existing path.

Caveats / residual

• Mechanism A is the cleanest single explanation for the reported incident and matches the live evidence (allowance 0 while funds were fine; 16 failovers). Mechanism B is a real secondary gap a complete fix should also close.
• The bug requires the broken primary to be responsive‑but‑stale / wrong‑chain / flapping, or ≥2 heterogeneous backups. A primary that errors cleanly (ECONNREFUSED / hard timeout) on every call would fail over consistently and recovery would likely succeed — so the exact failure depends on how the primary is broken.
• Mechanism inferred from code tracing + corroborating live telemetry; not yet reproduced in a controlled test. A focused repro (stale/wrong‑chain primary + 0 allowance + unregistered CG) would confirm which mechanism dominates.

---

_{Diagnosis produced by an adversarial multi‑agent investigation (code‑attribution + root‑cause trace + live‑node probe, both load‑bearing claims independently verified); corroborated by the publishing test agent which independently identified Mechanism A.}