feat: live in-place hot-reload engine (processor swap + apply decision) — PR1 of §4 (Tier 1) by devarismeroxa · Pull Request #2613 · ConduitIO/conduit

devarismeroxa · 2026-07-13T17:27:59Z

What

PR1 (engine) of the live in-place hot-reload arc (§4). Makes a processor-config or pipeline name/description change apply to a running pipeline in place — no stop/restart, no availability blip, no position replay, no record loss. Design: docs/design-documents/20260712-pipeline-dev-hot-reload.md (merged #2601). PR2 (the conduit run --dev surface) follows.

Risk tier: 1 (data path). Requesting DeVaris sign-off + failure-mode analysis (below) before merge.

The five increments

Swap primitive (pkg/lifecycle/stream) — ProcessorNode.Reconfigure: swaps a processor at a record boundary, applied by the run-loop goroutine itself (so n.Processor is never mutated concurrently with Process), open-before-teardown, idle-interruptible.
Classification (pkg/provisioning) — Change.LiveSwappable + Diff.LiveEligible(): processor + name/description → swappable; connector/DLQ/topology → restart.
Lifecycle wiring — ReconfigureProcessor reaches the live node; parallel/absent → ErrProcessorNotLiveReconfigurable → restart fallback; poc arch refuses in parity.
Apply decision — ApplyPlanLive applies live-eligible diffs in place; parallel processor → restart fallback; open-failure → rollback + error, never restart.
Tests — swap boundary, open-fail-keeps-old, idle, ctx-cancel; classification (every action type); apply in-place/fallback/rollback; batching contract test (1-in-1-out tripwire); property test (200 records × many swaps: no drop/reorder); mid-swap fault test.

Failure-mode analysis (Tier 1)

Could it drop/reorder records? The swap happens at a record boundary in the run-loop goroutine (no record mid-Process); backpressure pauses the source during the swap; open-before-teardown keeps the old processor on a bad edit. Proven by the property test (exactly-once, in-order across many swaps), the contract test (1-in-1-out), and the fault test. Invariants 1/3/4 hold.
Could a failed apply diverge store vs live? On a post-commit swap failure, rollbackInPlace restores the old config and re-swaps back; on a parallel-processor fallback, the restart rebuilds from the committed config. The pipeline keeps running its old processors throughout (open-before-teardown).
Crash mid-swap? The store commit is the single source of truth; node swaps are in-memory/non-durable, so restart resumes from the committed config (invariant 5). The mid-swap fault test covers ctx-cancellation; the full SIGKILL chaos test is honestly deferred to Phase 2 per the process-maturity table (design doc states this).
What would show a regression? A record-throughput drop or DLQ spike on a pipeline that took an in-place apply; pipeline status flapping.
Rollback: additive to ApplyPlanLive; the operator gate (allowRestartOnRunning) defaults off, so the in-place path is only reachable when explicitly authorized. Reverting the PR returns ApplyPlanLive to pure drain-and-restart (the shipped behavior). A stuck pipeline can always be stopped + started (full rebuild from stored config).

Decisions to confirm at sign-off

Operator gate (Option A, conservative): an in-place change to a running pipeline still requires allowRestartOnRunning. The in-place path is the non-disruptive execution of an authorized apply, not a gate bypass. A future refinement could relax it for provably-in-place diffs.
Rollback reverse-import is exercised by the fault path but not fully asserted by the stateless provisioning mock (noted in the test) — worth a chaos/integration follow-up in Phase 2.

Adversarial self-review

Re-read for: the swap boundary race (safe — single-goroutine application), open-before-teardown resource cleanup (failed new processor torn down), the ErrProcessorNotLiveReconfigurable fallback (restart is a safe superset), and the rollback path (best-effort, pipeline keeps running). All existing ApplyPlanLive restart tests still pass (connector/DLQ/topology diffs aren't live-eligible → restart path unchanged).

🤖 Generated with Claude Code

https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

The core of the hot-reload engine (design: docs/design-documents/20260712- pipeline-dev-hot-reload.md, §4). Adds ProcessorNode.Reconfigure, which swaps a running processor's config at a record boundary WITHOUT restarting the pipeline. Mechanism: - The swap is applied by the Run goroutine itself, between records (applyPendingSwap at the top of the loop), so n.Processor is never mutated concurrently with Process — no data race, no record straddles a swap. - Reconfigure (called from the apply goroutine) only stages the request under a mutex and nudges a buffered wake channel; Run's receive selects over the inbound channel AND the wake, so a swap applies promptly even on an idle pipeline (interruptible wait — replaces the blocking Trigger receive for ProcessorNode; adds base.In() accessor for the inbound channel). - Open-before-teardown: the new processor is Opened first; only on success is it switched in and the old torn down. On Open failure the old processor keeps running and the error is returned — a bad edit never drops the pipeline (invariant 3). The failed processor is best-effort torn down to avoid a leak. Data-safety (design's Data-safety §): the swap consumes no data record, forwards in order (old config before the boundary, new after — invariant 4), and relies on backpressure to pause the source during the swap (no drop/skip). Records never reach the DLQ due to a swap. Tests (race-clean): swap-at-record-boundary (record N old / N+1 new); Open-failure- keeps-old; idle-pipeline-applies-promptly (the interruptible-wait gate); context- cancelled. Log parity with nodeBase.Receive preserved (Example_complexStream). First increment of PR1 (Tier 1). Remaining: LiveSwappable/LiveEligible classification + batching contract test, ApplyPlanLiveInPlace apply decision + lifecycle wiring, in-place rollback, property + mid-swap fault tests. Roadmap: Phase 1 §4. Advances #2588's follow-on hot-reload arc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

…pply Adds the classification the live in-place apply path keys on (design doc §4, "Classification"): - Change.LiveSwappable (additive JSON field) + Change.liveSwappable(): a change is live-swappable iff it is a processor config update, or a pipeline update touching only name/description. Connector changes (position/ack/connection), the DLQ (a live destination with ack semantics), and every topology change are NOT swappable and force a restart-class apply. - Diff.LiveEligible(): all-or-nothing — a diff applies in place only if it is non-empty and every change is live-swappable; a mixed diff restarts (the source change needs it anyway). - Plan() populates LiveSwappable per change (derived from Resource/Action/ ConfigPaths, so it folds into the hash consistently). - Shared config-path name constants so the producer (diffPipelineFields) and consumer (liveSwappable) can't disagree on spellings. Tests cover every (Resource, Action) combination and the pipeline-update ConfigPaths cases (name/description swappable; dlq/connectors/processors not), plus the all-or-nothing LiveEligible rule. Second increment of PR1 (Tier 1). Next: ApplyPlanLiveInPlace apply decision + lifecycle wiring (reach the running ProcessorNode, call Reconfigure) + in-place rollback + batching contract test + property/fault tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

… swap Wires the provisioning apply path to the stream-level swap primitive. Adds lifecycle.Service.ReconfigureProcessor(pipelineID, processorID): finds the live *stream.ProcessorNode in the running pipeline, rebuilds the runnable processor from its (already-updated) stored instance, and swaps it via ProcessorNode.Reconfigure — no restart. - ErrProcessorNotLiveReconfigurable: returned when the target isn't a plain single-worker ProcessorNode (parallelized as a ParallelNode, or absent), so the apply path falls back to a restart — always a safe superset (restart re-imports and rebuilds every node). - Ordering contract documented: caller persists the new config first; open-before-teardown means a failed open keeps the old processor running (invariant 3). - Added to provisioning.LifecycleService and the pkg/conduit mirror interface; provisioning mock regenerated with the pinned mockgen (v0.6.0). lifecycle-poc refuses with CodeStopAndWaitUnsupported, matching its StopAndWait parity. Tests: not-running -> error; no plain ProcessorNode (parallel/absent) -> ErrProcessorNotLiveReconfigurable. The happy-path swap is covered by the stream-package Reconfigure tests and will be covered end-to-end by the ApplyPlanLiveInPlace integration test (next increment). Third increment of PR1 (Tier 1). Next: ApplyPlanLiveInPlace apply decision (in-place vs restart), metadata (name/description) live update, in-place rollback, batching contract test, property + mid-swap fault tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

The apply decision that composes the hot-reload engine (design §4). ApplyPlanLive now, for a running pipeline whose diff is LiveEligible, applies it IN PLACE via a new applyInPlace path instead of a StopAndWait drain-and-restart: it commits the new config (invariant 5), then swaps each changed processor into the live node graph via lifecycle.ReconfigureProcessor. Name/description changes take effect through the committed store instance. A processor tweak now lands with no availability blip, no position replay, no record loss. Fallback + rollback (the failure modes): - If a processor can't be swapped live (ErrProcessorNotLiveReconfigurable — e.g. it's parallelized), applyInPlace returns swappedAll=false and the config is already committed, so ApplyPlanLive falls through to the restart path, which rebuilds every node from it. Safe superset. - If the new processor fails to open, the old one keeps running (open-before-teardown); applyInPlace rolls back (restore old config, re-swap any already-swapped processors) so store and live agree, and returns the error WITHOUT restarting. The operator gate is unchanged (Option A, conservative): any change to a running pipeline still requires allowRestartOnRunning; the in-place path is the non-disruptive EXECUTION of an authorized apply, not a way around the gate. A future refinement could relax the gate for provably-in-place diffs. Tests: processor update -> in-place, no StopAndWait/Start; not-live-reconfigurable -> restart fallback (stop/start); open-fails -> rollback + error, no restart. All existing ApplyPlanLive restart tests still pass (connector/DLQ/topology diffs are not live-eligible, so the restart path is unchanged). Fourth increment of PR1 (Tier 1). Remaining: the batching contract test and the property + mid-swap fault tests, then this is ready to open as PR1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

… live swap Completes PR1's test suite (design doc's PR1 acceptance criteria): - Batching contract test: pins the non-buffering 1-in-1-out Process model the live-swap zero-drop guarantee depends on. The node hands Process exactly one record per call and never batches; if a future change makes it batch, this test breaks — the tripwire that forces whoever adds batching/windowing to confront the hot-swap interaction (restart-class opt-out, or flush-before-teardown) rather than silently introducing a drop. - Property test: interleaves 200 records with many live reconfigures and asserts every record is delivered exactly once, in order — no drop, no duplicate, no reorder across swaps (invariants 3, 4). Deterministic interleaving (no rapid/ gopter dependency added). - Mid-swap fault test (the design's PR1 gate for the new crash surface): cancelling the node context while a swap is mid-Open keeps the already-processed record, fails the reconfigure cleanly (old processor kept), and shuts the node down with no panic or deadlock. All race-clean and stable over 10x runs. Final increment of PR1 (Tier 1). PR1 is now complete: swap primitive, classification, lifecycle wiring, apply decision, and the full test suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

…etach in-place ctx Resolves the independent Tier-1 review's findings on #2613: - MAJOR-1 (blocker): a processor Workers change (1<->N) is a node-topology change (ProcessorNode <-> ParallelNode) that an in-place swap cannot make — yet liveSwappable returned true for any processor update, so a Workers 1->N change was classified live-eligible, committed, and reported as applied while the runtime stayed single-worker until a later restart (a silent wrong-apply, operator-reachable via ApplyPipeline). Fixed: a "workers" ConfigPath now disqualifies a processor update from live-swappable (it restarts). Regression cases added for workers / settings / plugin / condition. - MINOR-1: applyInPlace now detaches from the caller's context (context.WithoutCancel) once it starts committing live state. Previously a caller context cancelled mid-swap abandoned ProcessorNode.Reconfigure's wait while the Run goroutine completed the swap (reporting a failure the apply landed), and made the rollback's transactional import fail on the cancelled context, skipping it. New test proves the apply completes on a non-cancelled context and reports success despite a mid-apply caller cancel. - MINOR-2: added the missing lifecycle-poc parity test asserting ReconfigureProcessor refuses with CodeStopAndWaitUnsupported. - MINOR-3: documented that a name/description change updates the stored instance but the running pipeline's metric-label name lags until restart (cosmetic). All race-clean; lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

devarismeroxa · 2026-07-13T17:53:24Z

Tier-1 sign-off from DeVaris (maintainer). Independent fresh-context review complete; its MAJOR-1 (Workers misclassification) blocker and MINOR-1/2/3 are fixed in 731300a. All CI green. Merging.

…, Tier 1) (#2617) * fix(processor,provisioning): live in-place apply commits running-processor config The in-place hot-reload path (PR1, #2613) never worked end-to-end: applyInPlace committed the new processor config through processor.Service.Update, whose running-instance guard (ErrProcessorRunning) refused it, so ApplyPlanLive returned the error and lifecycle.ReconfigureProcessor — the actual live swap — was never reached. PR1's mock-based tests returned success for the mocked Update and hid it; PR2's real-engine integration test exposed it (found in the #2616 review). Fix (Option A): add processor.Service.UpdateWhileRunning — Update minus the running guard — and route provisioning's updateProcessorAction through it. This is safe for both provisioning paths: the restart path applies it on a stopped pipeline (processor not running, so equivalent to Update), and the in-place path immediately swaps the live node via ReconfigureProcessor to match the committed config. Update keeps its guard for direct API/orchestrator callers, who do NOT swap the node (verified: pkg/orchestrator still uses Update). Regression test: TestService_UpdateWhileRunning_BypassesRunningGuard — Update refuses a running instance (ErrProcessorRunning); UpdateWhileRunning commits the config. The provisioning apply/import tests now assert the engine calls UpdateWhileRunning. The full real-engine end-to-end assertion lives in PR2's tightened integration test. Risk tier 1 (data path). Failure-mode analysis in the PR. Fixes the §4 in-place hot-reload blocker; unblocks PR2 (#2616). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr * docs: fix stale godoc reference (Update -> UpdateWhileRunning) in updateProcessorAction Review NIT on #2617. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The developer-facing surface over the live in-place hot-reload engine (PR1, #2613). `conduit run --dev` watches the pipelines directory and applies config changes to running pipelines on save — a processor/name/description edit applies in place (no restart), a connector/DLQ/topology edit drain-restarts, each labeled. - The watcher lives in the runtime (pkg/conduit/dev), started after ProvisionService.Init and tied to the serve context so Ctrl-C cancels it (invariant 7). --dev sets Config.Dev. - Per-change flow: debounce -> parse all pipelines in the file + Enrich/Validate -> Plan (skip empty) -> ApplyPlanLive(..., allowRestartOnRunning=true) -> ensure- running. --dev is itself the operator authorization for the applies it drives; it does NOT enable the process-level API gate. - Ensure-running: a brand-new file or a pipeline left stopped by a prior failed apply is started (unless the config declares it stopped). - Parse/validate error keeps the running pipeline untouched; a deleted watched file leaves its pipeline running and logs (never auto-deletes); atomic-save (rename) tolerated by watching the directory. - --json structured events per apply; `conduit pipelines dev [dir]` thin alias. - fsnotify promoted to a direct dependency (was transitive) — no new module, just a direct import of an already-present dep. Tests: watcher event->file->pipeline mapping, debounce/coalesce, parse-error tolerance, ensure-running variants, file-delete keep-running, plus an integration test with real fsnotify + a generator->processor->file pipeline (edit the processor -> applied in place; edit a source -> labeled restart; syntax error -> pipeline keeps running). Operator runbook added. Advances execution-plan §4 (completes the hot-reload arc) + §3. Risk tier 2 (a watcher over the already-reviewed engine). Authored by a subagent; finalized (runbook + verify) by the main session after it stalled at the finish line. Pending independent review before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B3cLokwNJYLLBpdyt3cgGr

devarismeroxa and others added 5 commits July 13, 2026 10:26

devarismeroxa requested a review from a team as a code owner July 13, 2026 17:27

devarismeroxa merged commit 2966987 into main Jul 13, 2026
5 checks passed

devarismeroxa deleted the feat/live-inplace-engine branch July 13, 2026 17:53

This was referenced Jul 13, 2026

feat: conduit pipelines repair + MCP repair with fix-synthesis (§3/§2) #2614

Merged

feat(cli): conduit run --dev hot-reload watcher (PR2 of §4) #2616

Merged

fix: live in-place apply commits running-processor config (§4 blocker, Tier 1) #2617

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: live in-place hot-reload engine (processor swap + apply decision) — PR1 of §4 (Tier 1)#2613

feat: live in-place hot-reload engine (processor swap + apply decision) — PR1 of §4 (Tier 1)#2613
devarismeroxa merged 6 commits into
mainfrom
feat/live-inplace-engine

devarismeroxa commented Jul 13, 2026

Uh oh!

devarismeroxa commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

devarismeroxa commented Jul 13, 2026

What

The five increments

Failure-mode analysis (Tier 1)

Decisions to confirm at sign-off

Adversarial self-review

Uh oh!

devarismeroxa commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant