Migrate single-threaded pipelines to Tokio LocalRuntime

[lquerel]

This branch has been sitting in my repository for far too long. It’s time to turn it into a PR.

Change Summary

Migrates the engine from a current-thread Tokio Runtime combined with a LocalSet to Tokio's stable LocalRuntime.

Tokio stabilized LocalRuntime in Tokio 1.51.0 (https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0) on April 3, 2026. It is now the recommended direct abstraction for running !Send tasks on a single-threaded Tokio runtime, making this migration several months overdue...

This PR:

• Removes the extra LocalSet execution layer from pipelines, tests, and benchmarks.
• Centralizes engine-owned LocalRuntime construction.
• Preserves Tokio's scheduler defaults unless explicitly configured.
• Adds optional engine-level settings for event_interval, max_io_events_per_tick, and diagnostic poll-time histograms.
• Validates invalid zero-valued scheduler settings at configuration load time.
• Updates Tokio-related dependencies and socket2.
• Adds a reproducible real-pipeline performance harness.

What issue does this PR close?

• No tracking issue

Reviewer guide

Is this mostly mechanical?

Yes, but not entirely.

Most files apply the same Runtime + LocalSet to LocalRuntime conversion.
The apparent size is inflated primarily by pipeline_ctrl.rs, most
of that difference comes from removing LocalSet::run_until wrappers.

Reviewing the final Files changed view with GitHub's Hide whitespace option enabled is strongly recommended.

Suggested review order

1. Runtime construction and pipeline execution

   • crates/engine/src/runtime.rs
   • crates/engine/src/runtime_pipeline.rs
   • crates/engine/src/extension_lifecycle.rs

2. Configuration propagation and validation

   • crates/config/src/engine.rs
   • crates/config/src/engine/validate.rs
   • crates/controller/src/lib.rs

3. Runtime telemetry

   • crates/engine/src/pipeline_metrics.rs

4. Mechanical migration

   • Remaining component, test, and benchmark files

5. Supporting changes

   • Dependency updates, documentation, and changelog

Mechanical conversion pattern

The repeated migration generally follows this pattern:

• Runtime + LocalSet becomes LocalRuntime.
• .build() becomes .build_local(LocalOptions::default()).
• LocalSet::spawn_local becomes LocalRuntime::spawn_local.
• block_on(local_set.run_until(future)) becomes block_on(future).
• Redundant nested async blocks are removed and their contents de-indented.

Once this pattern has been reviewed in representative production and test
locations, most remaining occurrences should be safe to skim.

How are these changes tested?

Automated coverage includes:

• Run cargo xtask check.
• Benchmarks.

Performance was evaluated using pinned engine instances and a real OTAP pipeline:

4 traffic generators
-> OTAP receiver
-> attribute processor
-> transform processor
-> parallel fanout
-> 2 OTAP exporters
-> 2 independently pinned OTAP receiver/noop backends

Across five randomized steady-state repetitions at approximately 200K logs/s ingress and 400K logs/s aggregate egress, LocalRuntime maintained equivalent delivery with:

• Approximately 3.9% lower SUT CPU usage.
• Approximately 9.7% fewer process context switches.
• Approximately 51% fewer epoll completions, with fewer and longer blocking waits.
• No generator deadline misses, fanout nacks, or fanout throttling.

The CPU and context-switch improvements are directional rather than statistically conclusive with five repetitions. The epoll behavior consistently confirms that LocalRuntime avoids frequent short driver polls.

Are there any user-facing changes?

Yes.

The runtime migration is transparent for existing configurations. New optional engine settings are available under engine.runtime.local_runtime:

• event_interval
• max_io_events_per_tick
• poll_time_histogram

When omitted, Tokio's defaults remain in effect. The settings and their performance and latency tradeoffs are documented.

Changelog

• Added a .chloggen/*.yaml entry
• This PR is a chore (indicated in title)
• This is a documentation-only PR.

[codecov]

Codecov Report

❌ Patch coverage is 97.30392% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.37%. Comparing base (f2b0348) to head (b826989).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3731      +/-   ##
==========================================
- Coverage   87.37%   87.37%   -0.01%     
==========================================
  Files         840      842       +2     
  Lines      338187   338411     +224     
==========================================
+ Hits       295495   295670     +175     
- Misses      42160    42209      +49     
  Partials      532      532              

Components	Coverage Δ
otap-dataflow	88.46% <97.30%> (-0.01%)	⬇️
query_engine	89.46% <ø> (-0.01%)	⬇️
otel-arrow-go	52.39% <ø> (ø)
quiver	92.27% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.