feat(bench): measure memory over time via /proc sampling

[Wodann]

• Because this PR includes a bug fix, relevant tests have been included.
• Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
• I didn't do anything of this.

---

The e2e regression benchmarks previously captured only peak memory via GNU time -f %M. GNU time reads the kernel's ru_maxrss once at wait4() — an exact peak, but fundamentally incapable of producing a time series, and its %M is the max of any single process in the tree, not an aggregate.

This PR samples each measured run's process tree from /proc every 100 ms, producing a memory-over-time series per process (hardhat vs. solc vs. wrappers) alongside the existing metrics, and unifies all benchmark execution (single commands and step sequences, pnpm bench and pnpm bench:regression) onto one shared in-process runner — dropping both hyperfine and GNU time as dependencies.

How measurement works now

One bash spawn per measured run captures everything in a single pass (scripts/benchmark/helpers/runner.ts):

• Wall-clock: performance.now() around the child, minus a shell-spawn calibration offset (the wrapped no-op : timed 20× at startup, mean subtracted — the same calibration hyperfine applies). Measured offset: ~1.9 ms.
• CPU: bash's time builtin (%U %S) — a reserved word, not a process: zero fork/exec, and it reads the exact child rusage the kernel reports at wait().
• Memory (scripts/benchmark/helpers/mem-series.ts): every 100 ms, walk /proc/<pid>/task/*/children and read each process's /proc/<pid>/status. One sample costs ~50–200 µs in the otherwise-idle driver process.
  • VmRSS per process, aggregated by label (hardhat, solc-linux-amd64-v0.8.28+…, npx, …; generic script names like cli.js resolve to their npm package, including through .bin shims) → the series.
  • VmHWM — the kernel's exact per-process peak-RSS high-water mark (the same counter behind %M) → the peak, for free in the same read. It must be tracked during sampling: /proc/<pid> vanishes at process exit, and post-mortem rusage is only available to the parent (bash), not the driver.

On non-Linux (no /proc) memory entries are skipped with a warning; timing still works.

Report format

Every measured name now emits four entries (previously three):

entry	unit	value	extra
<scenario> / <name>	s	mean wall-clock	per-run times, min/max/median/mean, peakRssMb
… (cpu)	s	mean total CPU (user+system)	per-run totals + min/max/median/mean, mean user/system split
… (peak RSS)	MB	highest per-run peak (max single-process VmHWM, %M semantics)	per-run peaks + statistics
… (mem over time) (new)	MB	median over runs of the run's median tree-total RSS	see below

The (mem over time) value is median_r( median_s( Σ_p RSS[r][s][p] ) ) — a duration-weighted "typical working set", robust to spikes (which (peak RSS) tracks) and outlier runs, suitable for the 110% drift alerting. Note it aggregates the sum of the tree, so it can legitimately exceed (peak RSS) (max of a single process).

Its extra:

{
  "representativeRun": 1,
  "seriesGz": "H4sIAAAA…",   // raw series of the run with the median peak
  "runs": [
    { "durationMs": 4139, "peakRssMb": 315, "total": [3, 250, 383, 425, 569] },
    { "durationMs": 4174, "peakRssMb": 315, "total": [1, 247, 367, 424, 565] }
  ]
}

seriesGz = base64(gzip(JSON of { tMs, byProcess: { label: MB[] } } with every array delta-encoded)). Decode: gunzip (browsers: DecompressionStream("gzip")), JSON.parse, cumulative sum per array. When compression wouldn't help (tiny runs) the raw table ships as series instead, so the payload is never larger than uncompressed. total is the [p0,p25,p50,p75,p100] summary of each run's tree-total samples.

Validation

GNU time equivalence (gate for deleting the %M path):

case	GNU time	sampler
single 300 MB hog, peak	344 MB	343 MB
tree (200 MB + 350 MB children), peak	395 MB	394 MB (max-not-sum semantics confirmed)
CPU user+sys, 4 runs	mean 0.637 s	mean 0.651 s (ranges overlap; GNU time rounds to 10 ms)

Hyperfine equivalence (gate for dropping hyperfine; 10 runs each, calibrated runner):

command	hyperfine	this runner	Δ mean
sleep 0.5	501.5 ms ± 0.6	501.6 ms ± 0.4	+0.1 ms
CPU burn (~430 ms)	431.1 ms ± 7.0	432.3 ms ± 7.3	+1.2 ms (≪ noise)
shell pipeline (~4 ms)	3.7 ms ± 1.1	3.5 ms ± 0.4	−0.2 ms

Compression (real ens-verifiable-factory data, six (mem over time) entries, ~40-sample series):

encoding	total bytes	vs raw
raw JSON	4681	—
gzip + base64	2784	−40.5%
delta + gzip + base64 (implemented)	2360	−49.6%

It won on every entry even at these unfavorably short series lengths; long test solidity runs compress much better. The raw fallback guarantees it can never lose.

Also in this PR

• Single commands (previously hyperfine) gained per-run peak RSS and per-run CPU spread — hyperfine only exported one aggregate peak and mean CPU.
• pnpm bench keeps its full CLI (--warmup, --prepare, --show-output, --ignore-failure) with hyperfine-compatible --export-json output, extended with per-run memory series.
• CI workflow and scripts/setup.sh no longer install hyperfine/time.
• Failure output is unchanged: first error line, repro hint, full stdout/stderr.

Expected one-time effects on the dashboard

• Timing methodology changed (calibrated in-process vs. hyperfine): validated to agree within ~1 ms, so any step in the charts is ≪ the 110% alert threshold.
• (cpu) entries gain a real ± range and per-run statistics in extra.

Follow-up (hardhat-benchmark-results repo)

Consume the new entry: the scalar chart comes free from the existing renderer; add the commit-overlay view (one line per commit, gradient by age) and the commit × time-in-run heatmap — both read the same seriesGz payload. Interactive heatmap mockup shared separately.

🤖 Generated with Claude Code

[changeset-bot]

⚠️ No Changeset found

Latest commit: 3108d0d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[Copilot]

Pull request overview

This PR updates Hardhat’s benchmark tooling to sample memory over time on Linux by walking /proc during each measured run, while also unifying single-command and step-sequence benchmarking onto a shared in-process runner and removing the hyperfine/GNU time dependency chain.

Changes:

• Replaces hyperfine + GNU time-based benchmarking with a shared bash-spawn runner that measures wall-clock (calibrated), CPU (bash time), and /proc-sampled memory series + exact per-process peak RSS (VmHWM).
• Extends regression benchmark output with new (mem over time) entries and updates scenario schema/docs to reflect the new measurement semantics.
• Removes installation steps for hyperfine/time from setup scripts and CI workflow.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
scripts/setup.sh	Drops hyperfine installation from dev setup.
scripts/end-to-end/types.ts	Updates benchmark type docs to remove hyperfine wording.
scripts/end-to-end/schema/scenario.schema.json	Updates schema descriptions to match new runner semantics.
scripts/benchmark/regression.ts	Switches regression benchmarking to the shared measured runner; adds mem-over-time reporting.
scripts/benchmark/main.ts	Replaces hyperfine invocation with in-process runner and emits hyperfine-compatible JSON export.
scripts/benchmark/helpers/stats.ts	Updates stats docs to reflect per-run aggregation source.
scripts/benchmark/helpers/runner.ts	Adds shared measured runner (wall/CPU + optional /proc sampling).
scripts/benchmark/helpers/runner.test.ts	Adds unit tests for runner helpers (quoting, wrapping, parsing, formatting).
scripts/benchmark/helpers/plan.ts	Updates docs to remove hyperfine references.
scripts/benchmark/helpers/memory.ts	Removes GNU time wrapper implementation.
scripts/benchmark/helpers/memory.test.ts	Removes GNU time wrapper tests.
scripts/benchmark/helpers/mem-series.ts	Adds /proc process-tree sampler + series encoding utilities.
scripts/benchmark/helpers/mem-series.test.ts	Adds unit tests for labeling, encoding, summaries, and pivoting.
scripts/benchmark/helpers/args.ts	Removes memFile arg plumbing now that GNU time wrapper is gone.
.github/workflows/regression-benchmark.yml	Removes system dependency installation for hyperfine/GNU time.

Comments suppressed due to low confidence (1)

scripts/benchmark/main.ts:128

• The benchmark creates a temp dir for the CPU timing file but never removes it on success, which can leak many /tmp directories over repeated runs. Consider cleaning it up after a successful run, while still keeping it around on failure for debugging.

  const timingPath = path.join(
    mkdtempSync(path.join(tmpdir(), "hardhat-bench-")),
    "cpu.txt",
  );

[Wodann]

note: This was previously in e2eExec—which we no longer call—so we move the !existsSync check up here

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated no new comments.

Suppressed comments (1)

scripts/benchmark/regression.ts:809

• representativeRun is documented as the run with the median peak RSS, but the code picks the representative run using the max tree-total RSS sample (summaries.map(s => s[4])). This can select a different run than the median peakRssMb, making series/seriesGz inconsistent with the peakRssMb values and the PR description.

  const summaries = defined.map((memory) =>
    fiveNumberSummary(memory.samples.map(treeTotalMb)).map(Math.round),
  );
  const representative = pickRepresentativeRun(summaries.map((s) => s[4]));
  const p50Stats = computeStats(summaries.map((s) => s[2]));

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (3)

scripts/benchmark/helpers/runner.ts:79

• runMeasured starts a MemorySampler interval but only stops it on the success path. If execute() rejects (command fails and ignoreFailure is false, or spawn errors), the interval remains active (it’s unref()’d but will still keep sampling until process exit). Ensure the sampler is stopped in the error path as well.

  const sampler = procSamplingAvailable() ? new MemorySampler() : undefined;
  const { wallSeconds } = await execute(
    wrapWithCpuTiming(command, timingPath),
    options,
    sampler,

scripts/benchmark/render-mem-series.ts:140

• resolveArgs doesn’t validate that --output/--scenarios/--benchmarks are followed by a value. If the flag is last (or followed by another flag), argv[++i] becomes undefined and the tool silently falls back to defaults, which is hard to diagnose. Consider erroring out when an option value is missing.

      case "--output":
        output = argv[++i];
        break;
      case "--scenarios":
        scenarios = argv[++i];

scripts/benchmark/helpers/mem-series.ts:220

• encodeSeriesTable compares seriesGz.length to raw.length, but those are different payload shapes (base64 string vs JSON of the table). This can pick seriesGz even when the final serialized extra becomes larger than the raw table, contradicting the “never larger than uncompressed” intent. Compare the serialized {seriesGz: ...} vs {series: ...} payload sizes instead.

  const raw = JSON.stringify(table);
  const seriesGz = gzipSync(
    JSON.stringify({
      tMs: deltaEncode(table.tMs),
      byProcess: Object.fromEntries(

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Suppressed comments (4)

scripts/benchmark/helpers/runner.ts:83

• If the measured command fails, execute() rejects and runMeasured() exits before calling sampler.stop(). That leaves the MemorySampler interval running for the rest of the process, continuing /proc polling and potentially affecting subsequent measurements.

  const calibration = await shellSpawnOverheadSeconds();

  const sampler = procSamplingAvailable() ? new MemorySampler() : undefined;
  const { wallSeconds } = await execute(
    wrapWithCpuTiming(command, timingPath),
    options,
    sampler,
  );
  const memory = sampler?.stop();

  const cpu = parseCpuTiming(readFileSync(timingPath, "utf-8"), timingPath);

scripts/benchmark/main.ts:126

• mkdtempSync() creates a new temp directory per benchmark run, but the directory is never removed. Over time this can leak many hardhat-bench-* directories under the system temp dir (especially on CI/self-hosted runners).

  const timingPath = path.join(
    mkdtempSync(path.join(tmpdir(), "hardhat-bench-")),
    "cpu.txt",
  );

scripts/benchmark/main.ts:1

• The benchmark temp-dir cleanup uses rmSync(...), but it's not currently imported in this file.

import { existsSync, mkdtempSync, writeFileSync } from "node:fs";

scripts/benchmark/regression.ts:566

• This block declares const runs twice (outer run count for logging, then the array of measured runs). The shadowing is easy to misread and makes future edits riskier.

    const runs = await runSeries(
      cfg.command,
      path.join(scenarioTmpDir, `${slugify(name)}-cpu.txt`),
      { cwd: workingDir, env, runs: cfg.runs, prepare: cfg.prepare },
    );

    return measuredRunsToEntries(scenarioId, name, runs);

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (4)

scripts/benchmark/main.ts:126

• mkdtempSync(path.join(tmpdir(), "hardhat-bench-")) creates a new temp directory for every pnpm bench invocation, but it’s never cleaned up. Over time this can clutter /tmp on developer machines or CI runners. Consider using a try/finally to remove the directory (or reuse a stable temp path).

  const timingPath = path.join(
    mkdtempSync(path.join(tmpdir(), "hardhat-bench-")),
    "cpu.txt",
  );

scripts/benchmark/regression.ts:834

• representativeRun is documented as “the run with the median peak”, but the representative index is currently chosen from the tree-total p100 values (summaries.map((s) => s[4])), not from the run’s actual peak RSS (MemorySeries.peakRssMb). These can diverge, which would make the stored series not match the stated selection rule.

  const summaries = defined.map((memory) =>
    fiveNumberSummary(memory.samples.map(treeTotalMb)).map(Math.round),
  );
  const representative = pickRepresentativeRun(summaries.map((s) => s[4]));
  const p50Stats = computeStats(summaries.map((s) => s[2]));

scripts/benchmark/regression.ts:566

• runs is declared twice in the same function (const runs = … for the count, then const runs = await runSeries(…) for the measured runs array). This shadowing makes the code harder to follow and is easy to trip over during future edits.

    const runs = await runSeries(
      cfg.command,
      path.join(scenarioTmpDir, `${slugify(name)}-cpu.txt`),
      { cwd: workingDir, env, runs: cfg.runs, prepare: cfg.prepare },
    );

scripts/benchmark/helpers/mem-series.ts:220

• This compression decision compares seriesGz.length against JSON.stringify(table).length, but the actual payload size also includes the wrapper key ({seriesGz:...} vs {series:...}). This can pick seriesGz even when the wrapped JSON ends up larger (or vice versa). Comparing the wrapped JSON lengths avoids this edge case.

  return seriesGz.length < raw.length ? { seriesGz } : { series: table };

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Suppressed comments (3)

scripts/benchmark/main.ts:126

• mkdtempSync creates a new temp directory for the CPU timing file, but it’s never cleaned up. Repeated pnpm bench runs will leave many hardhat-bench-* directories under the OS temp dir.

  const timingPath = path.join(
    mkdtempSync(path.join(tmpdir(), "hardhat-bench-")),
    "cpu.txt",
  );

scripts/benchmark/main.ts:1

• The cleanup fix for the temp timing directory needs rmSync, but it isn’t imported in this file right now.

import { existsSync, mkdtempSync, writeFileSync } from "node:fs";

scripts/benchmark/render-mem-series.ts:140

• resolveArgs doesn’t validate that --output, --scenarios, and --benchmarks are followed by a value. If any of these options are the last argument, the script will proceed with undefined and fail later with a less helpful error.

      case "--output":
        output = argv[++i];
        break;
      case "--scenarios":
        scenarios = argv[++i];
        break;
      case "--benchmarks":
        benchmarks = argv[++i];
        break;

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

scripts/benchmark/main.ts:127

• mkdtempSync creates a new temp directory for the CPU timing file, but it is never removed. Repeated pnpm bench runs will leave many hardhat-bench-* directories in the system temp dir. Consider removing the temp directory in a finally block (also on error) after runSeries completes.

  const timingPath = path.join(
    mkdtempSync(path.join(tmpdir(), "hardhat-bench-")),
    "cpu.txt",
  );