Skip to content

Latest commit

 

History

History
119 lines (87 loc) · 11.2 KB

File metadata and controls

119 lines (87 loc) · 11.2 KB

Metrics contract

This document defines units, when each metric is meaningful (explicit vs simulated timing mode), and aggregation rules for benchmark results. Used by results.v0.2 (CI-stable) and results.v0.3 (paper-grade). RUN_REPORT, summary.md, and coordination leaderboards refer to this document for metric semantics.

Schema alignment: The per-episode metrics table below is the source of truth for result fields. The JSON schema policy/schemas/results.v0.2.schema.json (and v0.3) should stay aligned with this table when adding or changing metrics.

Pipeline and audit: Result and summary files are pipeline-agnostic in schema (same metrics regardless of pipeline mode). Every benchmark result file records pipeline_mode (deterministic | llm_offline | llm_live), llm_backend_id, allow_network, and non_deterministic for audit. Deterministic runs are required for baseline regression and for the canonical official baselines under benchmarks/baselines_official/v0.2/results/.

Timing modes

  • explicit: Step timestamps only; device completion times are omitted. p95 TAT is derived from step times. Device utilization and queue-length stats are omitted in explicit mode.
  • simulated: Device capacity and service-time models apply; p95 TAT is meaningful (accept→release includes queuing and device time). Device utilization and queue-length metrics are populated.

Per-episode metrics

Metric Unit Meaningful when Aggregation (summary)
throughput count (integer) Both Mean, std; v0.3: quantiles (p50, p90), 95% CI
p50_turnaround_s seconds Both (explicit: step-derived; simulated: completion-derived) Mean, std; v0.3: quantiles, 95% CI
p95_turnaround_s seconds Both; simulated preferred for interpretation Mean, std; v0.3: quantiles, 95% CI
p95_turnaround_s_note Informational Not aggregated
on_time_rate [0, 1] Both (requires sla_turnaround_s) Mean, std
violations_by_invariant_id dict (invariant_id → count) Both Sum over episodes; per-invariant totals
blocked_by_reason_code dict (reason_code → count) Both Sum over episodes; per-reason totals
critical_communication_compliance_rate [0, 1] or null Both Mean over episodes with non-null
tokens_minted count Both Sum or mean; v0.2 regression: integer only
tokens_consumed count Both Sum or mean; v0.2 regression: integer only
holds_count count Both Sum or mean; v0.2 regression: integer only
steps count Both Mean, std; v0.2 regression: integer only
transport_consignment_count count Both (multi_site_stat) Sum or mean
transport_temp_excursions count Both (multi_site_stat) Sum or mean
coc_breaks_count count Both (multi_site_stat) Sum or mean
detection_latency_s seconds adversarial_disruption Mean over episodes with value
containment_success boolean adversarial_disruption Fraction true (0 or 1 per episode)
time_to_first_detected_security_violation seconds insider_key_misuse Mean over episodes with value
fraction_of_attacks_contained [0, 1] insider_key_misuse Mean over episodes
forensic_quality_score [0, 1] insider_key_misuse Mean over episodes
device_utilization per-device [0, 1] Simulated only Mean over episodes (per device); v0.3
device_queue_length_mean per-device float Simulated only Mean over episodes; v0.3
device_queue_length_max per-device integer Simulated only Max over episodes; v0.3
llm_confidence_calibration object (ece, mce) Episodes with LLM decisions (action_proposal.confidence) Optional; ECE/MCE over (confidence, accepted) per step; v0.3 summary: llm_confidence_ece_mean, llm_confidence_mce_mean

CI-stable subset (v0.2 regression)

For cross-OS/Python stability, the baseline regression guard compares only:

  • Integers: throughput, holds_count, tokens_minted, tokens_consumed, steps
  • Structs: blocked_by_reason_code, violations_by_invariant_id

The regression test compares only integers and struct fields (throughput, holds_count, tokens, steps, blocked_by_reason_code, violations_by_invariant_id). Float metrics such as p50_turnaround_s, p95_turnaround_s, and on_time_rate stay outside CI comparison so small numerical differences do not fail the guard.

Paper-grade (v0.3)

v0.3 adds optional fields for reporting:

  • Quantiles: turnaround_quantiles_s (p10, p25, p50, p75, p90); throughput/summary quantiles in aggregated summary.
  • Confidence intervals: 95% CI (e.g. mean ± 1.96 * std/sqrt(n)) for throughput, p95_turnaround_s, etc., in summary.
  • Binomial rate CIs: For containment_success (adversarial_disruption), v0.3 summary includes containment_success_rate_ci_lower and containment_success_rate_ci_upper (Clopper-Pearson or Wilson 95% CI) when aggregating across episodes.
  • Simulated-mode distributions: device_utilization, device_queue_length_mean, device_queue_length_max per episode; aggregated in summary as mean/max over episodes.

Summary outputs (from labtrust summarize-results --in <paths> --out <dir> --basename summary):

  • summary_v0.2.csv — CI-stable; backward compatible. Columns: task, agent_baseline_id, partner_id, n_episodes, plus for each metric only *_mean and *_std (e.g. throughput_mean, throughput_std, p95_turnaround_s_mean, p95_turnaround_s_std). No quantile or CI columns. Used for baseline regression guard.
  • summary_v0.3.csv — Same rows as v0.2; columns include v0.2 columns plus paper-grade: *_p50, *_p90, *_mean_ci_lower, *_mean_ci_upper (when computable), containment_success_rate_ci_lower, containment_success_rate_ci_upper (binomial 95% CI when task has containment_success), and llm_confidence_ece_mean, llm_confidence_mce_mean (when episodes have llm_confidence_calibration). May contain empty/NaN for quantiles or CI when insufficient episodes. Quantile and CI columns are non-empty when there are enough episodes per group (e.g. n_episodes >= 2 for standard error and 95% CI).
  • summary.csv — Copy of summary_v0.2.csv (identical content).
  • summary.md — Markdown table derived from v0.2 aggregates only (same columns as summary_v0.2.csv). No quantile or CI columns in the table. When any result has metadata.run_duration_wall_s, the markdown also includes a Run info section (table of run_duration_wall_s, episodes_per_second per result) and a short footer.

Uncertainty metrics in standard reports

Uncertainty quantification fields are integrated into regular benchmark and coordination outputs so they appear in standard reports without custom tooling.

Output Uncertainty fields reported
summary_v0.3.csv (summarize-results) containment_success_rate_ci_lower, containment_success_rate_ci_upper; llm_confidence_ece_mean, llm_confidence_mce_mean (when applicable).
summary_coord.csv (coordination study) sec.attack_success_rate_ci_lower, sec.attack_success_rate_ci_upper, sec.worst_case_attack_success_upper_95.
pack_summary.csv (coordination security pack) sec.attack_success_rate_ci_lower, sec.attack_success_rate_ci_upper, sec.worst_case_attack_success_upper_95.
SECURITY/coordination_risk_matrix.csv and .md (pack) Same sec.* CI and worst-case columns as pack_summary.

Per-episode metrics (e.g. llm_confidence_calibration) are present in results JSON when computed; aggregation adds the summary columns above. See Uncertainty quantification for epistemic vs aleatoric mapping.

Schema compatibility: every v0.2 top-level field and every v0.2 episode.metrics field exists in v0.3 with compatible types; v0.3 may add optional fields. Enforced by tests/test_metrics_contract.py.

Coordination LLM economics (results.v0.2)

For LLM-based coordination methods (planner, allocator, bidder, repairer, local-decider), each episode may include an optional coordination.llm block with consistent economics fields. When the method is non-LLM or no LLM was invoked, this block is absent; aggregation fills 0 or null for summary outputs.

Field Type Meaning
llm.call_count integer Number of LLM calls in the episode (0 for deterministic unless simulated).
llm.total_tokens integer Total input + output tokens (0 for deterministic unless simulated).
llm.tokens_per_step float total_tokens / steps.
llm.mean_latency_ms float or null Mean latency per call in ms; null for offline/deterministic.
llm.p95_latency_ms float or null 95th percentile latency per call; null for offline.
llm.error_rate float Fraction of calls that failed (e.g. API errors).
llm.invalid_output_rate float Schema violations or parse fallbacks per call (invalid outputs / call_count).
llm.estimated_cost_usd float or null Estimated cost in USD when model pricing exists; null otherwise.

Aggregation (e.g. coordination study summary): cost.total_tokens is the sum of episode llm.total_tokens; cost.estimated_cost_usd is the sum of episode llm.estimated_cost_usd; llm.error_rate and llm.invalid_output_rate are means over episodes. For cells with no coordination.llm in any episode, these summary fields are 0 or null.

Contract: For any run that uses an LLM coordination method, results (and summary_coord.csv / pack_summary.csv) include cost and latency columns (cost.estimated_cost_usd, p95_llm_latency_ms, and related fields); they are null when the method is non-LLM or no LLM was invoked. The summarizer always emits these columns so downstream tools see a stable schema.

Run metadata (optional)

The benchmark runner writes optional fields under metadata for harness observability and environment fingerprinting (auditability for papers and release):

Field Type Meaning
metadata.run_duration_wall_s number Wall-clock duration of the full run (all episodes) in seconds. When always_record_step_timing is true, this is always present; otherwise it is 0 for deterministic runs.
metadata.run_duration_episodes_per_s number Episodes per second (num_episodes / run_duration_wall_s).
metadata.step_timing object Present when LABTRUST_STEP_TIMING=1 or always_record_step_timing is true. Contains step timing aggregates (e.g. step_ms_mean, step_ms_p95). Used for capacity planning.
metadata.python_version string Python version at run time (e.g. 3.11.0).
metadata.platform string Platform identifier (e.g. win32, linux).

Reproducibility remains defined by seed, policy, and git metadata. Run metadata supports performance regression and run comparison.

Schema versions

  • results.v0.2: Normative results JSON (task, seeds, episodes with metrics). No change to existing fields or semantics. Optional coordination.llm per episode as above.
  • results.v0.3: Extends v0.2; same required fields; adds optional metrics (quantiles, ci_, device_utilization, device_queue_length_). Results JSON may be emitted as v0.2 (current) or v0.3 (when paper-grade fields are populated).