Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
d1dc04f
refactor(lift-teardown): Phase A — relocate resource-doc/version disc…
hongwei1 May 29, 2026
84c3d9f
refactor(lift-teardown): Phase B — delete Http4sLiftWebBridge, wire J…
hongwei1 May 29, 2026
f1d3544
chore(c1): drop ResourceDoc.partialFunction (OBPEndpoint) field
hongwei1 May 29, 2026
44fc2bd
chore(c2): remove Lift Web session/locale config from Boot.scala
hongwei1 May 29, 2026
edf523f
chore(C3c): drop routes/OBPEndpoint from version traits and aggregators
hongwei1 May 29, 2026
c0aee38
chore(C3a): remove dead Lift RestHelper dispatch; strip self-type fro…
hongwei1 May 29, 2026
82694c4
chore(c3b): collapse OBPRestHelper; disable dead ConnectorEndpoints d…
hongwei1 May 29, 2026
df5b1e6
chore(phase-c): retire OBPEndpoint type and dead Lift dispatch code
hongwei1 May 29, 2026
c26aa52
chore(c3-cleanup): drop dead Lift deps from AccountsHelper, CustomAPI…
hongwei1 May 29, 2026
f23126e
chore(c3d-httpparam): replace net.liftweb.http.provider.HTTPParam wit…
hongwei1 May 29, 2026
b5e673e
chore(c3d): remove dead writeEndpointMetric(TimeSpan,Long,ResourceDoc…
hongwei1 May 29, 2026
f855555
chore(c3d): remove S.addCookie/S.findCookie and Lift-HTTP imports fro…
hongwei1 May 29, 2026
0ad1e3d
chore(c4): replace LiftRules.loadResourceAsString with classpath stre…
hongwei1 May 29, 2026
e5c9a5d
chore(c3d): replace LiftRules/TransientRequestMemoize with direct Res…
hongwei1 May 29, 2026
ae5308e
chore(c3d): remove LiftResponse/InMemoryResponse/JsonResponse/HTTPCoo…
hongwei1 May 29, 2026
8a6f826
chore(c3d): remove net.liftweb.http.S from OAuth and Helper; simplify…
hongwei1 May 29, 2026
68f5d9f
chore(c3d): remove dead Lift dispatch code from ConnectorEndpoints
hongwei1 May 29, 2026
05cc294
chore(c4): remove LiftRules/S/InMemoryResponse/PlainTextResponse from…
hongwei1 May 29, 2026
dad9b45
chore(c3d): clean net.liftweb.http.{_,JsRaw,RestContinuation} from AP…
hongwei1 May 29, 2026
99f7082
chore(c3d): remove net.liftweb.http from GatewayLogin, dauth, directl…
hongwei1 May 29, 2026
96f7e12
chore(c3d): replace net.liftweb.http.JsonResponse with code.api.JsonR…
hongwei1 May 29, 2026
f059869
test: update HTTPParam import to code.api.util.APIUtil.HTTPParam
hongwei1 May 29, 2026
49ede46
chore(C2): remove dead LiftRules.exceptionHandler and uriNotFound fro…
hongwei1 May 29, 2026
ec6f8f0
chore(C3b): remove dead jsonResponseBoxToJsonResponse implicit from O…
hongwei1 May 29, 2026
cd8cefa
chore(C2): remove LiftRules from Boot.scala; replace with JVM shutdow…
hongwei1 May 29, 2026
b7d685d
chore(C5): replace S.request with Empty in AuthUser.getCurrentUser
hongwei1 May 29, 2026
0f0cc5c
chore(C5): remove dead portal surface from AuthUser; drop net.liftweb…
hongwei1 May 29, 2026
01c046d
chore: Remove dead code for `registerRoutes`, correct outdated Lift b…
hongwei1 May 29, 2026
327d22e
refactor/rename OBPEndpointIO to Http4sEndpointIO
hongwei1 May 29, 2026
fc29fbf
docs(CLAUDE): correct stale Http4sLiftWebBridge references after brid…
hongwei1 May 29, 2026
61704da
test: remove Lift-vs-http4s comparison and dead/redundant tests
hongwei1 May 29, 2026
b3e6398
test: dynamic per-shard free ports in run_tests_parallel.sh
hongwei1 May 29, 2026
7c87d96
test: serialise concurrent obp-commons ~/.m2 installs with mkdir lock
hongwei1 May 29, 2026
229e326
docs: update LIFT_HTTP4S_MIGRATION.md — BG v1.3 and UK Open Banking a…
hongwei1 Jun 1, 2026
7fba456
docs: rewrite LIFT_HTTP4S_MIGRATION.md to reflect completed migration
hongwei1 Jun 1, 2026
250e67c
Merge remote-tracking branch 'OBP/develop' into refactor/RemoveLiftwe…
hongwei1 Jun 1, 2026
0c71ea6
Merge remote-tracking branch 'Hongwei/develop' into refactor/RemoveLi…
hongwei1 Jun 2, 2026
400c61c
Merge remote-tracking branch 'OBP/develop' into refactor/RemoveLiftwe…
hongwei1 Jun 2, 2026
963a85f
fix: drop stale null first-arg from ResourceDoc calls
hongwei1 Jun 2, 2026
48b7c9e
test: poll for server readiness instead of fixed 2s startup sleep
hongwei1 Jun 2, 2026
97f4f3a
docs: remove stale Lift HTTP references from project docs
hongwei1 Jun 2, 2026
5e3752f
test: comment out transactionRequests_enabled-gated ignore() placehol…
hongwei1 Jun 2, 2026
f5cfcd2
docs: fix stale ResourceDoc(null) in migration Before/After table
hongwei1 Jun 2, 2026
4937a21
Revert "test: comment out transactionRequests_enabled-gated ignore() …
hongwei1 Jun 2, 2026
570088c
test: create transactions/transaction-requests on demand (opt-in per …
hongwei1 Jun 2, 2026
c76c19f
test: per-shard Redis key-namespace isolation to fix rate-limit/cache…
hongwei1 Jun 2, 2026
8748965
test: add CounterpartyTest to transaction-data whitelist (fix determi…
hongwei1 Jun 2, 2026
094967d
refactor(ci): reframe per-test speed report as unit/pure vs integration
hongwei1 Jun 3, 2026
bc90f76
test: remove migration-era property suites, consolidate v5 SystemViews
hongwei1 Jun 3, 2026
551a359
test: drop empty feature block, de-stale v7 resource-docs titles
hongwei1 Jun 3, 2026
1bb2eca
ci: speed up build via Zinc cache, 8 shards, goal-only test runs
hongwei1 Jun 3, 2026
b746dc4
Remove final net.liftweb.http from OBP-API source
hongwei1 Jun 3, 2026
bd3309c
ci: revert test runs to mvn test (goal-only caused false green)
hongwei1 Jun 3, 2026
0b2dbe3
docs: update LIFT_HTTP4S_MIGRATION after the final AuthUser.logout cl…
hongwei1 Jun 3, 2026
4acc46a
ci: run tests via process-resources scalatest:test (skip lifecycle, k…
hongwei1 Jun 3, 2026
b204432
ci(experiment): split into 12 shards to test if it beats 8
hongwei1 Jun 3, 2026
3022269
Revert "ci(experiment): split into 12 shards to test if it beats 8"
hongwei1 Jun 3, 2026
bd71b95
ci: finalize at 8 shards + clean build (drop 12-shard & incremental-c…
hongwei1 Jun 3, 2026
f5e04be
ci: rebalance 8 shards - isolate v1_2_1, move berlin/mgmt/metrics to …
hongwei1 Jun 3, 2026
7e8c17e
ci: pin redis to redis:7-alpine to reduce docker-pull flakiness
hongwei1 Jun 3, 2026
b2cf02e
ci(Phase A): move fat-jar build off the test critical path (~100s fas…
hongwei1 Jun 3, 2026
15253aa
Revert "ci(Phase A): move fat-jar build off the test critical path (~…
hongwei1 Jun 3, 2026
b4f4ab3
fix(oidc): wrap callback provisioning in one DB transaction + add suc…
hongwei1 Jun 3, 2026
29f437b
fix(ci): run code.api-root OIDC + alive-check suites; document droppe…
hongwei1 Jun 3, 2026
174ff7f
fix(shutdown): merge two concurrent shutdown hooks into one ordered hook
hongwei1 Jun 3, 2026
9162ee9
fix(oidc): scrub client_secret/tokens from debug logs + document sess…
hongwei1 Jun 3, 2026
a9d03e3
fix(ci): correct catch-all log label (was hardcoded "shard 4")
hongwei1 Jun 3, 2026
81e5bea
fix(test): use portable timeout binary (timeout/gtimeout) in run_test…
constantine2nd Jun 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 161 additions & 93 deletions .github/scripts/test_speed_report.py
Original file line number Diff line number Diff line change
@@ -1,157 +1,222 @@
#!/usr/bin/env python3
"""
Parse surefire XML reports from all shards and print an http4s-vs-Lift
per-test speed table to stdout (plain text) and, if GITHUB_STEP_SUMMARY
is set, append a markdown version to that file.
Parse surefire XML reports from all shards and print a per-test speed table.

The Lift -> http4s migration is complete: *every* API version (v1.2.1 through
v7.0.0) is served by http4s. There is no Lift HTTP code left, so the old
"http4s vs Lift" split is meaningless. The meaningful axis now is **execution
model**:

* unit/pure — no embedded server; pure logic / JSON-factory / route-matcher
/ middleware tests. These are the speed win of the migration.
* integration — boots a real server (test class extends ServerSetup) or a
self-started http4s server; pays DB/HTTP cost per test.

Two tables are printed:
1. By execution model (unit/pure vs integration) — the migration KPI.
2. By API version (API v1 .. v7, all http4s) — per-version cost.

A suite counts as integration when its test class extends `ServerSetup`
(detected by scanning obp-api/src/test/scala) or is one of the self-starting
http4s server suites in HTTP4S_INTEGRATION_SUITES. Everything else is unit/pure.

Usage:
python3 test_speed_report.py <reports-root-dir>

<reports-root-dir> should contain the extracted artifacts from all shards,
e.g. after downloading test-reports-shard{1,2,3} into one directory.
<reports-root-dir> should contain the extracted artifacts from all shards.
Override the source root (used only for integration detection) with
OBP_TEST_SRC_ROOT; if sources are not found, the execution-model split degrades
gracefully (suites counted as integration) and the by-version table still prints.
"""

from __future__ import annotations

import os
import re
import sys
import glob
import xml.etree.ElementTree as ET
from collections import defaultdict

# ---------------------------------------------------------------------------
# Classification
# ---------------------------------------------------------------------------

# These suites run a real embedded server — they pay the same DB/HTTP cost
# as Lift integration tests.
# Self-starting http4s integration suites that boot a server WITHOUT extending
# ServerSetup (so the source scan can't detect them) — list them explicitly.
HTTP4S_INTEGRATION_SUITES = {
"code.api.v7_0_0.Http4s700RoutesTest",
"code.api.v7_0_0.Http4s700TransactionTest",
"code.api.http4sbridge.Http4sLiftBridgePropertyTest",
"code.api.http4sbridge.Http4sServerIntegrationTest",
"code.api.v5_0_0.Http4s500SystemViewsTest",
}

DEFAULT_SRC_ROOT = "obp-api/src/test/scala"


# ---------------------------------------------------------------------------
# Integration detection by source scan (does the test class extend ServerSetup?)
# ---------------------------------------------------------------------------

def build_integration_map(src_root):
"""Return ({fqClassName: extendsServerSetup}, scan_ok)."""
fqmap = {}
if not os.path.isdir(src_root):
return fqmap, False
for root, _dirs, files in os.walk(src_root):
for fname in files:
if not fname.endswith(".scala"):
continue
try:
with open(os.path.join(root, fname), encoding="utf-8", errors="ignore") as fh:
txt = fh.read()
except OSError:
continue
pm = re.search(r'^\s*package\s+([\w.]+)', txt, re.M)
pkg = pm.group(1) if pm else ""
# For each `class X ... {`, inspect the parents portion (everything
# up to the first brace) and check whether it mentions ServerSetup.
for cm in re.finditer(r'\bclass\s+(\w+)\b(.*?)\{', txt, re.S):
cls, parents = cm.group(1), cm.group(2)
fqmap[f"{pkg}.{cls}"] = ("ServerSetup" in parents)
return fqmap, True


def is_integration(fq, fqmap):
if fq in HTTP4S_INTEGRATION_SUITES:
return True
if fq in fqmap:
return fqmap[fq]
# Unknown (class/file-name mismatch or degraded scan): default to
# integration so we never overstate the unit/pure win.
return True

def categorize(suite_name: str) -> str | None:
"""Return a display category or None to exclude from the table."""
# http4s integration (real server)
if suite_name in HTTP4S_INTEGRATION_SUITES:
return "http4s v7 — integration"

# http4s unit/pure (no server) — everything http4s-flavoured that isn't
# in the integration set above
if (
"Http4s" in suite_name
or "http4s" in suite_name
or "v7_0_0" in suite_name
or suite_name.startswith("code.api.util.http4s.")
or suite_name.startswith("code.api.berlin.group.v2.Http4sBGv2")
):
return "http4s v7 — unit/pure"
# ---------------------------------------------------------------------------
# API version from the suite's package
# ---------------------------------------------------------------------------

# Lift versions
for v in ("v6_0_0", "v5_1_0", "v5_0_0", "v4_0_0", "v3_1_0", "v3_0_0",
"v2_2_0", "v2_1_0", "v2_0_0", "v1_4_0", "v1_3_0", "v1_2_1"):
if v in suite_name:
major = v[1] # "1" … "6"
return f"Lift v{major}"
_VERSIONS = ("v7_0_0", "v6_0_0", "v5_1_0", "v5_0_0", "v4_0_0", "v3_1_0", "v3_0_0",
"v2_2_0", "v2_1_0", "v2_0_0", "v1_4_0", "v1_3_0", "v1_2_1")

return None # exclude (util, berlin group non-http4s, etc.)

def api_version(fq):
for v in _VERSIONS:
if v in fq:
return f"API v{v[1]}" # "1" .. "7"
return "other"


# ---------------------------------------------------------------------------
# Parse
# ---------------------------------------------------------------------------

def collect(reports_root: str) -> dict:
stats = defaultdict(lambda: {"tests": 0, "time": 0.0})
def collect(reports_root, fqmap):
by_model = defaultdict(lambda: {"tests": 0, "time": 0.0})
by_version = defaultdict(lambda: {"tests": 0, "time": 0.0})

pattern = os.path.join(reports_root, "**", "TEST-*.xml")
for path in glob.glob(pattern, recursive=True):
try:
root = ET.parse(path).getroot()
name = root.get("name", "")
name = root.get("name", "")
tests = int(root.get("tests", 0))
time = float(root.get("time", 0))
t = float(root.get("time", 0))
if tests == 0:
continue
cat = categorize(name)
if cat is None:
continue
stats[cat]["tests"] += tests
stats[cat]["time"] += time
model = "integration" if is_integration(name, fqmap) else "unit/pure"
by_model[model]["tests"] += tests
by_model[model]["time"] += t
ver = api_version(name)
by_version[ver]["tests"] += tests
by_version[ver]["time"] += t
except Exception:
pass

return stats
return by_model, by_version


# ---------------------------------------------------------------------------
# Render
# ---------------------------------------------------------------------------

CATEGORY_ORDER = [
"http4s v7 — unit/pure",
"http4s v7 — integration",
"Lift v6",
"Lift v5",
"Lift v4",
"Lift v3",
"Lift v2",
"Lift v1",
]


def render_plain(stats: dict) -> str:
col_w = [25, 7, 12, 10]
sep = "+-" + "-+-".join("-" * w for w in col_w) + "-+"
hdr = "| " + " | ".join(
h.center(w) for h, w in zip(
["Category", "Tests", "Total time", "Avg/test"], col_w
)
) + " |"
MODEL_ORDER = ["unit/pure", "integration"]
VERSION_ORDER = ["API v7", "API v6", "API v5", "API v4", "API v3",
"API v2", "API v1", "other"]


def _table(stats, order, col_w=(24, 7, 12, 10)):
sep = "+-" + "-+-".join("-" * w for w in col_w) + "-+"
hdr = "| " + " | ".join(h.center(w) for h, w in zip(
["Category", "Tests", "Total time", "Avg/test"], col_w)) + " |"
lines = [sep, hdr, sep]
for cat in CATEGORY_ORDER:
for cat in order:
if cat not in stats:
continue
d = stats[cat]
d = stats[cat]
avg = d["time"] / d["tests"] if d["tests"] else 0
row = "| " + " | ".join([
lines.append("| " + " | ".join([
cat.ljust(col_w[0]),
str(d["tests"]).rjust(col_w[1]),
f"{d['time']:.1f}s".rjust(col_w[2]),
f"{avg:.3f}s".rjust(col_w[3]),
]) + " |"
lines.append(row)
]) + " |")
lines.append(sep)
return "\n".join(lines)


def render_markdown(stats: dict) -> str:
rows = ["## http4s v7 vs Lift — per-test speed",
"",
"| Category | Tests | Total time | Avg/test |",
"|---|---:|---:|---:|"]
for cat in CATEGORY_ORDER:
if cat not in stats:
def render_plain(by_model, by_version, scan_ok):
out = ["All API versions (v1-v7) are served by http4s — the split is by",
"execution model, not framework.\n",
"By execution model (unit/pure = no server, integration = embedded server):"]
if not scan_ok:
out.append(" (source scan unavailable — suites counted as integration)")
out.append(_table(by_model, MODEL_ORDER))
out += ["", "By API version:", _table(by_version, VERSION_ORDER)]

u = by_model.get("unit/pure")
i = by_model.get("integration")
if u and i and u["tests"] and i["tests"]:
ua = u["time"] / u["tests"]
ia = i["time"] / i["tests"]
if ua > 0:
out += ["",
f"--> unit/pure tests are {ia/ua:.0f}x faster per test than "
f"integration tests ({ua:.3f}s vs {ia:.3f}s)."]
return "\n".join(out)


def render_markdown(by_model, by_version, scan_ok):
rows = ["## Per-test speed — all endpoints served by http4s", "",
"_All API versions (v1-v7) run on http4s. The split below is by "
"execution model, not framework._", "",
"### By execution model", ""]
if not scan_ok:
rows += ["> source scan unavailable — suites counted as integration", ""]
rows += ["| Category | Tests | Total time | Avg/test |", "|---|---:|---:|---:|"]
for cat in MODEL_ORDER:
if cat not in by_model:
continue
d = by_model[cat]
avg = d["time"] / d["tests"] if d["tests"] else 0
rows.append(f"| {cat} | {d['tests']} | {d['time']:.1f}s | {avg:.3f}s |")

rows += ["", "### By API version", "",
"| Category | Tests | Total time | Avg/test |", "|---|---:|---:|---:|"]
for cat in VERSION_ORDER:
if cat not in by_version:
continue
d = stats[cat]
d = by_version[cat]
avg = d["time"] / d["tests"] if d["tests"] else 0
rows.append(f"| {cat} | {d['tests']} | {d['time']:.1f}s | {avg:.3f}s |")

# Highlight ratio
u = stats.get("http4s v7 — unit/pure")
lift_times = [stats[c]["time"] for c in CATEGORY_ORDER if c.startswith("Lift") and c in stats]
lift_tests = [stats[c]["tests"] for c in CATEGORY_ORDER if c.startswith("Lift") and c in stats]
if u and lift_tests:
lift_avg = sum(lift_times) / sum(lift_tests)
unit_avg = u["time"] / u["tests"]
rows += [
"",
f"> **Unit/pure tests are {lift_avg/unit_avg:.0f}× faster than Lift integration tests** "
f"({unit_avg:.3f}s vs {lift_avg:.3f}s per test).",
]
u = by_model.get("unit/pure")
i = by_model.get("integration")
if u and i and u["tests"] and i["tests"]:
ua = u["time"] / u["tests"]
ia = i["time"] / i["tests"]
if ua > 0:
rows += ["",
f"> **Unit/pure tests are {ia/ua:.0f}x faster per test than "
f"integration tests** ({ua:.3f}s vs {ia:.3f}s). This is the "
f"migration win: logic that used to need a running server is "
f"now pure unit-tested."]
return "\n".join(rows)


Expand All @@ -164,16 +229,19 @@ def render_markdown(stats: dict) -> str:
print(f"Usage: {sys.argv[0]} <reports-root-dir>", file=sys.stderr)
sys.exit(1)

stats = collect(sys.argv[1])
if not stats:
src_root = os.environ.get("OBP_TEST_SRC_ROOT", DEFAULT_SRC_ROOT)
fqmap, scan_ok = build_integration_map(src_root)

by_model, by_version = collect(sys.argv[1], fqmap)
if not by_model and not by_version:
print("No matching surefire XML reports found.", file=sys.stderr)
sys.exit(0)

print(render_plain(stats))
print(render_plain(by_model, by_version, scan_ok))

summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
if summary_path:
with open(summary_path, "a") as f:
f.write("\n")
f.write(render_markdown(stats))
f.write(render_markdown(by_model, by_version, scan_ok))
f.write("\n")
Loading
Loading