-
Notifications
You must be signed in to change notification settings - Fork 239
Introduce perf-report.yml #712
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
atuchin-m
wants to merge
15
commits into
master
Choose a base branch
from
introduce-perf-report
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+352
−2
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
8b0f0d0
introduce perf report
atuchin-m 6d47b40
Fix kptr_restrict
atuchin-m 505b9e5
Remove label when the report is ready
atuchin-m bb2563d
merge bench and release profiles
atuchin-m c7c9acc
fix inferno-flamegraph args
atuchin-m 570fed5
always remove the label
atuchin-m bd2f57a
fix perf call
atuchin-m d302935
exclude */brave-list-deserialize
atuchin-m f25070d
combine report
atuchin-m 0e795ae
resize svg
atuchin-m 9637ec8
use a direct URL
atuchin-m cabd395
use iframe for svg
atuchin-m 1a4845c
adjust bench set and texts
atuchin-m e1a1a07
remove --colordiffusion
atuchin-m 2fcf380
Copilot suggestions
atuchin-m File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| """Combine flamegraph SVGs (and the critcmp table) into one self-contained HTML file.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import html | ||
| import pathlib | ||
| import re | ||
| import subprocess | ||
| import sys | ||
|
|
||
| SVG_RE = re.compile(r"<svg\b.*?</svg>", re.DOTALL | re.IGNORECASE) | ||
|
|
||
|
|
||
| def extract_svg(path: pathlib.Path) -> str: | ||
| text = path.read_text(encoding="utf-8", errors="replace") | ||
| match = SVG_RE.search(text) | ||
| if not match: | ||
| raise SystemExit(f"no <svg> element in {path}") | ||
| svg = match.group(0) | ||
| # Drop fixed size so the SVG fills its iframe; inferno's fluiddrawing | ||
| # mode picks up the viewport via clientWidth/clientHeight. | ||
| svg = re.sub(r'\s(width|height)="[^"]*"', "", svg, count=2) | ||
| if "style=" not in svg[:80]: | ||
| svg = svg.replace("<svg", '<svg style="width:100%;height:100%"', 1) | ||
| return svg | ||
|
|
||
|
|
||
| def svg_iframe(path: pathlib.Path) -> str: | ||
| """Embed the SVG in an iframe so its scripts run in an isolated document. | ||
|
|
||
| Inlining several flamegraphs into one page breaks search/zoom: they all | ||
| share ids like #search and document.getElementsByTagName('svg')[0]. | ||
| """ | ||
| inner = ( | ||
| "<!DOCTYPE html><html><head><meta charset=utf-8></head>" | ||
| '<body style="margin:0;height:100vh">' | ||
| f"{extract_svg(path)}" | ||
| "</body></html>" | ||
| ) | ||
| # srcdoc is an HTML attribute value. | ||
| escaped = html.escape(inner, quote=True) | ||
| title = html.escape(path.name) | ||
| return ( | ||
| f'<iframe title="{title}" srcdoc="{escaped}" ' | ||
| f'loading="lazy" style="width:100%;height:100vh;border:0;display:block"></iframe>' | ||
| ) | ||
|
|
||
|
|
||
| def main() -> int: | ||
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("--out-dir", type=pathlib.Path, required=True) | ||
| parser.add_argument("--base-sha", required=True) | ||
| parser.add_argument("--head-sha", required=True) | ||
| parser.add_argument("--run-url", required=True) | ||
| parser.add_argument("--repo-dir", type=pathlib.Path, required=True) | ||
| args = parser.parse_args() | ||
|
|
||
| table = subprocess.run( | ||
| ["critcmp", "base", "head"], | ||
| check=True, | ||
| cwd=args.repo_dir, | ||
| capture_output=True, | ||
| text=True, | ||
| ).stdout.strip() | ||
|
|
||
| svgs = sorted(args.out_dir.glob("*.svg")) | ||
| if not svgs: | ||
| print(f"no .svg files in {args.out_dir}", file=sys.stderr) | ||
| return 1 | ||
|
|
||
| sections: list[str] = [] | ||
| for path in svgs: | ||
| sections.append( | ||
| f'<section id="{html.escape(path.stem)}">\n' | ||
| f"<h2>{html.escape(path.name)}</h2>\n" | ||
| f"{svg_iframe(path)}\n" | ||
| f"</section>" | ||
| ) | ||
|
|
||
| toc = "\n".join( | ||
| f'<li><a href="#{html.escape(path.stem)}">{html.escape(path.name)}</a></li>' | ||
| for path in svgs | ||
| ) | ||
|
|
||
| doc = f"""<!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="utf-8"> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1"> | ||
| <title>Detailed performance report</title> | ||
| <style> | ||
| html, body {{ margin: 0; color: #111; font-family: system-ui, sans-serif; }} | ||
| .intro {{ padding: 1.5rem; }} | ||
| pre {{ background: #f4f4f4; padding: 1rem; overflow: auto; }} | ||
| nav ul {{ columns: 2; }} | ||
| section {{ border-top: 1px solid #ddd; }} | ||
| section h2 {{ margin: 0; padding: 0.75rem 1.5rem; background: #f4f4f4; }} | ||
| </style> | ||
| </head> | ||
| <body> | ||
| <div class="intro"> | ||
| <h1>Performance report</h1> | ||
| <p><code>{html.escape(args.base_sha)}</code> (base) → | ||
| <code>{html.escape(args.head_sha)}</code> (head)</p> | ||
| <p>Run: <a href="{html.escape(args.run_url)}">{html.escape(args.run_url)}</a></p> | ||
| <h2>Benchmarks</h2> | ||
| <pre>{html.escape(table)}</pre> | ||
| <nav> | ||
| <h2>Flamegraphs</h2> | ||
| <ul> | ||
| {toc} | ||
| </ul> | ||
| </nav> | ||
| </div> | ||
| {chr(10).join(sections)} | ||
| </body> | ||
| </html> | ||
| """ | ||
|
|
||
| html_path = args.out_dir / "perf-report.html" | ||
| html_path.write_text(doc, encoding="utf-8") | ||
|
|
||
| # PR comment body: numbers + pointer to the HTML artifact. | ||
| report_md = f"""## Performance report | ||
|
|
||
| `{args.base_sha}` (base) → `{args.head_sha}` (head) | ||
|
|
||
| ``` | ||
| {table} | ||
| ``` | ||
| """ | ||
| (args.out_dir / "report.md").write_text(report_md, encoding="utf-8") | ||
| print(report_md) | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main()) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Runs the timing benchmarks and stores the results as a criterion baseline, | ||
| # so that two revisions can be compared with `critcmp`. | ||
| # | ||
| # Usage: ./.github/scripts/run-perf-benchmarks.sh <baseline-name> | ||
| # | ||
| # Expects RUSTFLAGS / CARGO_PROFILE_*_DEBUG from the caller (perf-report.yml) | ||
| # so benchmarks and profiles share the same build fingerprint. | ||
|
|
||
| set -eu | ||
|
|
||
| BASELINE="$1" | ||
|
|
||
| cargo bench --bench bench_matching 'rule-match-browserlike/brave-list' -- --save-baseline "$BASELINE" | ||
| cargo bench --bench bench_matching rule-match-first-request -- --save-baseline "$BASELINE" | ||
| cargo bench --bench bench_rules 'blocker_new/brave-list' -- --save-baseline "$BASELINE" | ||
| cargo bench --bench bench_cosmetic_matching -- --save-baseline "$BASELINE" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Records a CPU profile for the benchmarks of interest and renders a flamegraph. | ||
| # | ||
| # Usage: ./.github/scripts/run-perf-profile.sh <label> <output-dir> | ||
| # label: "base" or "head", used to name the produced files | ||
| # output-dir: where the .svg / .folded files are written | ||
| # | ||
| # Linux-only: records with `perf` around `cargo bench` (same binary as | ||
| # run-perf-benchmarks.sh) and writes demangled `.folded` stacks. Once both | ||
| # `base` and `head` folded files exist for a benchmark, a differential SVG is | ||
| # written too. | ||
| # | ||
| # Expects RUSTFLAGS / CARGO_PROFILE_*_DEBUG from the caller (perf-report.yml). | ||
| # Requires: perf, inferno, rustfilt. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| if [[ "$(uname -s)" != "Linux" ]]; then | ||
| echo "run-perf-profile.sh requires Linux (perf)" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| LABEL="$1" | ||
| OUT_DIR="$(cd "$2" && pwd)" | ||
|
|
||
| # `--profile-time` makes criterion run the benchmark for N seconds without its | ||
| # own measurement/analysis machinery, which keeps the profile clean. | ||
| PROFILE_TIME="${PROFILE_TIME:-10}" | ||
|
|
||
| build_diff_flamegraph() { | ||
| local name="$1" | ||
| local base="${OUT_DIR}/${name}.base.folded" | ||
| local head="${OUT_DIR}/${name}.head.folded" | ||
| [[ -f "$base" && -f "$head" ]] || return 0 | ||
|
|
||
| echo "Building differential flamegraph for ${name}..." | ||
| inferno-diff-folded "$base" "$head" \ | ||
| | inferno-flamegraph --title "${name} (base -> head)" \ | ||
| > "${OUT_DIR}/${name}.diff.svg" | ||
| rm -f "$base" "$head" | ||
| } | ||
|
|
||
| profile_one() { | ||
| local name="$1" bench="$2" filter="$3" | ||
| local prefix="${OUT_DIR}/${name}.${LABEL}" | ||
|
|
||
| echo "Profiling ${name} (${bench} ${filter})..." | ||
|
|
||
| perf record --call-graph fp -o perf.data -- \ | ||
| cargo bench --bench "$bench" "$filter" -- --profile-time "$PROFILE_TIME" | ||
|
|
||
| # perf's default demangler is C++-oriented and turns Rust names into the | ||
| # unreadable `_E14bench_function...` form. Keep symbols mangled, collapse, | ||
| # then demangle with rustfilt (rustc-demangle). | ||
| perf script -i perf.data --no-demangle \ | ||
| | inferno-collapse-perf \ | ||
| | rustfilt > "${prefix}.folded" | ||
| inferno-flamegraph --title "${name} (${LABEL})" \ | ||
| < "${prefix}.folded" > "${prefix}.svg" | ||
| rm -f perf.data | ||
| build_diff_flamegraph "$name" | ||
| } | ||
|
|
||
| profile_one matching bench_matching 'rule-match-browserlike/brave-list$' | ||
| profile_one startup bench_rules 'blocker_new/brave-list$' | ||
| profile_one cosmetic bench_cosmetic_matching 'cosmetic-class-id-match/brave-list' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| # On-demand performance report for a PR. | ||
| # | ||
| # Compares the PR head against its merge base and produces: | ||
| # * a benchmark table with the before/after diff (posted as a PR comment) | ||
| # * clickable flamegraphs for both revisions | ||
| # * a differential flamegraph of the CPU profile | ||
| # | ||
| # Trigger it by adding the `perf-report` label to a PR, or manually from the | ||
| # Actions tab. | ||
| name: Performance report | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [labeled] | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: write | ||
|
|
||
| jobs: | ||
| report: | ||
| name: Performance report | ||
| if: github.event_name == 'workflow_dispatch' || github.event.label.name == 'perf-report' | ||
| runs-on: ubuntu-latest-4-cores-public | ||
|
|
||
| env: | ||
| # --no-rosegment: required for accurate perf stacks with lld (Rust >= 1.90). | ||
| # force-frame-pointers: cheap unwinding for `perf record --call-graph fp`. | ||
| # See: https://github.com/flamegraph-rs/flamegraph/blob/main/README.md | ||
| RUSTFLAGS: "-Cforce-frame-pointers=yes -Clink-arg=-Wl,--no-rosegment" | ||
| # Keep release and bench equivalent (bench inherits release) and leave | ||
| # symbols in both so perf can resolve stacks. | ||
| CARGO_PROFILE_RELEASE_DEBUG: "true" | ||
| CARGO_PROFILE_BENCH_DEBUG: "true" | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| fetch-depth: 0 | ||
| ref: ${{ github.event.pull_request.head.sha || github.ref }} | ||
|
|
||
| - name: Install perf and profiling tools | ||
| run: | | ||
| # No runner image ships `perf`, so it has to be installed here. | ||
| sudo apt-get update | ||
| sudo apt-get install -y linux-tools-common linux-tools-generic | ||
| # /usr/bin/perf is a wrapper that looks for tools matching the running | ||
| # kernel, which the runner does not have packages for. Use the binary | ||
| # from linux-tools-generic directly instead. | ||
| sudo ln -sf "$(ls -d /usr/lib/linux-tools/*/perf | tail -1)" /usr/local/bin/perf | ||
| perf --version | ||
|
|
||
| - name: Disable kernel perf event restrictions | ||
| run: | | ||
| sudo sysctl -w kernel.perf_event_paranoid=0 | ||
| sudo sysctl -w kernel.kptr_restrict=0 | ||
|
|
||
| - name: Install profiling tools | ||
| run: cargo install --locked inferno critcmp rustfilt | ||
|
|
||
| - name: Determine merge base | ||
| id: base | ||
| run: | | ||
| BASE_REF="${{ github.event.pull_request.base.ref || github.event.repository.default_branch }}" | ||
| git fetch --no-tags origin "$BASE_REF" | ||
| echo "sha=$(git merge-base FETCH_HEAD HEAD)" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Prepare | ||
| run: | | ||
| mkdir -p /tmp/perf-report | ||
| # The scripts are used for both revisions, so keep a copy outside the | ||
| # working tree that `git checkout` will not touch. | ||
| cp -r .github/scripts /tmp/perf-scripts | ||
|
|
||
| - name: Benchmark and profile the merge base | ||
| run: | | ||
| git checkout --detach ${{ steps.base.outputs.sha }} | ||
| /tmp/perf-scripts/run-perf-benchmarks.sh base | ||
| /tmp/perf-scripts/run-perf-profile.sh base /tmp/perf-report | ||
|
|
||
| - name: Benchmark and profile the PR head | ||
| run: | | ||
| git checkout --detach ${{ github.event.pull_request.head.sha || github.sha }} | ||
| /tmp/perf-scripts/run-perf-benchmarks.sh head | ||
| /tmp/perf-scripts/run-perf-profile.sh head /tmp/perf-report | ||
|
|
||
| - name: Build the report | ||
| run: | | ||
| /tmp/perf-scripts/build-perf-report.py \ | ||
| --out-dir /tmp/perf-report \ | ||
| --repo-dir "$GITHUB_WORKSPACE" \ | ||
| --base-sha "${{ steps.base.outputs.sha }}" \ | ||
| --head-sha "${{ github.event.pull_request.head.sha || github.sha }}" \ | ||
| --run-url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | ||
| # Drop individual SVGs; the HTML embeds them. | ||
| rm -f /tmp/perf-report/*.svg | ||
|
|
||
| - name: Upload perf-report.html | ||
| id: upload | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| path: /tmp/perf-report/perf-report.html | ||
| archive: false | ||
|
|
||
| - name: Comment on the PR | ||
| if: github.event_name == 'pull_request' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| ARTIFACT_URL: ${{ steps.upload.outputs.artifact-url }} | ||
| run: | | ||
| { | ||
| cat /tmp/perf-report/report.md | ||
| echo | ||
| echo "Flamegraphs: [${ARTIFACT_URL}](${ARTIFACT_URL})." | ||
| } > /tmp/perf-report/comment.md | ||
| gh pr comment ${{ github.event.pull_request.number }} --body-file /tmp/perf-report/comment.md | ||
|
|
||
| - name: Remove perf-report label | ||
| if: always() && github.event_name == 'pull_request' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: gh pr edit ${{ github.event.pull_request.number }} --remove-label perf-report |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reported by reviewdog 🐶
[opengrep] Consider possible security implications associated with subprocess module.
Source: https://semgrep.dev/r/gitlab.bandit.B404
Cc @thypon
Please consider an alternative approach that avoids this security concern, or request a review from the sec-team on slack.