update phase-2 HLT timing script and use `alpaka` modifier in NGT workflows by mmusich · Pull Request #49821 · cms-sw/cmssw

mmusich · 2026-01-14T13:03:07Z

PR description:

Following the discussion had at the NGT meeting of Jan 13 2025 this PR implements:

use explicitly the alpaka process modifier in all the NGT scouting related workflows offload part of the HGCal reconstruction to GPUs

PR validation:

To be tested by the bot.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, no backport needed.

mmusich · 2026-01-14T13:03:16Z

type ngt

cmsbuild · 2026-01-14T13:03:35Z

cms-bot internal usage

cmsbuild · 2026-01-14T13:04:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49821/47477

cmsbuild · 2026-01-14T13:05:23Z

A new Pull Request was created by @mmusich for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv)
HLTrigger/Configuration (hlt)

@AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @cmsbuild, @miquork, @mmusich can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @SohamBhattacharya, @VourMa, @fabiocos, @makortel, @missirol, @rovere, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2026-01-14T13:05:48Z

test parameters:

enable = hlt_p2_integration, hlt_p2_timing
workflows = ph2_hlt

mmusich · 2026-01-14T13:06:00Z

@cmsbuild, please test

cmsbuild · 2026-01-14T14:20:43Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/50620/summary.html
COMMIT: 43410e9
CMSSW: CMSSW_16_1_X_2026-01-13-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49821/50620/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Failed RelVals

34634.999

----- Begin Fatal Exception 14-Jan-2026 14:46:42 CET-----------------------
An exception of category 'OutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'HLTriggerFinalPath'
   [2] Prefetching for module TriggerSummaryProducerAOD/'hltTriggerSummaryAOD'
   [3] Prefetching for module L1HPSPFTauProducer/'l1tHPSPFTauProducer'
   [4] Prefetching for module L1TPFCandMultiMerger/'l1tLayer1'
   [5] Prefetching for module L1TCorrelatorLayer1Producer/'l1tLayer1HGCal'
   [6] Calling method for module HGCalBackendLayer2Producer/'l1tHGCalBackEndLayer2Producer'
Exception Message:
TC X1 = 0.0713466 out of the seeding histogram bounds 0.076 - 0.58
----- End Fatal Exception -------------------------------------------------

mmusich · 2026-01-15T09:00:41Z

The updated timing result, adding the alpaka process modifier is odd:

mmusich · 2026-01-15T10:59:37Z

@cmsbuild, please test

cmsbuild · 2026-01-15T14:24:13Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/50652/summary.html
COMMIT: 43410e9
CMSSW: CMSSW_16_1_X_2026-01-14-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49821/50652/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially added 5 lines to the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 71
DQMHistoTests: Total histograms compared: 4580671
DQMHistoTests: Total failures: 1010
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4579641
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 70 files compared)
Checked 285 log files, 240 edm output root files, 71 DQM output files
TriggerResults: found differences in 3 / 69 workflows

cmsbuild · 2026-05-31T14:26:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49821/49587

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, [WIP] Prototype TruthInfo graph for truth history navigation and association studies #50913, [NGT] Add workflow to produce Phase2 NANOAOD with L1 and HLT objects #51004, Displaced Particle Gun #51006, Phase2-hgx367 Try to create a scenario with T37, C27, N17, I22, F9 #51080

cmsbuild · 2026-05-31T14:26:45Z

Pull request #49821 was updated. @AdrianoDee, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @cmsbuild, @kfjack, @miquork, @mmusich, @sroychow can you please check and sign again.

cmsbuild · 2026-05-31T14:27:27Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-49821/49588

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, [WIP] Prototype TruthInfo graph for truth history navigation and association studies #50913, [NGT] Add workflow to produce Phase2 NANOAOD with L1 and HLT objects #51004, Displaced Particle Gun #51006, Phase2-hgx367 Try to create a scenario with T37, C27, N17, I22, F9 #51080

mmusich · 2026-06-01T05:51:17Z

@cmsbuild, please test

cmsbuild · 2026-06-01T23:28:31Z

+1

Size: This PR adds an extra 44KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/53605/summary.html
COMMIT: 95e06f6
CMSSW: CMSSW_17_0_X_2026-05-31-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION,HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49821/53605/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 66
DQMHistoTests: Total histograms compared: 4598269
DQMHistoTests: Total failures: 29
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4598220
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 65 files compared)
Checked 276 log files, 232 edm output root files, 66 DQM output files
TriggerResults: found differences in 2 / 64 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34434.775_TTbar_14TeV+Run4D121_NGTScoutingCAExtensionMergeT5 step2 max memory diff 33.4 exceeds +/- 30.0 MiB

mmusich · 2026-06-02T09:55:51Z

The new effect of using the alpaka modifier in the NGT scouting workflows is:

$\color{Red}\Large{\textbf{+14\%}}$ GPU memory consumption:

around $\color{Green}\Large{\textbf{300ms}}$ faster execution:

mmusich · 2026-06-02T09:56:18Z

+hlt

see update phase-2 HLT timing script and use alpaka modifier in NGT workflows #49821 (comment)

cmsbuild · 2026-06-05T16:36:07Z

Milestone for this pull request has been moved to CMSSW_20_0_X. Please open a backport if it should also go in to CMSSW_17_0_X.

makortel · 2026-06-05T18:51:54Z

I just want to note that if this change is desired also to 17_0_X (Run 3 legacy) in addition of 20_1_X (Phase 2), please open a separate backport PR (you can use the same topic branch).

AdrianoDee · 2026-06-08T13:04:31Z

+pdmv

AdrianoDee · 2026-06-08T13:04:50Z

please test
(just as a pedantic refresh)

cmsbuild · 2026-06-08T13:05:00Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @ftenchini (and backports should be raised in the release meeting by the corresponding L2)

mmusich · 2026-06-08T13:10:46Z

@cmsbuild, please abort

mmusich · 2026-06-08T13:11:42Z

please abort

the timing test is non-functional at the moment in 20_0_X, see #51143.

mmusich · 2026-06-08T13:12:10Z

test parameters:

enable = hlt_p2_integration
workflows = ph2_hlt

mmusich · 2026-06-08T13:12:16Z

@cmsbuild, please test

cmsbuild · 2026-06-08T15:06:36Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-480d8a/53732/summary.html
COMMIT: 95e06f6
CMSSW: CMSSW_20_0_X_2026-06-08-1100/el8_amd64_gcc13
Additional Tests: HLT_P2_INTEGRATION
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/49821/53732/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 4 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 57
DQMHistoTests: Total histograms compared: 3647546
DQMHistoTests: Total failures: 42
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3647486
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 56 files compared)
Checked 242 log files, 197 edm output root files, 57 DQM output files
TriggerResults: found differences in 2 / 56 workflows

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34434.775_TTbar_14TeV+Run4D121_NGTScoutingCAExtensionMergeT5 step2 max memory diff 35.3 exceeds +/- 30.0 MiB

cmsbuild added this to the CMSSW_16_1_X milestone Jan 14, 2026

cmsbuild added hlt-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending ngt labels Jan 14, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 14, 2026

cmsbuild added tests-started and removed tests-pending labels Jan 14, 2026

cmsbuild mentioned this pull request Jan 14, 2026

Remove alpaka procModfier from workflows in which it is no longer useful #49755

Closed

cmsbuild added tests-rejected and removed tests-started labels Jan 14, 2026

This was referenced Jan 14, 2026

Deprecate autoCondPhase2 keys for inexistent geometries #49790

Merged

CPU vs. GPU for LST in HLT and updates to the offline #49832

Closed

cmsbuild added tests-started and removed tests-rejected labels Jan 15, 2026

cmsbuild removed the tests-started label Jan 15, 2026

mmusich force-pushed the mm_alpaka_in_ngt_scouting branch from dd7d44d to 95e06f6 Compare May 31, 2026 14:26

cmsbuild added code-checks-approved and removed code-checks-pending labels May 31, 2026

cmsbuild added tests-started and removed tests-pending labels Jun 1, 2026

cmsbuild added tests-approved and removed tests-started labels Jun 1, 2026

cmsbuild added hlt-approved and removed hlt-pending labels Jun 2, 2026

cmsbuild modified the milestones: CMSSW_17_0_X, CMSSW_20_0_X Jun 5, 2026

This was referenced Jun 8, 2026

Disable MC-based TSG tests in 20_0_X #51141

Open

Simplification in the definition of Phase 2 HLT workflows #51142

Open

Conversation

mmusich commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

mmusich commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Uh oh!

mmusich commented Jan 14, 2026

Uh oh!

mmusich commented Jan 14, 2026

Uh oh!

cmsbuild commented Jan 14, 2026

Failed RelVals

Uh oh!

mmusich commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Jan 15, 2026

Uh oh!

cmsbuild commented Jan 15, 2026

Comparison Summary

Uh oh!

cmsbuild commented May 31, 2026

Uh oh!

cmsbuild commented May 31, 2026

Uh oh!

cmsbuild commented May 31, 2026

Uh oh!

mmusich commented Jun 1, 2026

Uh oh!

cmsbuild commented Jun 1, 2026

Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

mmusich commented Jun 2, 2026

Uh oh!

mmusich commented Jun 2, 2026

Uh oh!

cmsbuild commented Jun 5, 2026

Uh oh!

makortel commented Jun 5, 2026

Uh oh!

AdrianoDee commented Jun 8, 2026

Uh oh!

AdrianoDee commented Jun 8, 2026

Uh oh!

cmsbuild commented Jun 8, 2026

Uh oh!

mmusich commented Jun 8, 2026

Uh oh!

mmusich commented Jun 8, 2026

Uh oh!

mmusich commented Jun 8, 2026

Uh oh!

mmusich commented Jun 8, 2026

Uh oh!

cmsbuild commented Jun 8, 2026

Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mmusich commented Jan 14, 2026 •

edited

Loading

cmsbuild commented Jan 14, 2026 •

edited

Loading

mmusich commented Jan 15, 2026 •

edited

Loading