LST: add LSTGeometry package and associated ESProducer#50679
Conversation
|
cms-bot internal usage |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50679/48907
|
|
A new Pull Request was created by @ariostas for master. It involves the following packages:
The following packages do not have a category, yet: RecoTracker/LSTGeometry @Martin-Grunewald, @Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @mmusich, @srimanob can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
test parameters:
|
|
@cmsbuild, please test |
|
-1 Failed Tests: UnitTests HLTP2Timing Failed Unit TestsI found 1 errors in the following unit tests: ---> test test-das-selected-lumis had ERRORS Comparison SummarySummary:
Max Memory Comparisons exceeding threshold@cms-sw/core-l2 , I found 17 workflow step(s) with memory usage exceeding the error threshold: Expand to see workflows ...
|
|
Is ~190 MB increase in memory usage expected? |
That seems a bit high, but it's likely. I'll double-check. Either way, it is only temporarily. Most of it is freed once the maps are constructed. |
According to the monitoring the peak memory usage would increase by ~190 MB, and thus freeing it afterwards doesn't help much if the job was killed because of going over the limit. |
|
test parameters:
|
|
@cmsbuild, please test Maybe one round of profiling tests would be worth it. |
cmsbuild
added
tests-pending
code-checks-pending
and removed
tests-rejected
requires-external
code-checks-approved
labels
Rebased to fix conflicts. I'll get back to looking into this.
Thank you! I'll see what I can learn from the |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50679/49455
|
|
Pull request #50679 was updated. @Martin-Grunewald, @Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @mmusich, @srimanob can you please check and sign again. |
|
@cmsbuild please test IIUC, the HLT timing tests moved to L4 GPUs with more memory. Let's see if the tests complete this time. |
|
+1 Size: This PR adds an extra 16KB to repository HLT P2 Timing: chart Comparison SummarySummary:
Max Memory Comparisons exceeding threshold@cms-sw/core-l2 , I found 18 workflow step(s) with memory usage exceeding the error threshold: Expand to see workflows ...
|
the test did pass. 
interestingly also the CPU memory usage profile is altered (for the worse): 
|
Yeah, so quick update on this. I did some more experiments with the setup where the producer runs, but the product is not used. If I trivialize the producer by immediately returning an empty product, then the issue goes away. So then I tried adding a sleep timer to simulate the producer doing some work. When the producer takes about a second you start seeing some vram increase (although not as consistent). With about 5 second delay you see exactly the plots above. So the issue is indeed due to kernel scheduling. The producer takes around 6-9 seconds (depending on the machine), so I seems like that's too long and all streams end up waiting around for it to finish. So then all streams sync and launch kernels at the same time causing a spike in vram usage.
I think it's the same cause, since it looks similar to vram usage with the caching allocator disabled. #50679 (comment) I'm currently working on speeding up the producer to get the time under a second, which should solve this issue. |
|
Hi @ariostas I haven't followed the whole thread, but I found your latest comments interesting, and I may need some clarifications to understand them better.
Is that an For an For an |
|
@fwyzard said
it could happen if there is no other work that can be done by all the Events (i.e. insufficient concurrency within an Event at that point in the schedule) |
|
@fwyzard yeah, it's an |
|
@ariostas ah, thanks for the clarification. Is the spike is caused by the |
|
@fwyzard the spike is caused by the |
|
OK, thanks for confirming it. |
|
Milestone for this pull request has been moved to CMSSW_20_0_X. Please open a backport if it should also go in to CMSSW_17_0_X. |
|
(I assume this PR does not need a backport to 17_0_X (Run 3 legacy)) |
|
Milestone for this pull request has been moved to CMSSW_20_1_X. Please open a backport if it should also go in to CMSSW_20_0_X. |
8 participants
This PR adds a new
RecoTracker/LSTGeometrypackage containing the module map computation used by the LST algorithm. Currently, the maps are pre-computed by the code in https://github.com/SegmentLinking/LSTGeometry and they are stored in https://github.com/cms-data/RecoTracker-LSTCore. This PR allows for the on-the-fly computation of these maps via an ESProducer, ensuring that they stay consistent with the tracker geometry being used.This is the last major task in #46746.
c.c. @slava77