cmake: Add dynamic test discovery to improve parallelism

[hebasto]

This PR implements the idea suggested in #1734 (review) and is based on the work from bitcoin/bitcoin#33483.

Here is an example of the ctest output:

$ ctest --test-dir build -j $(nproc)
Test project /home/hebasto/dev/secp256k1/secp256k1/build
        Start   1: secp256k1.noverify_tests.selftest_tests
        Start   2: secp256k1.noverify_tests.all_proper_context_tests
        Start   3: secp256k1.noverify_tests.all_static_context_tests
        Start   4: secp256k1.noverify_tests.deprecated_context_flags_test
<snip>
193/196 Test  #31: secp256k1.noverify_tests.ecmult_constants .........................   Passed    5.32 sec
194/196 Test #184: secp256k1.tests.ellswift_xdh_correctness_tests ....................   Passed    5.62 sec
195/196 Test #191: secp256k1.exhaustive_tests ........................................   Passed    6.97 sec
196/196 Test #126: secp256k1.tests.ecmult_constants ..................................   Passed    9.60 sec

100% tests passed, 0 tests failed out of 196

Label Time Summary:
secp256k1_example           =   0.02 sec*proc (5 tests)
secp256k1_exhaustive        =   6.97 sec*proc (1 test)
secp256k1_noverify_tests    =  23.77 sec*proc (95 tests)
secp256k1_tests             =  43.67 sec*proc (95 tests)

Total Test time (real) =  10.21 sec

For comparison, here is the output for the master branch on the same machine:

$ ctest --test-dir build -j $(nproc)
Test project /home/hebasto/dev/secp256k1/secp256k1/build
    Start 1: secp256k1_noverify_tests
    Start 2: secp256k1_tests
    Start 3: secp256k1_exhaustive_tests
    Start 4: secp256k1_ecdsa_example
    Start 5: secp256k1_ecdh_example
    Start 6: secp256k1_schnorr_example
    Start 7: secp256k1_ellswift_example
    Start 8: secp256k1_musig_example
1/8 Test #4: secp256k1_ecdsa_example ..........   Passed    0.00 sec
2/8 Test #5: secp256k1_ecdh_example ...........   Passed    0.00 sec
3/8 Test #6: secp256k1_schnorr_example ........   Passed    0.00 sec
4/8 Test #7: secp256k1_ellswift_example .......   Passed    0.00 sec
5/8 Test #8: secp256k1_musig_example ..........   Passed    0.00 sec
6/8 Test #3: secp256k1_exhaustive_tests .......   Passed    6.26 sec
7/8 Test #1: secp256k1_noverify_tests .........   Passed   14.31 sec
8/8 Test #2: secp256k1_tests ..................   Passed   31.65 sec

100% tests passed, 0 tests failed out of 8

Total Test time (real) =  31.65 sec

---

New Feature: As the number of tests has grown, the labels have been introduced to simplify test management. Now, one can run:

$ ctest --test-dir build -j $(nproc) -L example
Test project /home/hebasto/dev/secp256k1/secp256k1/build
    Start 192: secp256k1.example.ecdsa
    Start 193: secp256k1.example.ecdh
    Start 194: secp256k1.example.schnorr
    Start 195: secp256k1.example.ellswift
    Start 196: secp256k1.example.musig
1/5 Test #192: secp256k1.example.ecdsa ..........   Passed    0.00 sec
2/5 Test #193: secp256k1.example.ecdh ...........   Passed    0.00 sec
3/5 Test #194: secp256k1.example.schnorr ........   Passed    0.00 sec
4/5 Test #195: secp256k1.example.ellswift .......   Passed    0.00 sec
5/5 Test #196: secp256k1.example.musig ..........   Passed    0.00 sec

100% tests passed, 0 tests failed out of 5

Label Time Summary:
secp256k1_example    =   0.01 sec*proc (5 tests)

Total Test time (real) =   0.01 sec

or

$ ctest --test-dir build -j $(nproc) -LE tests
Test project /home/hebasto/dev/secp256k1/secp256k1/build
    Start 192: secp256k1.example.ecdsa
    Start 193: secp256k1.example.ecdh
    Start 194: secp256k1.example.schnorr
    Start 195: secp256k1.example.ellswift
    Start 196: secp256k1.example.musig
    Start 191: secp256k1.exhaustive_tests
1/6 Test #192: secp256k1.example.ecdsa ..........   Passed    0.00 sec
2/6 Test #193: secp256k1.example.ecdh ...........   Passed    0.00 sec
3/6 Test #194: secp256k1.example.schnorr ........   Passed    0.00 sec
4/6 Test #195: secp256k1.example.ellswift .......   Passed    0.00 sec
5/6 Test #196: secp256k1.example.musig ..........   Passed    0.00 sec
6/6 Test #191: secp256k1.exhaustive_tests .......   Passed    6.19 sec

100% tests passed, 0 tests failed out of 6

Label Time Summary:
secp256k1_example       =   0.01 sec*proc (5 tests)
secp256k1_exhaustive    =   6.19 sec*proc (1 test)

Total Test time (real) =   6.20 sec

[real-or-random]

The output is nice for sure, but wouldn't it be much simpler to just invoke the test binaries with -j?

My concern is about the list of tests here. It will require maintenance work, and it's easy to forget a test.

[sipa]

The output is nice for sure, but wouldn't it be much simpler to just invoke the test binaries with -j?

I don't think that's desirable, because it lacks the ability to share the granularity with other test targets.

Specifically, I'm thinking in the context of the Bitcoin Core build, which has tons of test cases of its own. The ideal scenario is that you have a single process pool (controlled by the -j argument to cmake), and it is used to run both the Bitcoin Core and libsecp256k1 (and other) tests. This will generally result in all processes making progress throughout the entire run, until cmake runs out of test tasks to schedule.

In contrast, if you push the task scheduling down to libsecp256k1's own test binary, you at best end up with a situation where you first run all Bitcoin Core tests, wait for those to complete (temporarily running out of tasks in the process), and then run libsecp256k1's tests with -j. I don't know if that's actually possible to do in cmake's supported generators, and more likely you'll end up in a situation where the entire libsecp256k1 test run (-j and all) is seen as a single process from the perspective of the build system - meaning it'd simultaneously run N-1 Bitcoin Core tests, plus 1 libsecp256k1 test (which itself runs in N processes), temporarily raising the parallellism to 2N-1.

[furszy]

Regarding the maintenance topic, wouldn't be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target).
The next step would be to connect this to run separate ctest targets in some way, which doesn't seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too.

[hebasto]

The output is nice for sure, but wouldn't it be much simpler to just invoke the test binaries with -j?

I don't think that's desirable, because it lacks the ability to share the granularity with other test targets.

Specifically, I'm thinking in the context of the Bitcoin Core build, which has tons of test cases of its own. The ideal scenario is that you have a single process pool (controlled by the -j argument to cmake), and it is used to run both the Bitcoin Core and libsecp256k1 (and other) tests. This will generally result in all processes making progress throughout the entire run, until cmake runs out of test tasks to schedule.

I share the same perspective.

[hebasto]

Regarding the maintenance topic, wouldn't be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target). The next step would be to connect this to run separate ctest targets in some way, which doesn't seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too.

@purpleKarrot's approach could be applied as a follow-up.

[hebasto]

Feedback from @furszy has been addressed.

[real-or-random]

Is there a reason why you have quotation marks in the first line here but not in the last?

[hebasto]

The first line has quotes to combine the quoted test cases into a single ctest's test "ecmult".

In the last line, all mentioned test cases and modules are wrapped as its own ctest's tests.

[purpleKarrot]

I take @real-or-random's question as a proof that procedural logic in CMakeLists.txt is hard to reason about and should be avoided.

[purpleKarrot]

If we switch to automated test discovery in a follow-up, we will no longer have the ability to combine test cases. Maybe we should keep them separated so that the follow-up will not change the number of tests again?

[real-or-random]

Maybe we should keep them separated so that the follow-up will not change the number of tests again?

Sounds good, let's keep it simple.

For test ordering, we could perhaps simply make sure that automated discovery runs the tests in the order they are listed by tests -l.

[furszy]

For test ordering, we could perhaps simply make sure that automated discovery runs the tests in the order they are listed by tests -l.

We could reorder them internally based on their execution time, so the slow ones run first.
It is just matter of moving a few lines in the registry anyway, very simple.

[purpleKarrot]

We could reorder them internally based on their execution time, so the slow ones run first.
It is just matter of moving a few lines in the registry anyway, very simple.

The order in which they are registered to CTest is not the order they will be executed. If you want slow tests to run first, you need to set the COST property.

[real-or-random]

Oh, I see, also fine! I suggest we just run the test suite once on some dev's machine and hard-code the resulting numbers.

[furszy]

Created #1788 so we don’t have to worry about the test costs for now. The longest running test is just an all-in-one test that we can split very easily (the gains of it are very noticeable).

[furszy]

Can we simplify anything now that #1788 was merged?

[purpleKarrot]

The target name tests is very generic and likely to cause clashes when the project is used as a subproject. This is orthogonal to the PR, but should be kept in mind for a follow up.

[hebasto]

Regarding the maintenance topic, wouldn't be better to call --list_tests to fetch the test names instead of manually adding them one by one? We could make --list_tests=2 output the tests names separated by commas to make your life easier at the parsing side.

I experimented a bit with this idea and was able to dynamically generate a file containing all the tests names. See https://github.com/furszy/secp256k1/tree/2025_ci_concurrent_tests (just need to run the tests_discover target). The next step would be to connect this to run separate ctest targets in some way, which doesn't seem to be so simple because the binary does not exist during configure time but ctest targets are added there.. (chicken-egg problem). Maybe we could execute a custom script that reads the file and launches per target ctest runs dynamically too.

Reworked. Using the dynamic test discovery now.

[furszy]

nice! will review soon.

[furszy]

it would be nice to explain what LABELS are.

[hebasto]

From Professional CMake: A Practical Guide 21st Edition:

Selecting tests individually by name or number can become cumbersome if a large set of related tests needs to be executed. Tests can be assigned an arbitrary list of labels using the LABELS test property, and then tests can be selected by these labels. The -L and -LE options are analogous to the -R and -E options respectively, except they operate on test labels rather than test names.
...
Labels not only enable convenient grouping for test execution, they also provide grouping for basic execution time statistics. As seen in the example output above, the ctest command prints a label summary when any test in the set of executed tests has its LABELS property set. This allows the developer to get an idea how each label group is contributing to the overall test time.

[furszy]

This assumes properties come in pairs, but someone could:

discover_tests(${exe_name}
      DISCOVERY_ARGS        "--list_tests"
      DISCOVERY_MATCH       "^\\t\\\\[ *[0-9]+\\\\] ([^ ].*)$"
      TEST_NAME_REPLACEMENT "secp256k1.${exe_name}.\\\\1"
      TEST_ARGS_REPLACEMENT "--target=\\\\1 --log=1"
      PROPERTIES
        LABELS "secp256k1_${exe_name}"
        NO_TIMEOUT   // <-- property with no value
)

It would be good to check that the value exist before accessing it, and fail with a clear error if not: "Property 'name' has no associated value. Must be specified as name/value pair" or something similar.

[hebasto]

Every set property has a value. The syntax in the snippet above is simply incorrect. Compare it with the syntax of the set_tests_properties command.

[furszy]

Aside from the minimal comments just left, this works nicely.

For reviewers, just run:

ctest -j <threads_num>

[hebasto]

@furszy

Thank you for the review! Your feedback has been addressed.

[furszy]

Tested ACK 8354618

[purpleKarrot]

ACK 8354618

Reviewed and tested.