Introduce (mini) unit test framework

[furszy]

Early Note:
Don’t be scared by the PR’s line changes count — most of it’s just doc or part of the test framework API.

Context:
Currently, all tests run single-threaded sequentially and the library lacks the ability to specify which test (or group of tests) you would like to run. This is not only inconvenient as more tests are added but also time consuming during development and affects downstream projects that may want to parallelize the workload (such as Bitcoin-Core CI).

PR Goal:
Introduce a lightweight, extensible C89 unit test framework with no dynamic memory allocations, providing a structured way to register, execute, and report tests. The framework supports named command-line arguments in -key=value form, parallel test execution across multiple worker processes, granular test selection (selecting tests either by name or by module name), and time accumulation reports.

The introduced framework supports:

• -help or -h: display list of available commands along with their descriptions.
• -jobs=<num>: distribute tests across multiple worker processes (default: sequential if 0).
• -target=<name> or -t=<name>: run only specific tests by name; can be repeated to select multiple tests.
• -target=<module name>, -t=<module> Run all tests within a specific module (can be provided multiple times)
• -seed=<hex>: set a specific RNG seed (defaults to random if unspecified).
• -iterations=<n>: specify the number of iterations.
• -list_tests: display list of available tests and modules you can run.
• -log=<0|1>: enable or disable test execution logging (default: 0 = disabled).

Beyond these features, the idea is to also make future developments smoother, as adding new tests require only a single entry in the central test registry, and new command-line options can be introduced easily by extending the framework’s parse_arg() function.

Compatibility Note:
The framework continues accepting the two positional arguments previously supported (iterations and seed), ensuring existing workflows remain intact.

Testing Notes:
Have fun. You can quickly try it through ./tests -j=<workers_num> for parallel execution or ./tests -t=<test_name> to run a specific test (call ./tests -print_tests to display all available tests and modules).

Extra Note:
I haven't checked the exhaustive tests file so far, but I will soon. For now, this only runs all tests declared in the tests binary.

Testing Results: (Current master branch vs PR in seconds)

• Raspberry Pi 5: master ~100 s → PR ~38 s (5 jobs)
• MacBook Pro M1: master ~30 s → PR ~10 s (6 jobs)

[real-or-random]

Nice!

I looked at existing unit test frameworks in the past, but nothing seemed appropriate for us. There are not that many for C, and they were either overkill or too simple (just handful of ifdefs) so they didn't add any functionality. I thought writing our own is too annoying (or I was just lazy). But the framework is ~300 lines, that seems fine to me.

[real-or-random]

see also #1211

[jonasnick]

Thanks @furszy. I played around with this a little bit. It reduces the execution time on my machine from 26 seconds to 10 seconds (-jobs=16). Very nice! Some observations:

• It would be helpful if a helptext would be output when tests is run with -h or --help.
• I think showing all the tests that have passed is a bit overkill. I'm already assuming that the tests pass if they do not show up in the output. I only need to see tests that don't pass.
• Maybe a future PR can autodetect the number of cores and set -jobs automatically by default?
• There's an -iter command line flag, but the test output shows "test count". It would be better if we were consistent.

[jonasnick]

• The available "targets" seem to be a bit arbitary. Maybe we can try to put them into groups (perhaps similar to the grouping in #1211)?

  ./tests -print_tests
  Available tests (58):
      --------------------------------------------------
      [  1] selftest_tests
      [  2] all_proper_context_tests
      [  3] all_static_context_tests
      [  4] deprecated_context_flags_test
      [  5] scratch_tests
      [  6] int128_tests
      [  7] ctz_tests
      [  8] modinv_tests
      [  9] inverse_tests
      [ 10] hsort_tests
      [ 11] sha256_known_output_tests
      [ 12] sha256_counter_tests
      [ 13] hmac_sha256_tests
      [ 14] rfc6979_hmac_sha256_tests
      [ 15] tagged_sha256_tests
      [ 16] scalar_tests
      [ 17] field_half
      [ 18] field_misc
      [ 19] field_convert
      [ 20] field_be32_overflow
      [ 21] fe_mul
      [ 22] sqr
      [ 23] sqrt
      [ 24] ge
      [ 25] gej
      [ 26] group_decompress
      [ 27] ecmult_pre_g
      [ 28] wnaf
      [ 29] point_times_order
      [ 30] ecmult_near_split_bound
      [ 31] ecmult_chain
      [ 32] ecmult_constants
      [ 33] ecmult_gen_blind
      [ 34] ecmult_const_tests
      [ 35] ecmult_multi_tests
      [ 36] ec_combine
      [ 37] endomorphism_tests
      [ 38] ec_pubkey_parse_test
      [ 39] eckey_edge_case_test
      [ 40] eckey_negate_test
      [ 41] ecdh_tests
      [ 42] ec_illegal_argument_tests
      [ 43] pubkey_comparison
      [ 44] pubkey_sort
      [ 45] random_pubkeys
      [ 46] ecdsa_der_parse
      [ 47] ecdsa_sign_verify
      [ 48] ecdsa_end_to_end
      [ 49] ecdsa_edge_cases
      [ 50] ecdsa_wycheproof
      [ 51] extrakeys_tests
      [ 52] schnorrsig_tests
      [ 53] musig_tests
      [ 54] ellswift_tests
      [ 55] secp256k1_memczero_test
      [ 56] secp256k1_is_zero_array_test
      [ 57] secp256k1_byteorder_tests
      [ 58] cmov_tests
  

[furszy]

Thanks for the review jonasnick!

Thanks @furszy. I played around with this a little bit. It reduces the execution time on my machine from 26 seconds to 10 seconds (-jobs=16). Very nice!

Awesome :). I think we can actually do even better, will do some changes.

• It would be helpful if a helptext would be output when tests is run with -h or --help.

The help message was actually already there, but for -help only (with a single - and I forgot to add it to the PR description).
I just pushed support for -h as well.

• I think showing all the tests that have passed is a bit overkill. I'm already assuming that the tests pass if they do not show up in the output. I only need to see tests that don't pass.

Sure. Will hide the logging behind a -log option (or -silent if we want the opposite behavior).
From my experience, logging sometimes helps spot regressions or areas that can be improved (like when a test suddenly takes longer than usual), and it’s also reassuring to see them run and pass when you have a large number of them.
I’ve also been thinking about adding per-test execution time loggings, might be a good opportunity to include it too.

• Maybe a future PR can autodetect the number of cores and set -jobs automatically by default?

I'm not sure we want that. Sequential execution is usually "standard" on any system because we don’t know what else the user might be running. Picking a number of parallel tasks automatically (even if it is a low number) could hang the CPU or even make it run slower than sequential if the system is overloaded.

• There's an -iter command line flag, but the test output shows "test count". It would be better if we were consistent.

Sure 👍🏼. That was carried over from the previous code; will improve it.

[furszy]

The available "targets" seem to be a bit arbitary. Maybe we can try to put them into groups (perhaps similar to the grouping in #1211)?

Yeah. Just reworked the framework to support registering and running groups of tests in a generic manner. This means we can now run specific tests and/or specific groups of tests via the -target/-t arg.

On top of that, made the framework reusable across binaries and improved the overall API (we can now easily connect the tests_exhaustive binary to it too — probably something for a follow-up), along with improvements to the consumers’ structure, enforcing consistency and a specific pattern that all consumers will follow.

Other than that, the -print_tests option was improved to display the available modules and tests.

A simple usage example:
./tests -print_tests → pick any module (like field) and run: ./tests -t=field
You can also combine this with -j=<num_workers>, plus specify multiple targets as needed.

[hebasto]

While reviewing #1732, I realized that the commit "refactor: move 'gettime_i64()' to tests_common.h" might have been a suboptimal approach. The tests do not really require nanosecond precision, and the standard clock() should suffice. Moreover, changing the gettime_i64 implementation would require unnecessary modifications to test source code and/or build targets.

[furszy]

While reviewing #1732, I realized that the commit "refactor: move 'gettime_i64()' to tests_common.h" might have been a suboptimal approach. The tests do not really require nanosecond precision, and the standard clock() should suffice. Moreover, changing the gettime_i64 implementation would require unnecessary modifications to test source code and/or build targets.

After your CMake changes, the approach seems acceptable for the test framework as well. You are handling everything at the build-system level right now. If #1732 weren’t changing the function’s name, the test code wouldn’t require any changes.

Apart from that, it seems we could have a separate function that retrieves the elapsed time rather than the CPU time. A small wrapper that internally calls to the other time function, providing CLOCK_REALTIME or CLOCK_MONOTONIC. But that seems like an easy change to #1732.
I think we should first decide whether #1732 is the final way to go. This adaptation can be done at the end as it would be minimal.