Write benchmarks for current DS-MPS impementation

[LSchueler]

[jeilealr]

Benchmarking update

I decided to add an ASV benchmark suite for the current GSTools/DS-MPS workflows, plus a small cProfile helper to inspect where runtime is spent in the current checkout.

Why ASV and cProfile?

ASV, Airspeed Velocity, is used for repeatable performance benchmarking across commits. It is especially useful when we want to know whether a code change made a workflow faster or slower over time. In this implementation, I also added a backend comparison between the Cython fallback and the Rust backend.

ASV is used by well-known scientific Python projects such as NumPy, SciPy, pandas, scikit-image, and Astropy to track performance regressions across versions.

cProfile is complementary. It does not compare commits. Instead, it profiles one run in the current checkout and shows where time is spent inside that workflow. I used it after ASV to understand which internal functions dominate runtime.

A walkthrough of the benchmark implementation is available in the README in my fork: benchmarks/README.md. I think this is a solid first step to track the current performance status of the repository in a reproducible way.

Initial results

The implementation produces several ASV plots and terminal summaries. I will share the main results here, with two representative plots.

The benchmark was run across the last five commits on main, comparing both backends. The differences between commits are not large, but the backend differences are clear.

ASV time example: kriging workflow

The y-axis is runtime in microseconds, so lower is faster. The x-axis is the git commit history included in the ASV run. Each line is one combination of backend and kriging case:

• backend=cython_fallback: GSTools running without the Rust core backend.
• backend=rust_core: GSTools running with gstools_core.
• small_30x500: 30 conditioning points, 500 target prediction points.
• large_120x2000: 120 conditioning points, 2,000 target prediction points.
• extra_large_360x6000: 360 conditioning points, 6,000 target prediction points.

The lines are mostly flat across the plotted commits, which means the benchmark is stable over the selected history and does not show a strong performance regression in this range.

The main result is that the Rust backend becomes increasingly beneficial as the kriging workload grows. For the small case, both backends are very close and the Rust backend is slightly slower, likely because fixed overhead dominates such a small problem. For larger cases, the Rust backend clearly improves runtime.

The same pattern appears across the other workflow benchmarks:

• Variogram estimation: Rust backend is about 6x-7x faster.
• Global kriging:
  • small case: Rust is slightly slower because overhead dominates.
  • large case: Rust is about 3x faster.
  • extra-large case: Rust is about 7x-8x faster.
• SRF generation: Rust is about 4x-6x faster.
• CondSRF: Rust is about 2x faster.

ASV peak memory example: structured random field

The y-axis is peak memory usage in megabytes. The x-axis compares the two backends:

• cython_fallback: GSTools running without the Rust core backend.
• rust_core: GSTools running with gstools_core.

The benchmark case is:

• srf_structured_randmeth: spatial random field generation on a structured 64 x 64 grid using the default randomization method, RandMeth.

The main result is that both backends use almost the same peak memory, around 117-119 MB. The Rust backend is slightly higher in this plot, but the difference is small compared with the total memory footprint. This is also the general pattern in the current results: the Rust backend gives clear runtime improvements, but it does not significantly reduce peak memory usage for these benchmark cases.

cProfile results

For the current GSTools workflows, the profiling results showed that the dominant time is mostly inside compiled/numerical routines:

• Variogram estimation: gstools_core.gstools_core.variogram_unstructured
• Kriging: gstools_core.gstools_core.calc_field_krige_and_variance, plus kriging matrix setup/inversion
• CondSRF: comparatively small/fast in the current benchmark cases

This was useful because it showed that, for those workflows, Python-level cleanup or vectorization is probably not the main optimization path.

[MuellerSeb]

Next step:

• use config.NUM_THREADS = 1 to compare cython/rust with 1 thread
• create PR to GSTools main branch

[jeilealr]

Hi @MuellerSeb and @LSchueler

I have created PR #410 fixing the problem with Cython not being able to run multiple threads in parallel, as expected Rust is still faster, I was able to only test this on macOS. I will try later to test it on other systems.

SIDE-NOTE: I wanted to add Assignees and Reviewers, but it seems I don't have the permissions to do so. Could someone give me permission level to do so?

[jeilealr]

Last version of the benchmarking is ready.

[jeilealr]

Hi @MuellerSeb @n0228a and @LSchueler,

Benchmarking has been updated following your suggestions.

The benchmark suite was developed in my fork on the benchmarking branch).

Results are stored at gstools-benchmarks-asv in my GitHub account. Once deployed in the main GSTools repo, they will live at GeoStat-Framework/GSTools-Benchmarks.

The results in my GitHub are live and automatically updated via GitHub Pages. The same setup will apply once the benchmark repository moves to GeoStat-Framework.

Co-kriging benchmarks have not been added yet, this can be done later with no issues once the branch is merged into main.

I tried to keep the file structure as lean as possible, but we still have 4 GitHub Actions workflow files and a handful of supporting Python scripts. Happy to discuss this further.

I added a bot that runs in GitHub Actions and posts a summary comment on each PR, flagging whether significant performance changes were detected. It uses ASV's Mann-Whitney U statistical test with a 5% threshold: changes above 5% are only reported if they are statistically significant — either as a regression warning or an improvement note.

We now track both commits and tags in the benchmark history.

Any thoughts?