gpu-direct: make it optional and have fallback solution with device-host explicit transfers

[svalat]

Handle when there is no-gpu-direct

The patch mostly wrap the MPI interface to make the transfers manually between the host & device when GPU direct is not available.

In order to avoid allocating an host buffer for every transfers, the IdefixArrayXD classes has been extended as IdefixCommArrayXD which contains the standard array on device, and optionally (if GPU direct is disabled and if running on GPU) a host buffer to make the communications.

In order to apply in the code, all the MPI communication calls (MPI_Send, MPI_Recv, MPI_Start....) need to be replaced by the idefix::MPI_* corresponding frunctions which expect an IdefixCommArray as parameter.

It means that we should not continue to use .data() to feed the MPI calls (the compiler will complain for the idefix::MPI_* functions).

When the wrapper is called it ensure to sync the communication buffer in & out during the communication process so it is transparent for the calling code.

One care

You should not use directly the IdefixCommArrayXDin a GPU loop (get a compiler warning in case due to host array in use). This is way the patch uses the .deviceView() to get it casted in a friendly way for the GPU.

Note: in case you still keep it that way, it should not be a problem appart the warning in the compilation output because the host array will not be used in the kernel because not accessible (private).

Note: this semantic is only used if we communicate with the buffer, all the rest of the code using standard IdefixArray is not concerned.

Case of persitent comms

For the persistent comms and async comms, an new idefix::MPI_Request_XD has been defined to carry the information on how to sync the communication buffer with its device counter part when required. Again if you use idefix::MPI_* just follow the compiler which will force you to type correctly using idefix::MPI_Request_XD instead of MPI_Request.

Validation

To validate, you need to :

# Edit src/mpi/mpiView.hpp and enable the define : FORCE_TRANSFERS_ALL_TIME
# Then
PYTHONPATH=$PWD ./test.py -subdir ./test/MHD/ -k "mpi" -cmake "Idefix_MPI_GPU_DIRECT=OFF" -cuda

Note: Also make a second validation, by playing with the various internal modes of idefix about MPI (persistent, async, sync) => to be sure that all paths are still working (not yet validated).

Issue

I currently pass most of the MPI test except the AsixFluxTube which crash => investigation ongoing.

When having GPU direct

in this case, the IdefixCommArrayGpuDirect implementation is used, which is just an empty stub which just does nothing.

Questions

1. When making a MPI_waitall with multiple comms, I start all the host-device transfers asynchronously and using Kokkos::fence(). I can also avoid it to get simpler code as we have max 2 comms at this location.
2. I currently named the MPI_* API the same inside the idefix:: namespace, we can also name them all MPI_View_ to clearly separate the concept. To be discussed what is best.
3. I patched Mpi::CheckConfig() so it uses the copies when not having GPU Direct, is it what we want ?

TODO

• Fix the issue on Axis cases
• Validate all configuration (mpi sync/async/persistent)
• Validate on a non gpu-direct condition.
• Some minor cleanup