Static destructor ordering in `Gpu::Device` for `external_stream_stack`

[ax3l]

Isolated from BLAST-WarpX/warpx#6801

There seems to be a destructor ordering issue on GPU for external_stream_stack.

[...] finalization can crash in amrex::Gpu::Device::Finalize() when it synchronizes streams whose backing C++ static objects (external_stream_stack) have already been destroyed.

We did not get a user-reported reproducer yet, but my guess is that this appears when a users loads a

• GPU-aware mpi4py that calls MPI_Init or
• a GPU library like cupy/dpnp

before amrex::Initialize() (importing WarpX in our case).

[WeiqunZhang]

My understanding is pywarpx uses atexit to register a call to amrex::Finalize, which calls amrex::Gpu::Device::Finalize. At the time the finalizers are called, the static objects in amrex have already been destroyed (by the runtime, not by amrex).

[WeiqunZhang]

But Codex says

  So for the normal case:

  python script exits
    -> CPython runs Python atexit callbacks
       -> PyWarpX calls amrex::Finalize()
    -> CPython tears down interpreter/modules
    -> process exits
    -> C/C++ static destructors run

and

the static-destruction explanation is plausible only for abnormal teardown/embedding/dlopen cases, not clearly for normal PyWarpX exit.

[ax3l]

Thanks for digging! It is a bit puzzling when this state would be met.

Maybe this is then related to the MPI dynamic library load issue that @zippylab debugged on Aurora...? More of an error handling case then.