Hang in Torch copy_to_device cudaStreamSynchronize

[albertz]

This occured after training a bit. It's the first time I think that I see this. Node n23g0014 in RWTH cluster.

py-spy:

Thread 390647 (active): "MainThread"
    copy_to_device (returnn/returnn/torch/frontend/_backend.py:169)
    copy_to_device (returnn/returnn/frontend/device.py:40)
    get_dyn_size_ext_for_device (returnn/returnn/tensor/_dim_extra.py:837)
    get_size_tensor (returnn/returnn/tensor/_dim_extra.py:1941)
    pad (returnn/returnn/torch/frontend/_backend.py:538)
    pad (returnn/returnn/frontend/array_.py:519)
    aed_training (denoising_lm_aed_2024/models/aed_v3.py:252)
    _returnn_train_step (i6_experiments/users/zeyer/train_v4.py:248)
    _run_step (returnn/returnn/torch/engine.py:932)
    train_epoch (returnn/returnn/torch/engine.py:465)
    train (returnn/returnn/torch/engine.py:269)
    execute_main_task (returnn/returnn/__main__.py:543)
    main (returnn/returnn/__main__.py:744)
    <module> (returnn/rnn.py:11)

gdb:

#0  0x0000148ff604fff7 in ?? () from /lib64/libcuda.so.1
#1  0x0000148ff5e1a6b3 in ?? () from /lib64/libcuda.so.1
#2  0x0000148ff5e6b87b in ?? () from /lib64/libcuda.so.1
#3  0x0000148ff6951388 in ?? () from /lib64/libcuda.so.1
#4  0x0000148ff69517a8 in ?? () from /lib64/libcuda.so.1
#5  0x0000148ff5e3faec in ?? () from /lib64/libcuda.so.1
#6  0x0000148ff5eecb42 in ?? () from /lib64/libcuda.so.1
#7  0x0000148ff6912375 in ?? () from /lib64/libcuda.so.1
#8  0x0000148ff5f80e67 in ?? () from /lib64/libcuda.so.1
#9  0x00001490c01cf3a5 in ?? ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#10 0x00001490c02307d8 in cudaStreamSynchronize ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#11 0x000014906b917fae in at::native::copy_kernel_cuda(at::TensorIterator&, bool) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so
#12 0x00001490a5857252 in at::native::copy_impl(at::Tensor&, at::Tensor const&, bool) [clone .isra.0] ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#13 0x00001490a5858bac in at::native::copy_(at::Tensor&, at::Tensor const&, bool) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#14 0x00001490a651fd35 in at::_ops::copy_::call(at::Tensor&, at::Tensor const&, bool) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#15 0x00001490a5b303f5 in at::native::_to_copy(at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, std::optional<c10::MemoryFormat>) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#16 0x00001490a695488d in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, std::optional<c10::MemoryFormat>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd___to_copy>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, std::optional<c10::MemoryFormat> > >, at::Tensor (at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, std::optional<c10::MemoryFormat>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>, bool, std::optional<c10::MemoryFormat>) ()
...

nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.195.03             Driver Version: 570.195.03     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100                    On  |   00000000:1B:00.0 Off |                    0 |
| N/A   38C    P0            121W /  700W |   40161MiB /  95830MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100                    On  |   00000000:2C:00.0 Off |                    0 |
| N/A   44C    P0            253W /  700W |   32999MiB /  95830MiB |     58%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100                    On  |   00000000:9D:00.0 Off |                    0 |
| N/A   37C    P0            121W /  700W |   32507MiB /  95830MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100                    On  |   00000000:AD:00.0 Off |                    0 |
| N/A   45C    P0            263W /  700W |   32581MiB /  95830MiB |     38%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A          390647      C   ...vs/py3.12-torch2.7/bin/python      40134MiB |
|    1   N/A  N/A          255143      C   ...vs/py3.12-torch2.7/bin/python      32872MiB |
|    2   N/A  N/A          255142      C   ...vs/py3.12-torch2.7/bin/python      32478MiB |
|    3   N/A  N/A          236563      C   ...vs/py3.12-torch2.7/bin/python      32552MiB |
+-----------------------------------------------------------------------------------------+

Probably some hardware issue. Not sure we can do much about it.

[albertz]

Note sure if #323 is maybe related. (There it is cuEventSynchronize instead of cudaStreamSynchronize.) But maybe just in the way that both are hardware issues, nothing more.

[albertz]

Maybe related:

Another hang, weirdly at the same time, same cluster (RWTH), but different node (w23g0009):

Python:

Thread 367714 (active): "MainThread"
    ctc_loss (torch/nn/functional.py:3086)
    ctc_loss (returnn/returnn/torch/frontend/_backend.py:748)
    ctc_loss (returnn/returnn/frontend/loss.py:98)
    aed_training (denoising_lm_aed_2024/models/aed_v3.py:207)
    _returnn_train_step (i6_experiments/users/zeyer/train_v4.py:248)
    _run_step (returnn/returnn/torch/engine.py:932)
    train_epoch (returnn/returnn/torch/engine.py:465)
    train (returnn/returnn/torch/engine.py:269)
    execute_main_task (returnn/returnn/__main__.py:543)
    main (returnn/returnn/__main__.py:744)
    <module> (returnn/rnn.py:11)

And native:

#0  0x000014e37c579940 in clock_gettime@plt () from /lib64/libcuda.so.1
#1  0x000014e37c724994 in ?? () from /lib64/libcuda.so.1
#2  0x000014e37c63fd8c in ?? () from /lib64/libcuda.so.1
#3  0x000014e37c6ecb42 in ?? () from /lib64/libcuda.so.1
#4  0x000014e37d112375 in ?? () from /lib64/libcuda.so.1
#5  0x000014e37c780e67 in ?? () from /lib64/libcuda.so.1
#6  0x000014e4468293a5 in ?? ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#7  0x000014e44688a7d8 in cudaStreamSynchronize ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#8  0x000014e3f1f71fae in at::native::copy_kernel_cuda(at::TensorIterator&, bool) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so
#9  0x000014e42beb1252 in at::native::copy_impl(at::Tensor&, at::Tensor const&, bool) [clone .isra.0] ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#10 0x000014e42beb2bac in at::native::copy_(at::Tensor&, at::Tensor const&, bool) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so
#11 0x000014e42cb79d35 in at::_ops::copy_::call(at::Tensor&, at::Tensor const&, bool) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so

I.e. again cudaStreamSynchronize.

It's a bit weird that this happened really about the same time. But a different node. Maybe some recent GPU driver update or so is causing this?

[albertz]

We could maybe implement a watchdog process.

[albertz]

I realize: I have custom CUDA kernels in this job (e.g. edit distance). The hang in cudaStreamSynchronize could mean that the GPU is hanging in any previously async scheduled CUDA kernel.

So I attached with cuda-gdb. On the CPU side, for this particular example that I have now:

(cuda-gdb) bt                                                                                                                              
#0  0x000015384f23fae1 in ?? () from /lib64/libcuda.so.1                                                                                   
#1  0x000015384f2ecb42 in ?? () from /lib64/libcuda.so.1                                                                                   
#2  0x000015384fd12375 in ?? () from /lib64/libcuda.so.1                                                                                   
#3  0x000015384f380e67 in ?? () from /lib64/libcuda.so.1                                                                                   
#4  0x00001539194243a5 in ?? ()                                                                                                            
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#5  0x00001539194857d8 in cudaStreamSynchronize ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#6  0x00001538c4a8d200 in at::native::_local_scalar_dense_cuda(at::Tensor const&)::{lambda()#1}::operator()() const ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so
#7  0x00001538c4a8dc06 in at::native::_local_scalar_dense_cuda(at::Tensor const&) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so

CUDA kernels:

(cuda-gdb) info cuda kernels    
No CUDA kernels.

CUDA devices:

(cuda-gdb) info cuda devices
No CUDA devices.

[albertz]

When setting CUDA_VISBLE_DEVICES to the same as from the training proc, cuda-gdb seems to behave a bit different. Now, I get the warning:

Timed out waiting for the CUDA API to initialize.

I'm not really sure whether I'm using cuda-gdb maybe incorrectly. Or maybe the GPU device is really stuck, and it's not some faulty kernel.

[albertz]

After I triggered a restart via the train proc manager, it restarted and then it hangs already in the very first step. With cuda-gdb, the stacktrace looks like:

#0  0x000014eea8c8722a in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x000014eea8c921d8 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x000014eb3d32f3bc in ?? () from /lib64/libcudadebugger.so.1
#3  0x000014eb3d10dad2 in ?? () from /lib64/libcudadebugger.so.1
#4  0x000014eb3d1157e4 in ?? () from /lib64/libcudadebugger.so.1
#5  0x000014eb3d3dc5b6 in ?? () from /lib64/libcudadebugger.so.1
#6  0x000014ed986ad126 in ?? () from /lib64/libcuda.so.1
#7  0x000014ee627597de in cudaGetDevice ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.12
#8  0x000014ee0c3b7cd2 in c10::cuda::GetDevice(signed char*) ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libc10_cuda.so
#9  0x000014ee0c385773 in c10::cuda::impl::CUDAGuardImpl::getDevice() const ()
   from /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/lib/libc10_cuda.so

So, this looks like the GPU is really in some faulty state.