WARN: torch-ipc: CUDA IPC evicting 0x431ee20000 due to ptr reuse, performance will drastically suffer

[chienlinhuang1116]

Hi,

Sometimes, I got messages like these. The jobs are still running and results seem correct. Do you have any idea about this? How can we avoid this situation?

WARN: torch-ipc: CUDA IPC evicting 0x431ee20000 due to ptr reuse, performance will drastically suffer
WARN: torch-ipc: CUDA IPC evicting 0x431ef20000 due to ptr reuse, performance will drastically suffer
WARN: torch-ipc: CUDA IPC evicting 0x431ee20000 due to ptr reuse, performance will drastically suffer
WARN: torch-ipc: CUDA IPC evicting 0x431ef20000 due to ptr reuse, performance will drastically suffer
WARN: torch-ipc: CUDA IPC evicting 0x431ee20000 due to ptr reuse, performance will drastically suffer

Thank you,
Chien-Lin

[xternalz]

I'm experiencing this too, even with stableGradients = true. Anyone has idea why?