torch.distributed.DistNetworkError: The client socket has timed out after 300000ms while trying to connect to (036c56dce41b, 41331).

[nghiemkythu]

Hi authors,

When I run the example command below:

cosmos-rl --config ./configs/stable-diffusion-3-5/stable-diffusion-3-5-image-sft-lora.toml cosmos_rl.tools.dataset.diffusers_dataset

I encounter the following error:

[cosmos] 2026-05-14 05:18:01,344 - cosmos - INFO - [Controller] Redis server started on port 12802 with command redis-server /tmp/tmpu45ilqsl.redis_config.conf --dbfilename cosmos_rl_14039904-9095-400f-a702-8fe22eb5ecf0.rdb --save ""
[cosmos] 2026-05-14 05:18:01,346 - cosmos - INFO - Successfully loaded configuration from /tmp/tmpu30g4bnr.toml
INFO:     Started server process [1035098]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[cosmos] 2026-05-14 05:18:02,167 - cosmos - INFO - Controller is ready at localhost:8000
[cosmos] 2026-05-14 05:18:02,167 - cosmos - INFO - Command items to be executed for worker 0:
CommandItem(command=/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/cosmos_rl/launcher/launch_replica.sh --type policy --ngpus 4 --backend vllm --config /tmp/tmpu30g4bnr.toml --script cosmos_rl.tools.dataset.diffusers_dataset , gpu_devices=0,1,2,3, control_url=localhost:8000, output_file=None, env={'PYTORCH_CUDA_ALLOC_CONF': 'expandable_segments:True'})
[cosmos] 2026-05-14 05:18:02,168 - cosmos - INFO - Launching process with command: /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/cosmos_rl/launcher/launch_replica.sh --type policy --ngpus 4 --backend vllm --config /tmp/tmpu30g4bnr.toml --script cosmos_rl.tools.dataset.diffusers_dataset 
[cosmos] 2026-05-14 05:18:02,169 - cosmos - INFO - Registering policy process 1035379 for signal handling /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/cosmos_rl/launcher/launch_replica.sh --type policy --ngpus 4 --backend vllm --config /tmp/tmpu30g4bnr.toml --script cosmos_rl.tools.dataset.diffusers_dataset 
[cosmos] 2026-05-14 05:18:02,169 - cosmos - INFO - Signal handler for signal 10 is set to <function DistributedSignalHandler.__init__.<locals>.handler at 0x7fc743117e20>, pid 1034777
[Cosmos-RL] POLICY Pre-setting environment variable NCCL_CUMEM_ENABLE=0
[Cosmos-RL] POLICY Pre-setting environment variable TORCH_CPP_LOG_LEVEL=ERROR
Launching command: torchrun --nproc-per-node=4 --nnodes=1 --role rank --tee 3 --rdzv_backend c10d --rdzv_endpoint=localhost:0 -m cosmos_rl.tools.dataset.diffusers_dataset --config /tmp/tmpu30g4bnr.toml
/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
W0514 05:18:04.365000 1035396 nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/run.py:774] 
W0514 05:18:04.365000 1035396 nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/run.py:774] *****************************************
W0514 05:18:04.365000 1035396 nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0514 05:18:04.365000 1035396 nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/run.py:774] *****************************************
[E514 05:22:29.941862255 socket.cpp:1019] [c10d] The client socket has timed out after 300000ms while trying to connect to (036c56dce41b, 41331).
[E514 05:28:05.991052957 socket.cpp:1019] [c10d] The client socket has timed out after 300000ms while trying to connect to (036c56dce41b, 41331).
[E514 05:28:05.991166639 TCPStore.cpp:328] [c10d] TCP client failed to connect/validate to host 036c56dce41b:41331 - timed out (try=1, timeout=300000ms): The client socket has timed out after 300000ms while trying to connect to (036c56dce41b, 41331).
Exception raised from throwTimeoutError at /pytorch/torch/csrc/distributed/c10d/socket.cpp:1021 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x80 (0x7f8b6a1deeb0 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5d694d1 (0x7f8babbbb4d1 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x1369132 (0x7f8ba71bb132 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5ddddeb (0x7f8babc2fdeb in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0x5dddf8b (0x7f8babc2ff8b in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x5dde324 (0x7f8babc30324 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x5d61dc3 (0x7f8babbb3dc3 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #7: c10d::TCPStore::TCPStore(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10d::TCPStoreOptions const&) + 0x41d (0x7f8babbba7cd in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0xc11045 (0x7f8bbaf1b045 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0xc4b5c6 (0x7f8bbaf555c6 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0x381cdf (0x7f8bba68bcdf in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x2926e2 (0x5620ace276e2 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #12: _PyObject_MakeTpCall + 0x132 (0x5620accef942 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #13: <unknown function> + 0x27340e (0x5620ace0840e in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #14: _PyObject_Call + 0x118 (0x5620acf04708 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #15: <unknown function> + 0x2a6464 (0x5620ace3b464 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #16: <unknown function> + 0x2a7c75 (0x5620ace3cc75 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #17: <unknown function> + 0x380a3b (0x7f8bba68aa3b in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #18: <unknown function> + 0x2fe102 (0x5620ace93102 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #19: _PyEval_EvalFrameDefault + 0x3ccb (0x5620ace5fe0b in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #20: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #21: _PyEval_EvalFrameDefault + 0x8a10 (0x5620ace64b50 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #22: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #23: _PyEval_EvalFrameDefault + 0x8a10 (0x5620ace64b50 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #24: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #25: _PyEval_EvalFrameDefault + 0x87c8 (0x5620ace64908 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #26: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x8a10 (0x5620ace64b50 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #28: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #29: _PyEval_EvalFrameDefault + 0x87c8 (0x5620ace64908 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #30: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #31: _PyEval_EvalFrameDefault + 0x8a10 (0x5620ace64b50 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #32: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #33: _PyEval_EvalFrameDefault + 0x8a10 (0x5620ace64b50 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #34: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #35: _PyEval_EvalFrameDefault + 0x87c8 (0x5620ace64908 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #36: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #37: _PyEval_EvalFrameDefault + 0x8a10 (0x5620ace64b50 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #38: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #39: <unknown function> + 0x2fdf19 (0x5620ace92f19 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #40: _PyEval_EvalFrameDefault + 0x4164 (0x5620ace602a4 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #41: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #42: _PyObject_Call_Prepend + 0x96 (0x5620acf049a6 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #43: <unknown function> + 0x3a3856 (0x5620acf38856 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #44: _PyEval_EvalFrameDefault + 0xa24c (0x5620ace6638c in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #45: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #46: <unknown function> + 0x2fdf19 (0x5620ace92f19 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #47: _PyEval_EvalFrameDefault + 0x4164 (0x5620ace602a4 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #48: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #49: _PyEval_EvalFrameDefault + 0x87c8 (0x5620ace64908 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #50: _PyFunction_Vectorcall + 0x202 (0x5620ace06222 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #51: <unknown function> + 0x2fdf19 (0x5620ace92f19 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #52: _PyEval_EvalFrameDefault + 0x4164 (0x5620ace602a4 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #53: <unknown function> + 0x3d5ab4 (0x5620acf6aab4 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #54: <unknown function> + 0x40b6c3 (0x5620acfa06c3 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #55: <unknown function> + 0x1b3562 (0x5620acd48562 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #56: _PyRun_SimpleFileObject + 0x109 (0x5620acd48099 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #57: _PyRun_AnyFileObject + 0xa5 (0x5620acd47a95 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #58: <unknown function> + 0x1bcd59 (0x5620acd51d59 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #59: <unknown function> + 0x1bc8c2 (0x5620acd518c2 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #60: Py_RunMain + 0x680 (0x5620acfac0b0 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #61: <unknown function> + 0x148028 (0x5620accdd028 in /pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/python3)
frame #62: <unknown function> + 0x29d90 (0x7f8bc8548d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Traceback (most recent call last):
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/bin/torchrun", line 10, in <module>
    sys.exit(main())
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
    run(args)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
    elastic_launch(
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 143, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    result = agent.run()
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
    result = f(*args, **kwargs)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 715, in run
    result = self._invoke_run(role)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 873, in _invoke_run
    self._initialize_workers(self._worker_group)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
    result = f(*args, **kwargs)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 683, in _initialize_workers
    self._rendezvous(worker_group)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
    result = f(*args, **kwargs)
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 498, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1210, in next_rendezvous
    self._shared_tcp_store_server = self._create_tcp_store_server(
  File "/pfss/mlde/workspaces/mlde_wsp_IAS_SAMMerge/VLA/nghiem/cosmos-rl/.venv/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/dynamic_rendezvous.py", line 1126, in _create_tcp_store_server
    return dist.TCPStore(
torch.distributed.DistNetworkError: The client socket has timed out after 300000ms while trying to connect to (036c56dce41b, 41331).
[cosmos] 2026-05-14 05:28:06,261 - cosmos - ERROR - Process 1 failed with return code 1

I would like to ask how to solve this issue.

My environment versions are:

Python: 3.10.19

PyTorch: 2.8.0 + CUDA 12.8 

vLLM: 0.11.0

Redis server: 7.4.7

Thank you very much for your help.