[CI] Parity: include PR head SHA in run name for PR-triggered runs#3389
Closed
ethanwee1 wants to merge 1 commit into
Closed
[CI] Parity: include PR head SHA in run name for PR-triggered runs#3389ethanwee1 wants to merge 1 commit into
ethanwee1 wants to merge 1 commit into
ROCm Repo Management API / Jenkins
failed
Jul 2, 2026 in 3h 49m 29s
Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-02T00:32:06.915Z] + ./test_pytorch_test.sh
[2026-07-02T00:32:06.915Z] + [[ -z '' ]]
[2026-07-02T00:32:06.915Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-02T00:32:06.915Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-02T00:32:06.915Z] ++ set -ex
[2026-07-02T00:32:06.915Z] ++ set -o pipefail
[2026-07-02T00:32:06.915Z] ++ copy_whls=0
[2026-07-02T00:32:06.915Z] ++ envvars=()
[2026-07-02T00:32:06.915Z] ++ [[ 13 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=-e
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e $2")
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 11 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=-e
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e $2")
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 9 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=-e
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e $2")
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 7 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=-e
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e $2")
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 5 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=-e
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e $2")
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 3 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=-e
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e $2")
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 1 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ key=--copy-whls
[2026-07-02T00:32:06.915Z] ++ case $key in
[2026-07-02T00:32:06.915Z] ++ copy_whls=1
[2026-07-02T00:32:06.915Z] ++ shift
[2026-07-02T00:32:06.915Z] ++ [[ 0 -gt 0 ]]
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-02T00:32:06.915Z] ++ [[ false == \t\r\u\e ]]
[2026-07-02T00:32:06.915Z] ++ [[ true == \t\r\u\e ]]
[2026-07-02T00:32:06.915Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-02T00:32:06.915Z] ++ docker ps -a
[2026-07-02T00:32:06.915Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-02T00:32:06.915Z] ++ docker rm -f pytorch-ci-container
[2026-07-02T00:32:06.915Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:32:08.242Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-02T00:32:08.664Z] 1057fba0c7df: Pulling fs layer
[2026-07-02T00:32:08.664Z] cb259a83ac3d: Pulling fs layer
[2026-07-02T00:32:08.664Z] 4c8a34755929: Pulling fs layer
[2026-07-02T00:32:08.664Z] 99e6462d03e0: Pulling fs layer
[2026-07-02T00:32:08.664Z] 07b45639fc6f: Pulling fs layer
[2026-07-02T00:32:08.664Z] ba1fabf878bd: Pulling fs layer
[2026-07-02T00:32:08.664Z] 42239a12ed73: Pulling fs layer
[2026-07-02T00:32:08.664Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-02T00:32:08.664Z] eab367f1ca39: Pulling fs layer
[2026-07-02T00:32:08.664Z] 65f281ce2e70: Pulling fs layer
[2026-07-02T00:32:08.664Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-02T00:32:08.664Z] b96eaed0e751: Pulling fs layer
[2026-07-02T00:32:08.664Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-02T00:32:08.664Z] fef65a673dc9: Pulling fs layer
[2026-07-02T00:32:08.664Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:32:08.664Z] 423e26ddb517: Pulling fs layer
[2026-07-02T00:32:08.664Z] 95d704643610: Pulling fs layer
[2026-07-02T00:32:08.664Z] dedc784bccba: Pulling fs layer
[2026-07-02T00:32:08.664Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-02T00:32:08.664Z] 27fcad4d96a6: Pulling fs layer
[2026-07-02T00:32:08.664Z] 8d78d529f04d: Pulling fs layer
[2026-07-02T00:32:08.664Z] 4f4fb700ef54: Pulling fs layer
[2026-07-02T00:32:08.664Z] 86cc05da26d1: Pulling fs layer
[2026-07-02T00:32:08.664Z] 8b3d007242f8: Pulling fs layer
[2026-07-02T00:32:08.664Z] 4106a853273f: Pulling fs layer
[2026-07-02T00:32:08.664Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-02T00:32:08.664Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-02T00:32:08.664Z] f3058198fba5: Pulling fs layer
[2026-07-02T00:32:08.664Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-02T00:32:08.664Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:32:08.664Z] 8d998c79d309: Pulling fs layer
[2026-07-02T00:32:08.664Z] 78cb638d6686: Pulling fs layer
[2026-07-02T00:32:08.664Z] b09b7d04a8d9: Pulling fs layer
[2026-07-02T00:32:08.664Z] f790d498bebb: Pulling fs layer
[2026-07-02T00:32:08.664Z] ed874654c321: Pulling fs layer
[2026-07-02T00:32:08.664Z] 906ac4de0a2b: Pulling fs layer
[2026-07-02T00:32:08.664Z] f5afebc61536: Pulling fs layer
[2026-07-02T00:32:08.664Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:32:08.664Z] c0664a7365cb: Pulling fs layer
[2026-07-02T00:32:08.664Z] 9548298aa5f4: Pulling fs layer
[2026-07-02T00:32:08.664Z] 78a3c5370db4: Pulling fs layer
[2026-07-02T00:32:08.664Z] d9a8b776bea7: Pulling fs layer
[2026-07-02T00:32:08.664Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:32:08.664Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-02T00:32:08.664Z] 241cbb1c2f26: Pulling fs layer
[2026-07-02T00:32:08.664Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:32:08.664Z] 5edf52847c35: Pulling fs layer
[2026-07-02T00:32:08.664Z] f0ef294e4f9b: Pulling fs layer
[2026-07-02T00:32:08.664Z] f391428cfe0f: Pulling fs layer
[2026-07-02T00:32:08.664Z] c608e00b1282: Pulling fs layer
[2026-07-02T00:32:08.664Z] 284e8cbb8a05: Pulling fs layer
[2026-07-02T00:32:08.664Z] cccff27d82fb: Pulling fs layer
[2026-07-02T00:32:08.664Z] 4f4fb700ef54: Pulling fs layer
[2026-07-02T00:32:08.664Z] 34e29023481f: Pulling fs layer
[2026-07-02T00:32:08.664Z] db0af168a1a4: Pulling fs layer
[2026-07-02T00:32:08.664Z] 747272797eac: Pulling fs layer
[2026-07-02T00:32:08.664Z] d7df6379f9cc: Pulling fs layer
[2026-07-02T00:32:08.664Z] 69145e0460f6: Download complete
[2026-07-02T00:32:09.082Z] eab367f1ca39: Download complete
[2026-07-02T00:32:09.082Z] 42239a12ed73: Download complete
[2026-07-02T00:32:09.082Z] 0fcc3b23bd99: Download complete
[2026-07-02T00:32:09.082Z] 8d78d529f04d: Download complete
[2026-07-02T00:32:09.082Z] 99e6462d03e0: Download complete
[2026-07-02T00:32:09.082Z] 07b45639fc6f: Download complete
[2026-07-02T00:32:09.082Z] 5f8dd10c9c78: Download complete
[2026-07-02T00:32:09.082Z] 65f281ce2e70: Download complete
[2026-07-02T00:32:09.082Z] b96eaed0e751: Download complete
[2026-07-02T00:32:09.082Z] ed86e5a5d9b1: Download complete
[2026-07-02T00:32:09.082Z] bf3e4cf00cd6: Download complete
[2026-07-02T00:32:09.082Z] 423e26ddb517: Download complete
[2026-07-02T00:32:09.082Z] fef65a673dc9: Download complete
[2026-07-02T00:32:09.082Z] 3d56742d201d: Download complete
[2026-07-02T00:32:09.082Z] 4f4fb700ef54: Download complete
[2026-07-02T00:32:09.082Z] c608e00b1282: Download complete
[2026-07-02T00:32:09.082Z] 9548298aa5f4: Download complete
[2026-07-02T00:32:09.467Z] 78a3c5370db4: Download complete
[2026-07-02T00:32:09.467Z] 241cbb1c2f26: Download complete
[2026-07-02T00:32:09.467Z] 86cc05da26d1: Download complete
[2026-07-02T00:32:09.467Z] db0af168a1a4: Download complete
[2026-07-02T00:32:09.467Z] c0664a7365cb: Download complete
[2026-07-02T00:32:09.467Z] 0a0ffadd8b4f: Download complete
[2026-07-02T00:32:09.467Z] 4cfd21f31c8b: Download complete
[2026-07-02T00:32:09.467Z] f5afebc61536: Download complete
[2026-07-02T00:32:09.467Z] 27fcad4d96a6: Download complete
[2026-07-02T00:32:09.467Z] 284e8cbb8a05: Download complete
[2026-07-02T00:32:09.467Z] d7df6379f9cc: Download complete
[2026-07-02T00:32:09.467Z] f790d498bebb: Download complete
[2026-07-02T00:32:09.467Z] f391428cfe0f: Download complete
[2026-07-02T00:32:09.467Z] 8d998c79d309: Download complete
[2026-07-02T00:32:09.467Z] 9ae773ccc4b2: Download complete
[2026-07-02T00:32:09.467Z] f3058198fba5: Download complete
[2026-07-02T00:32:09.467Z] b09b7d04a8d9: Download complete
[2026-07-02T00:32:09.467Z] 8b3d007242f8: Download complete
[2026-07-02T00:32:09.467Z] 747272797eac: Download complete
[2026-07-02T00:32:09.467Z] ed874654c321: Download complete
[2026-07-02T00:32:09.467Z] 906ac4de0a2b: Download complete
[2026-07-02T00:32:09.467Z] f0ef294e4f9b: Download complete
[2026-07-02T00:32:09.467Z] 34e29023481f: Download complete
[2026-07-02T00:32:09.467Z] 4106a853273f: Download complete
[2026-07-02T00:32:09.467Z] 0e6d9b5f5e8d: Download complete
[2026-07-02T00:32:09.467Z] ba1fabf878bd: Download complete
[2026-07-02T00:32:09.849Z] cb259a83ac3d: Download complete
[2026-07-02T00:32:09.849Z] 4c8a34755929: Download complete
[2026-07-02T00:32:10.675Z] 99e6462d03e0: Pull complete
[2026-07-02T00:32:10.675Z] cb259a83ac3d: Pull complete
[2026-07-02T00:32:10.675Z] 95d704643610: Download complete
[2026-07-02T00:32:12.624Z] cccff27d82fb: Download complete
[2026-07-02T00:32:16.156Z] 1057fba0c7df: Download complete
[2026-07-02T00:32:22.744Z] 34e29023481f: Pull complete
[2026-07-02T00:32:22.744Z] 0e6d9b5f5e8d: Pull complete
[2026-07-02T00:32:22.744Z] 1057fba0c7df: Pull complete
[2026-07-02T00:32:22.744Z] 8d78d529f04d: Pull complete
[2026-07-02T00:32:22.744Z] 65f281ce2e70: Pull complete
[2026-07-02T00:32:22.744Z] c608e00b1282: Pull complete
[2026-07-02T00:32:22.744Z] d7df6379f9cc: Pull complete
[2026-07-02T00:32:22.744Z] 906ac4de0a2b: Pull complete
[2026-07-02T00:32:26.901Z] 78cb638d6686: Download complete
[2026-07-02T00:32:55.671Z] 5edf52847c35: Download complete
[2026-07-02T00:33:36.714Z] 9548298aa5f4: Pull complete
[2026-07-02T00:33:36.714Z] 86cc05da26d1: Pull complete
[2026-07-02T00:33:36.714Z] db0af168a1a4: Pull complete
[2026-07-02T00:33:36.714Z] 0a0ffadd8b4f: Pull complete
[2026-07-02T00:33:36.714Z] 8b3d007242f8: Pull complete
[2026-07-02T00:33:36.714Z] 747272797eac: Pull complete
[2026-07-02T00:33:36.714Z] ed874654c321: Pull complete
[2026-07-02T00:33:36.714Z] 5edf52847c35: Pull complete
[2026-07-02T00:34:11.174Z] d9a8b776bea7: Download complete
[2026-07-02T00:34:52.598Z] dedc784bccba: Download complete
[2026-07-02T00:36:32.432Z] 27fcad4d96a6: Pull complete
[2026-07-02T00:36:32.432Z] 9ae773ccc4b2: Pull complete
[2026-07-02T00:36:32.432Z] dedc784bccba: Pull complete
[2026-07-02T00:36:32.432Z] bf3e4cf00cd6: Pull complete
[2026-07-02T00:36:37.706Z] f3058198fba5: Pull complete
[2026-07-02T00:36:37.706Z] 78cb638d6686: Pull complete
[2026-07-02T00:36:37.706Z] f5afebc61536: Pull complete
[2026-07-02T00:36:39.032Z] 07b45639fc6f: Pull complete
[2026-07-02T00:36:39.032Z] b96eaed0e751: Pull complete
[2026-07-02T00:36:39.032Z] c0664a7365cb: Pull complete
[2026-07-02T00:36:39.032Z] f790d498bebb: Pull complete
[2026-07-02T00:36:39.032Z] cccff27d82fb: Pull complete
[2026-07-02T00:36:39.443Z] 423e26ddb517: Pull complete
[2026-07-02T00:36:39.443Z] 95d704643610: Pull complete
[2026-07-02T00:36:39.443Z] 42239a12ed73: Pull complete
[2026-07-02T00:36:39.443Z] 0fcc3b23bd99: Pull complete
[2026-07-02T00:36:39.443Z] 5f8dd10c9c78: Pull complete
[2026-07-02T00:36:39.888Z] 3d56742d201d: Pull complete
[2026-07-02T00:36:39.888Z] 4f4fb700ef54: Pull complete
[2026-07-02T00:36:39.888Z] 78a3c5370db4: Pull complete
[2026-07-02T00:36:39.888Z] 241cbb1c2f26: Pull complete
[2026-07-02T00:36:39.888Z] 4cfd21f31c8b: Pull complete
[2026-07-02T00:36:39.888Z] 284e8cbb8a05: Pull complete
[2026-07-02T00:36:39.888Z] b09b7d04a8d9: Pull complete
[2026-07-02T00:36:39.888Z] f0ef294e4f9b: Pull complete
[2026-07-02T00:36:39.888Z] 4106a853273f: Pull complete
[2026-07-02T00:37:50.576Z] eab367f1ca39: Pull complete
[2026-07-02T00:37:50.576Z] f391428cfe0f: Pull complete
[2026-07-02T00:37:50.576Z] d9a8b776bea7: Pull complete
[2026-07-02T00:37:50.576Z] ed86e5a5d9b1: Pull complete
[2026-07-02T00:37:50.576Z] fef65a673dc9: Pull complete
[2026-07-02T00:37:50.576Z] ba1fabf878bd: Pull complete
[2026-07-02T00:37:50.576Z] 8d998c79d309: Pull complete
[2026-07-02T00:37:50.576Z] 4c8a34755929: Pull complete
[2026-07-02T00:37:50.576Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-02T00:37:50.576Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:37:50.576Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:37:50.576Z] ++ GPU_FLAGS=
[2026-07-02T00:37:50.576Z] ++ echo 'NODE_LABELS: gfx90a k8s-mi250-1-wjbs9 mlir gfx90a k8s-mi250-1-wjbs9 mlir'
[2026-07-02T00:37:50.576Z] NODE_LABELS: gfx90a k8s-mi250-1-wjbs9 mlir gfx90a k8s-mi250-1-wjbs9 mlir
[2026-07-02T00:37:50.576Z] ++ [[ ! gfx90a k8s-mi250-1-wjbs9 mlir =~ build-only ]]
[2026-07-02T00:37:50.576Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-02T00:37:50.576Z] ++++ dirname prepare_docker_env.sh
[2026-07-02T00:37:50.576Z] +++ cd .
[2026-07-02T00:37:50.576Z] +++ pwd
[2026-07-02T00:37:50.576Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389
[2026-07-02T00:37:50.576Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-02T00:37:50.576Z] d618b40aec05cd455cdb17fbebc7c11cd146a8d4df3a396e71d47d3a2c308ea3
[2026-07-02T00:37:50.576Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T00:37:50.576Z] Exporting test times from test-infra
[2026-07-02T00:37:50.576Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:37:50.576Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T00:37:50.576Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T00:37:50.576Z] Exporting test times from test-infra
[2026-07-02T00:37:50.576Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:37:50.576Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T00:37:50.576Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files ]]
[2026-07-02T00:37:50.576Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:37:50.576Z] ++ [[ false != \t\r\u\e ]]
[2026-07-02T00:37:50.576Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-02T00:37:50.576Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T00:37:54.669Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-02T00:37:54.669Z] ++ [[ 1 -eq 1 ]]
[2026-07-02T00:37:54.669Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3389/torch-2.14.0a0+git8ddc477-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T00:37:54.669Z] docker: 'docker cp' requires 2 arguments
[2026-07-02T00:37:54.669Z]
[2026-07-02T00:37:54.669Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-02T00:37:54.669Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-02T00:37:54.669Z]
[2026-07-02T00:37:54.669Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal
Error in error step, with arguments pytorch_test_1 failed.
pytorch_test_1 failed
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test.sh.
script returned exit code 1
Build log
[2026-07-02T01:45:08.167Z] + ./test_pytorch_test.sh
[2026-07-02T01:45:08.167Z] + [[ -z '' ]]
[2026-07-02T01:45:08.167Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-02T01:45:08.167Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-02T01:45:08.167Z] ++ set -ex
[2026-07-02T01:45:08.167Z] ++ set -o pipefail
[2026-07-02T01:45:08.167Z] ++ copy_whls=0
[2026-07-02T01:45:08.167Z] ++ envvars=()
[2026-07-02T01:45:08.167Z] ++ [[ 13 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=-e
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e $2")
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 11 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=-e
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e $2")
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 9 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=-e
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e $2")
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 7 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=-e
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e $2")
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 5 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=-e
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e $2")
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 3 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=-e
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e $2")
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 1 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ key=--copy-whls
[2026-07-02T01:45:08.167Z] ++ case $key in
[2026-07-02T01:45:08.167Z] ++ copy_whls=1
[2026-07-02T01:45:08.167Z] ++ shift
[2026-07-02T01:45:08.167Z] ++ [[ 0 -gt 0 ]]
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-02T01:45:08.167Z] ++ [[ false == \t\r\u\e ]]
[2026-07-02T01:45:08.167Z] ++ [[ true == \t\r\u\e ]]
[2026-07-02T01:45:08.167Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-02T01:45:08.167Z] ++ docker ps -a
[2026-07-02T01:45:08.167Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-02T01:45:08.167Z] 88102abc6606 rocm/pytorch-private:ubuntu20.04_py3.9_pip3.9 "bash -lc 'sleep inf…" 2 hours ago Up 2 hours pytorch-ci-container
[2026-07-02T01:45:08.167Z] 33af8c9e058f dashboard-dashboard-ui "/docker-entrypoint.…" 9 days ago Up 19 hours 80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp rocagents-dashboard-ui
[2026-07-02T01:45:08.167Z] 7c6be83a2db1 dashboard-dashboard-api "uvicorn main:app --…" 9 days ago Up 19 hours (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp rocagents-dashboard-api
[2026-07-02T01:45:08.167Z] 780dc8fe5a7b rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen "/scripts/run_tests_…" 4 weeks ago Exited (1) 4 weeks ago infallible_chatelet
[2026-07-02T01:45:08.167Z] 2d7786cfd066 deployment-rocagents "python src/multi_re…" 7 weeks ago Up 19 hours (healthy) rocagents-orchestrator
[2026-07-02T01:45:08.167Z] ++ docker rm -f pytorch-ci-container
[2026-07-02T01:45:09.989Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T01:45:10.565Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-02T01:45:10.565Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-02T01:45:10.565Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T01:45:10.565Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T01:45:10.565Z] ++ GPU_FLAGS=
[2026-07-02T01:45:10.565Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-07-02T01:45:10.565Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-07-02T01:45:10.565Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-07-02T01:45:10.565Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-02T01:45:10.565Z] ++++ dirname prepare_docker_env.sh
[2026-07-02T01:45:10.565Z] +++ cd .
[2026-07-02T01:45:10.565Z] +++ pwd
[2026-07-02T01:45:10.565Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389
[2026-07-02T01:45:10.565Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-02T01:45:10.565Z] 0e2e404fcfbdc408866680a904f331277b60c3a2748c72eb7d602e365813abd1
[2026-07-02T01:45:10.565Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T01:45:11.901Z] Exporting test times from test-infra
[2026-07-02T01:45:11.901Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T01:45:11.901Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T01:45:11.901Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T01:45:11.901Z] Exporting test times from test-infra
[2026-07-02T01:45:11.901Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T01:45:11.901Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T01:45:11.901Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files ]]
[2026-07-02T01:45:11.901Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T01:45:11.901Z] ++ [[ false != \t\r\u\e ]]
[2026-07-02T01:45:11.901Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-02T01:45:11.901Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T01:45:16.399Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-02T01:45:16.937Z] ++ [[ 1 -eq 1 ]]
[2026-07-02T01:45:16.937Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/torch-2.14.0a0+git8ddc477-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T01:45:16.937Z] docker: 'docker cp' requires 2 arguments
[2026-07-02T01:45:16.937Z]
[2026-07-02T01:45:16.937Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-02T01:45:16.937Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-02T01:45:16.937Z]
[2026-07-02T01:45:16.937Z] Run 'docker cp --help' for more information
Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal
Error in error step, with arguments pytorch_test_2 failed.
pytorch_test_2 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-02T00:31:19.589Z] + ./test_pytorch_test_distributed.sh
[2026-07-02T00:31:19.589Z] + [[ -z '' ]]
[2026-07-02T00:31:19.589Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-02T00:31:19.589Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-02T00:31:19.589Z] ++ set -ex
[2026-07-02T00:31:19.589Z] ++ set -o pipefail
[2026-07-02T00:31:19.589Z] ++ copy_whls=0
[2026-07-02T00:31:19.589Z] ++ envvars=()
[2026-07-02T00:31:19.589Z] ++ [[ 13 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=-e
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e $2")
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 11 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=-e
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e $2")
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 9 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=-e
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e $2")
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 7 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=-e
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e $2")
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 5 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=-e
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e $2")
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 3 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=-e
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e $2")
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 1 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ key=--copy-whls
[2026-07-02T00:31:19.589Z] ++ case $key in
[2026-07-02T00:31:19.589Z] ++ copy_whls=1
[2026-07-02T00:31:19.589Z] ++ shift
[2026-07-02T00:31:19.589Z] ++ [[ 0 -gt 0 ]]
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-02T00:31:19.589Z] ++ [[ false == \t\r\u\e ]]
[2026-07-02T00:31:19.589Z] ++ [[ true == \t\r\u\e ]]
[2026-07-02T00:31:19.589Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-02T00:31:19.589Z] ++ docker ps -a
[2026-07-02T00:31:19.589Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-02T00:31:19.589Z] 211c4975ae20 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-1ae06b266a82a9290b48a6f384f86d9bfcc747b2-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 7 hours ago Up 7 hours pytorch-ci-container
[2026-07-02T00:31:19.589Z] ++ docker rm -f pytorch-ci-container
[2026-07-02T00:31:24.934Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:31:26.069Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-02T00:31:26.595Z] cb259a83ac3d: Already exists
[2026-07-02T00:31:26.595Z] 99e6462d03e0: Pulling fs layer
[2026-07-02T00:31:26.595Z] 1057fba0c7df: Pulling fs layer
[2026-07-02T00:31:26.595Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-02T00:31:26.595Z] 65f281ce2e70: Pulling fs layer
[2026-07-02T00:31:26.595Z] 34e29023481f: Pulling fs layer
[2026-07-02T00:31:26.595Z] 8d78d529f04d: Pulling fs layer
[2026-07-02T00:31:26.595Z] 906ac4de0a2b: Pulling fs layer
[2026-07-02T00:31:26.595Z] c608e00b1282: Pulling fs layer
[2026-07-02T00:31:26.595Z] d7df6379f9cc: Pulling fs layer
[2026-07-02T00:31:26.595Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:31:26.595Z] 5edf52847c35: Pulling fs layer
[2026-07-02T00:31:26.595Z] 65f281ce2e70: Waiting
[2026-07-02T00:31:26.595Z] 9548298aa5f4: Pulling fs layer
[2026-07-02T00:31:26.595Z] d7df6379f9cc: Waiting
[2026-07-02T00:31:26.595Z] 34e29023481f: Waiting
[2026-07-02T00:31:26.595Z] 86cc05da26d1: Pulling fs layer
[2026-07-02T00:31:26.595Z] 3d56742d201d: Waiting
[2026-07-02T00:31:26.595Z] 8d78d529f04d: Waiting
[2026-07-02T00:31:26.595Z] 747272797eac: Pulling fs layer
[2026-07-02T00:31:26.595Z] 5edf52847c35: Waiting
[2026-07-02T00:31:26.595Z] 9548298aa5f4: Waiting
[2026-07-02T00:31:26.595Z] 86cc05da26d1: Waiting
[2026-07-02T00:31:26.595Z] db0af168a1a4: Pulling fs layer
[2026-07-02T00:31:26.595Z] 906ac4de0a2b: Waiting
[2026-07-02T00:31:26.595Z] ed874654c321: Pulling fs layer
[2026-07-02T00:31:26.595Z] c608e00b1282: Waiting
[2026-07-02T00:31:26.595Z] 747272797eac: Waiting
[2026-07-02T00:31:26.595Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-02T00:31:26.595Z] db0af168a1a4: Waiting
[2026-07-02T00:31:26.595Z] 8b3d007242f8: Pulling fs layer
[2026-07-02T00:31:26.595Z] dedc784bccba: Pulling fs layer
[2026-07-02T00:31:26.595Z] 0a0ffadd8b4f: Waiting
[2026-07-02T00:31:26.595Z] 8b3d007242f8: Waiting
[2026-07-02T00:31:26.595Z] 27fcad4d96a6: Pulling fs layer
[2026-07-02T00:31:26.595Z] dedc784bccba: Waiting
[2026-07-02T00:31:26.595Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-02T00:31:26.595Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-02T00:31:26.595Z] 27fcad4d96a6: Waiting
[2026-07-02T00:31:26.595Z] 78cb638d6686: Pulling fs layer
[2026-07-02T00:31:26.595Z] bf3e4cf00cd6: Waiting
[2026-07-02T00:31:26.595Z] f3058198fba5: Pulling fs layer
[2026-07-02T00:31:26.595Z] 9ae773ccc4b2: Waiting
[2026-07-02T00:31:26.595Z] f5afebc61536: Pulling fs layer
[2026-07-02T00:31:26.595Z] f3058198fba5: Waiting
[2026-07-02T00:31:26.595Z] 78cb638d6686: Waiting
[2026-07-02T00:31:26.595Z] cccff27d82fb: Pulling fs layer
[2026-07-02T00:31:26.595Z] f5afebc61536: Waiting
[2026-07-02T00:31:26.595Z] c0664a7365cb: Pulling fs layer
[2026-07-02T00:31:26.595Z] cccff27d82fb: Waiting
[2026-07-02T00:31:26.595Z] 07b45639fc6f: Pulling fs layer
[2026-07-02T00:31:26.595Z] c0664a7365cb: Waiting
[2026-07-02T00:31:26.595Z] b96eaed0e751: Pulling fs layer
[2026-07-02T00:31:26.595Z] 07b45639fc6f: Waiting
[2026-07-02T00:31:26.595Z] f790d498bebb: Pulling fs layer
[2026-07-02T00:31:26.595Z] b96eaed0e751: Waiting
[2026-07-02T00:31:26.595Z] 95d704643610: Pulling fs layer
[2026-07-02T00:31:26.595Z] f790d498bebb: Waiting
[2026-07-02T00:31:26.595Z] 423e26ddb517: Pulling fs layer
[2026-07-02T00:31:26.595Z] 95d704643610: Waiting
[2026-07-02T00:31:26.595Z] 4106a853273f: Pulling fs layer
[2026-07-02T00:31:26.595Z] 423e26ddb517: Waiting
[2026-07-02T00:31:26.595Z] 4106a853273f: Waiting
[2026-07-02T00:31:26.595Z] 5f8dd10c9c78: Pulling fs layer
[2026-07-02T00:31:26.595Z] b09b7d04a8d9: Pulling fs layer
[2026-07-02T00:31:26.595Z] 5f8dd10c9c78: Waiting
[2026-07-02T00:31:26.595Z] b09b7d04a8d9: Waiting
[2026-07-02T00:31:26.595Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-02T00:31:26.595Z] 241cbb1c2f26: Pulling fs layer
[2026-07-02T00:31:26.595Z] f0ef294e4f9b: Pulling fs layer
[2026-07-02T00:31:26.595Z] 4cfd21f31c8b: Waiting
[2026-07-02T00:31:26.595Z] 241cbb1c2f26: Waiting
[2026-07-02T00:31:26.595Z] 4f4fb700ef54: Pulling fs layer
[2026-07-02T00:31:26.595Z] f0ef294e4f9b: Waiting
[2026-07-02T00:31:26.595Z] 4f4fb700ef54: Waiting
[2026-07-02T00:31:26.595Z] 78a3c5370db4: Pulling fs layer
[2026-07-02T00:31:26.595Z] 42239a12ed73: Pulling fs layer
[2026-07-02T00:31:26.595Z] 78a3c5370db4: Waiting
[2026-07-02T00:31:26.595Z] 42239a12ed73: Waiting
[2026-07-02T00:31:26.595Z] 284e8cbb8a05: Pulling fs layer
[2026-07-02T00:31:26.595Z] 0fcc3b23bd99: Pulling fs layer
[2026-07-02T00:31:26.595Z] 284e8cbb8a05: Waiting
[2026-07-02T00:31:26.595Z] d9a8b776bea7: Pulling fs layer
[2026-07-02T00:31:26.595Z] 0fcc3b23bd99: Waiting
[2026-07-02T00:31:26.595Z] f391428cfe0f: Pulling fs layer
[2026-07-02T00:31:26.595Z] eab367f1ca39: Pulling fs layer
[2026-07-02T00:31:26.595Z] d9a8b776bea7: Waiting
[2026-07-02T00:31:26.595Z] f391428cfe0f: Waiting
[2026-07-02T00:31:26.595Z] ba1fabf878bd: Pulling fs layer
[2026-07-02T00:31:26.595Z] eab367f1ca39: Waiting
[2026-07-02T00:31:26.595Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-02T00:31:26.595Z] ba1fabf878bd: Waiting
[2026-07-02T00:31:26.595Z] ed86e5a5d9b1: Waiting
[2026-07-02T00:31:26.595Z] fef65a673dc9: Pulling fs layer
[2026-07-02T00:31:26.595Z] fef65a673dc9: Waiting
[2026-07-02T00:31:26.595Z] 8d998c79d309: Pulling fs layer
[2026-07-02T00:31:26.595Z] 4c8a34755929: Pulling fs layer
[2026-07-02T00:31:26.595Z] 8d998c79d309: Waiting
[2026-07-02T00:31:26.595Z] 4c8a34755929: Waiting
[2026-07-02T00:31:27.125Z] 99e6462d03e0: Download complete
[2026-07-02T00:31:27.125Z] 0e6d9b5f5e8d: Verifying Checksum
[2026-07-02T00:31:27.125Z] 0e6d9b5f5e8d: Download complete
[2026-07-02T00:31:27.125Z] 99e6462d03e0: Pull complete
[2026-07-02T00:31:27.125Z] 65f281ce2e70: Download complete
[2026-07-02T00:31:27.125Z] 8d78d529f04d: Download complete
[2026-07-02T00:31:27.655Z] 34e29023481f: Download complete
[2026-07-02T00:31:27.655Z] 906ac4de0a2b: Verifying Checksum
[2026-07-02T00:31:27.655Z] 906ac4de0a2b: Download complete
[2026-07-02T00:31:27.655Z] c608e00b1282: Download complete
[2026-07-02T00:31:27.655Z] d7df6379f9cc: Verifying Checksum
[2026-07-02T00:31:27.655Z] d7df6379f9cc: Download complete
[2026-07-02T00:31:28.202Z] 3d56742d201d: Download complete
[2026-07-02T00:31:28.202Z] 9548298aa5f4: Download complete
[2026-07-02T00:31:28.202Z] 86cc05da26d1: Download complete
[2026-07-02T00:31:29.257Z] 747272797eac: Verifying Checksum
[2026-07-02T00:31:29.257Z] 747272797eac: Download complete
[2026-07-02T00:31:29.257Z] db0af168a1a4: Verifying Checksum
[2026-07-02T00:31:29.257Z] db0af168a1a4: Download complete
[2026-07-02T00:31:29.257Z] ed874654c321: Verifying Checksum
[2026-07-02T00:31:29.257Z] ed874654c321: Download complete
[2026-07-02T00:31:29.257Z] 0a0ffadd8b4f: Verifying Checksum
[2026-07-02T00:31:29.257Z] 0a0ffadd8b4f: Download complete
[2026-07-02T00:31:30.040Z] 8b3d007242f8: Download complete
[2026-07-02T00:31:33.847Z] 1057fba0c7df: Verifying Checksum
[2026-07-02T00:31:33.847Z] 1057fba0c7df: Download complete
[2026-07-02T00:31:34.478Z] 27fcad4d96a6: Download complete
[2026-07-02T00:31:34.478Z] bf3e4cf00cd6: Verifying Checksum
[2026-07-02T00:31:34.478Z] bf3e4cf00cd6: Download complete
[2026-07-02T00:31:34.478Z] 9ae773ccc4b2: Download complete
[2026-07-02T00:31:59.585Z] 1057fba0c7df: Pull complete
[2026-07-02T00:31:59.585Z] 0e6d9b5f5e8d: Pull complete
[2026-07-02T00:31:59.585Z] 65f281ce2e70: Pull complete
[2026-07-02T00:31:59.585Z] 34e29023481f: Pull complete
[2026-07-02T00:31:59.585Z] 8d78d529f04d: Pull complete
[2026-07-02T00:31:59.585Z] 906ac4de0a2b: Pull complete
[2026-07-02T00:31:59.585Z] c608e00b1282: Pull complete
[2026-07-02T00:31:59.585Z] d7df6379f9cc: Pull complete
[2026-07-02T00:31:59.585Z] 3d56742d201d: Pull complete
[2026-07-02T00:32:01.201Z] 78cb638d6686: Verifying Checksum
[2026-07-02T00:32:01.201Z] 78cb638d6686: Download complete
[2026-07-02T00:32:01.201Z] f3058198fba5: Verifying Checksum
[2026-07-02T00:32:01.201Z] f3058198fba5: Download complete
[2026-07-02T00:32:01.814Z] f5afebc61536: Verifying Checksum
[2026-07-02T00:32:01.814Z] f5afebc61536: Download complete
[2026-07-02T00:32:07.213Z] cccff27d82fb: Verifying Checksum
[2026-07-02T00:32:07.213Z] cccff27d82fb: Download complete
[2026-07-02T00:32:08.340Z] c0664a7365cb: Download complete
[2026-07-02T00:32:08.924Z] 07b45639fc6f: Verifying Checksum
[2026-07-02T00:32:08.924Z] 07b45639fc6f: Download complete
[2026-07-02T00:32:08.924Z] b96eaed0e751: Download complete
[2026-07-02T00:32:09.493Z] f790d498bebb: Download complete
[2026-07-02T00:32:11.327Z] 95d704643610: Verifying Checksum
[2026-07-02T00:32:11.327Z] 95d704643610: Download complete
[2026-07-02T00:32:11.327Z] 423e26ddb517: Verifying Checksum
[2026-07-02T00:32:11.327Z] 423e26ddb517: Download complete
[2026-07-02T00:32:11.891Z] 4106a853273f: Verifying Checksum
[2026-07-02T00:32:11.891Z] 4106a853273f: Download complete
[2026-07-02T00:32:12.484Z] 5f8dd10c9c78: Download complete
[2026-07-02T00:32:12.484Z] b09b7d04a8d9: Verifying Checksum
[2026-07-02T00:32:12.484Z] b09b7d04a8d9: Download complete
[2026-07-02T00:32:13.094Z] 4cfd21f31c8b: Verifying Checksum
[2026-07-02T00:32:13.094Z] 4cfd21f31c8b: Download complete
[2026-07-02T00:32:14.521Z] 241cbb1c2f26: Download complete
[2026-07-02T00:32:14.521Z] f0ef294e4f9b: Verifying Checksum
[2026-07-02T00:32:14.521Z] f0ef294e4f9b: Download complete
[2026-07-02T00:32:15.845Z] 4f4fb700ef54: Verifying Checksum
[2026-07-02T00:32:15.845Z] 4f4fb700ef54: Download complete
[2026-07-02T00:32:16.479Z] 78a3c5370db4: Verifying Checksum
[2026-07-02T00:32:16.479Z] 78a3c5370db4: Download complete
[2026-07-02T00:32:16.479Z] 42239a12ed73: Download complete
[2026-07-02T00:32:17.026Z] 284e8cbb8a05: Verifying Checksum
[2026-07-02T00:32:17.026Z] 284e8cbb8a05: Download complete
[2026-07-02T00:32:17.026Z] 0fcc3b23bd99: Verifying Checksum
[2026-07-02T00:32:17.026Z] 0fcc3b23bd99: Download complete
[2026-07-02T00:32:24.978Z] 5edf52847c35: Verifying Checksum
[2026-07-02T00:32:24.978Z] 5edf52847c35: Download complete
[2026-07-02T00:32:24.978Z] f391428cfe0f: Download complete
[2026-07-02T00:32:25.535Z] eab367f1ca39: Verifying Checksum
[2026-07-02T00:32:25.535Z] eab367f1ca39: Download complete
[2026-07-02T00:32:27.517Z] ba1fabf878bd: Verifying Checksum
[2026-07-02T00:32:27.517Z] ba1fabf878bd: Download complete
[2026-07-02T00:32:27.517Z] ed86e5a5d9b1: Verifying Checksum
[2026-07-02T00:32:27.517Z] ed86e5a5d9b1: Download complete
[2026-07-02T00:32:27.517Z] fef65a673dc9: Verifying Checksum
[2026-07-02T00:32:27.517Z] fef65a673dc9: Download complete
[2026-07-02T00:32:28.190Z] 8d998c79d309: Verifying Checksum
[2026-07-02T00:32:28.190Z] 8d998c79d309: Download complete
[2026-07-02T00:32:29.353Z] 4c8a34755929: Verifying Checksum
[2026-07-02T00:32:29.353Z] 4c8a34755929: Download complete
[2026-07-02T00:34:14.970Z] d9a8b776bea7: Verifying Checksum
[2026-07-02T00:34:14.970Z] d9a8b776bea7: Download complete
[2026-07-02T00:34:19.424Z] dedc784bccba: Verifying Checksum
[2026-07-02T00:34:19.424Z] dedc784bccba: Download complete
[2026-07-02T00:34:21.248Z] 5edf52847c35: Pull complete
[2026-07-02T00:34:21.248Z] 9548298aa5f4: Pull complete
[2026-07-02T00:34:21.248Z] 86cc05da26d1: Pull complete
[2026-07-02T00:34:21.855Z] 747272797eac: Pull complete
[2026-07-02T00:34:21.855Z] db0af168a1a4: Pull complete
[2026-07-02T00:34:22.385Z] ed874654c321: Pull complete
[2026-07-02T00:34:22.916Z] 0a0ffadd8b4f: Pull complete
[2026-07-02T00:34:23.445Z] 8b3d007242f8: Pull complete
[2026-07-02T00:40:41.782Z] dedc784bccba: Pull complete
[2026-07-02T00:40:41.782Z] 27fcad4d96a6: Pull complete
[2026-07-02T00:40:42.334Z] bf3e4cf00cd6: Pull complete
[2026-07-02T00:40:42.334Z] 9ae773ccc4b2: Pull complete
[2026-07-02T00:41:27.586Z] 78cb638d6686: Pull complete
[2026-07-02T00:41:27.586Z] f3058198fba5: Pull complete
[2026-07-02T00:41:27.586Z] f5afebc61536: Pull complete
[2026-07-02T00:41:32.324Z] cccff27d82fb: Pull complete
[2026-07-02T00:41:32.324Z] c0664a7365cb: Pull complete
[2026-07-02T00:41:32.324Z] 07b45639fc6f: Pull complete
[2026-07-02T00:41:32.324Z] b96eaed0e751: Pull complete
[2026-07-02T00:41:32.324Z] f790d498bebb: Pull complete
[2026-07-02T00:41:34.369Z] 95d704643610: Pull complete
[2026-07-02T00:41:34.369Z] 423e26ddb517: Pull complete
[2026-07-02T00:41:34.369Z] 4106a853273f: Pull complete
[2026-07-02T00:41:35.067Z] 5f8dd10c9c78: Pull complete
[2026-07-02T00:41:35.791Z] b09b7d04a8d9: Pull complete
[2026-07-02T00:41:36.380Z] 4cfd21f31c8b: Pull complete
[2026-07-02T00:41:36.945Z] 241cbb1c2f26: Pull complete
[2026-07-02T00:41:36.945Z] f0ef294e4f9b: Pull complete
[2026-07-02T00:41:37.503Z] 4f4fb700ef54: Pull complete
[2026-07-02T00:41:37.503Z] 78a3c5370db4: Pull complete
[2026-07-02T00:41:37.503Z] 42239a12ed73: Pull complete
[2026-07-02T00:41:37.503Z] 284e8cbb8a05: Pull complete
[2026-07-02T00:41:37.503Z] 0fcc3b23bd99: Pull complete
[2026-07-02T00:45:54.744Z] d9a8b776bea7: Pull complete
[2026-07-02T00:45:54.744Z] f391428cfe0f: Pull complete
[2026-07-02T00:45:54.744Z] eab367f1ca39: Pull complete
[2026-07-02T00:45:54.744Z] ba1fabf878bd: Pull complete
[2026-07-02T00:45:54.744Z] ed86e5a5d9b1: Pull complete
[2026-07-02T00:45:54.744Z] fef65a673dc9: Pull complete
[2026-07-02T00:45:54.744Z] 8d998c79d309: Pull complete
[2026-07-02T00:45:54.744Z] 4c8a34755929: Pull complete
[2026-07-02T00:45:54.744Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-02T00:45:54.744Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:45:54.744Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:45:54.744Z] ++ GPU_FLAGS=
[2026-07-02T00:45:54.744Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-02T00:45:54.744Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-02T00:45:54.744Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-02T00:45:54.744Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-02T00:45:54.744Z] ++++ dirname prepare_docker_env.sh
[2026-07-02T00:45:54.744Z] +++ cd .
[2026-07-02T00:45:54.744Z] +++ pwd
[2026-07-02T00:45:54.744Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389
[2026-07-02T00:45:54.744Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-02T00:45:55.269Z] f86703430d15f6894bd893acb8756641b8debae313403d2379b7ca3692eb0640
[2026-07-02T00:45:55.269Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T00:45:57.100Z] Exporting test times from test-infra
[2026-07-02T00:45:57.100Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:45:57.100Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T00:45:57.100Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T00:45:57.100Z] Exporting test times from test-infra
[2026-07-02T00:45:57.100Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:45:57.100Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T00:45:57.100Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files ]]
[2026-07-02T00:45:57.100Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:45:57.100Z] ++ [[ false != \t\r\u\e ]]
[2026-07-02T00:45:57.100Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-02T00:45:57.643Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T00:46:04.251Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-02T00:46:04.251Z] ++ [[ 1 -eq 1 ]]
[2026-07-02T00:46:04.251Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/torch-2.14.0a0+git8ddc477-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T00:46:04.251Z] "docker cp" requires exactly 2 arguments.
[2026-07-02T00:46:04.251Z] See 'docker cp --help'.
[2026-07-02T00:46:04.251Z]
[2026-07-02T00:46:04.251Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-02T00:46:04.251Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-02T00:46:04.251Z]
[2026-07-02T00:46:04.251Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal
Error in error step, with arguments pytorch_distributed_1 failed.
pytorch_distributed_1 failed
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script
Error in sh step, with arguments ./test_pytorch_test_distributed.sh.
script returned exit code 1
Build log
[2026-07-02T00:49:42.827Z] + ./test_pytorch_test_distributed.sh
[2026-07-02T00:49:42.827Z] + [[ -z '' ]]
[2026-07-02T00:49:42.827Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-02T00:49:42.827Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-02T00:49:42.827Z] ++ set -ex
[2026-07-02T00:49:42.827Z] ++ set -o pipefail
[2026-07-02T00:49:42.827Z] ++ copy_whls=0
[2026-07-02T00:49:42.827Z] ++ envvars=()
[2026-07-02T00:49:42.827Z] ++ [[ 13 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=-e
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e $2")
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 11 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=-e
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e $2")
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 9 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=-e
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e $2")
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 7 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=-e
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e $2")
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 5 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=-e
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e $2")
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 3 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=-e
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e $2")
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 1 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ key=--copy-whls
[2026-07-02T00:49:42.827Z] ++ case $key in
[2026-07-02T00:49:42.827Z] ++ copy_whls=1
[2026-07-02T00:49:42.827Z] ++ shift
[2026-07-02T00:49:42.827Z] ++ [[ 0 -gt 0 ]]
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-02T00:49:42.827Z] ++ [[ false == \t\r\u\e ]]
[2026-07-02T00:49:42.827Z] ++ [[ true == \t\r\u\e ]]
[2026-07-02T00:49:42.827Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-02T00:49:42.827Z] ++ docker ps -a
[2026-07-02T00:49:42.828Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-02T00:49:42.828Z] f86703430d15 rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 "bash -lc 'sleep inf…" 3 minutes ago Up 3 minutes pytorch-ci-container
[2026-07-02T00:49:42.828Z] ++ docker rm -f pytorch-ci-container
[2026-07-02T00:49:44.596Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:49:45.716Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-02T00:49:45.716Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-07-02T00:49:45.716Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:49:45.716Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:49:45.716Z] ++ GPU_FLAGS=
[2026-07-02T00:49:45.716Z] ++ echo 'NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908'
[2026-07-02T00:49:45.716Z] NODE_LABELS: multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908
[2026-07-02T00:49:45.716Z] ++ [[ ! multi_gpu rocm pytorch-nightly selftest ixt-hq-15 gfx908 =~ build-only ]]
[2026-07-02T00:49:45.716Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-07-02T00:49:45.716Z] ++++ dirname prepare_docker_env.sh
[2026-07-02T00:49:45.716Z] +++ cd .
[2026-07-02T00:49:45.716Z] +++ pwd
[2026-07-02T00:49:45.716Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389
[2026-07-02T00:49:45.716Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-07-02T00:49:45.716Z] 5cdf837c5f9860ab1a1f69f72f5c774aabc67ed8500928a09dfd6f5f4eb65997
[2026-07-02T00:49:45.716Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T00:49:46.830Z] Exporting test times from test-infra
[2026-07-02T00:49:46.830Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:49:46.830Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T00:49:46.830Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/tools/stats/export_test_times.py
[2026-07-02T00:49:46.830Z] Exporting test times from test-infra
[2026-07-02T00:49:46.830Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:49:46.830Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-class-times.json
[2026-07-02T00:49:46.830Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files ]]
[2026-07-02T00:49:46.830Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch/.additional_ci_files/test-times.json
[2026-07-02T00:49:46.830Z] ++ [[ false != \t\r\u\e ]]
[2026-07-02T00:49:46.830Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-07-02T00:49:46.830Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T00:49:54.595Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-07-02T00:49:54.595Z] ++ [[ 1 -eq 1 ]]
[2026-07-02T00:49:54.595Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3389/torch-2.14.0a0+git8ddc477-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-07-02T00:49:54.595Z] "docker cp" requires exactly 2 arguments.
[2026-07-02T00:49:54.595Z] See 'docker cp --help'.
[2026-07-02T00:49:54.595Z]
[2026-07-02T00:49:54.595Z] Usage: docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-07-02T00:49:54.595Z] docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-07-02T00:49:54.595Z]
[2026-07-02T00:49:54.595Z] Copy files/folders between a container and the local filesystem
Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal
Error in error step, with arguments pytorch_distributed_2 failed.
pytorch_distributed_2 failed
Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script
Error in sh step, with arguments ./test_pytorch_inductor.sh.
script returned exit code 1
Build log
[2026-07-02T00:42:53.040Z] + ./test_pytorch_inductor.sh
[2026-07-02T00:42:53.040Z] + [[ -z '' ]]
[2026-07-02T00:42:53.040Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls'
[2026-07-02T00:42:53.040Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-07-02T00:42:53.040Z] ++ set -ex
[2026-07-02T00:42:53.040Z] ++ set -o pipefail
[2026-07-02T00:42:53.040Z] ++ copy_whls=0
[2026-07-02T00:42:53.040Z] ++ envvars=()
[2026-07-02T00:42:53.040Z] ++ [[ 11 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ key=-e
[2026-07-02T00:42:53.040Z] ++ case $key in
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e $2")
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ [[ 9 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ key=-e
[2026-07-02T00:42:53.040Z] ++ case $key in
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e $2")
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ [[ 7 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ key=-e
[2026-07-02T00:42:53.040Z] ++ case $key in
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e $2")
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ [[ 5 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ key=-e
[2026-07-02T00:42:53.040Z] ++ case $key in
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e $2")
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ [[ 3 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ key=-e
[2026-07-02T00:42:53.040Z] ++ case $key in
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e $2")
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ [[ 1 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ key=--copy-whls
[2026-07-02T00:42:53.040Z] ++ case $key in
[2026-07-02T00:42:53.040Z] ++ copy_whls=1
[2026-07-02T00:42:53.040Z] ++ shift
[2026-07-02T00:42:53.040Z] ++ [[ 0 -gt 0 ]]
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-07-02T00:42:53.040Z] ++ [[ false == \t\r\u\e ]]
[2026-07-02T00:42:53.040Z] ++ [[ true == \t\r\u\e ]]
[2026-07-02T00:42:53.040Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-07-02T00:42:53.040Z] ++ docker ps -a
[2026-07-02T00:42:53.040Z] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[2026-07-02T00:42:53.040Z] ++ docker rm -f pytorch-ci-container
[2026-07-02T00:42:53.040Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-07-02T00:42:54.380Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-07-02T00:42:55.190Z] cb259a83ac3d: Pulling fs layer
[2026-07-02T00:42:55.190Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-07-02T00:42:55.190Z] dedc784bccba: Pulling fs layer
[2026-07-02T00:42:55.190Z] 1057fba0c7df: Pulling fs layer
[2026-07-02T00:42:55.190Z] 4cfd21f31c8b: Pulling fs layer
[2026-07-02T00:42:55.190Z] 99e6462d03e0: Pulling fs layer
[2026-07-02T00:42:55.190Z] c0664a7365cb: Pulling fs layer
[2026-07-02T00:42:55.190Z] 42239a12ed73: Pulling fs layer
[2026-07-02T00:42:55.190Z] 78cb638d6686: Pulling fs layer
[2026-07-02T00:42:55.190Z] 8b3d007242f8: Pulling fs layer
[2026-07-02T00:42:55.190Z] 34e29023481f: Pulling fs layer
[2026-07-02T00:42:55.190Z] 8d78d529f04d: Pulling fs layer
[2026-07-02T00:42:55.190Z] 95d704643610: Pulling fs layer
[2026-07-02T00:42:55.190Z] c608e00b1282: Pulling fs layer
[2026-07-02T00:42:55.190Z] cccff27d82fb: Pulling fs layer
[2026-07-02T00:42:55.190Z] ed86e5a5d9b1: Pulling fs layer
[2026-07-02T00:42:55.190Z] b96eaed0e751: Pulling fs layer
[2026-07-02T00:42:55.190Z] d9a8b776bea7: Pulling fs layer
[2026-07-02T00:42:55.190Z] 3d56742d201d: Pulling fs layer
[2026-07-02T00:42:55.190Z] f391428cfe0f: Pulling fs layer
[2026-07-02T00:42:55.190Z] f5afebc61536: Pulling fs layer
[2026-07-02T00:42:55.190Z] 8d998c79d309: Pulling fs layer
[2026-07-02T00:42:55.190Z] fef65a673dc9: Pulling fs layer
[2026-07-02T00:42:55.190Z] 906ac4de0a2b: Pulling fs layer
[2026-07-02T00:42:55.190Z] 0a0ffadd8b4f: Pulling fs layer
[2026-07-02T00:42:55.190Z] f3058198fba5: Pulling fs layer
[2026-07-02T00:42:55.190Z] 9ae773ccc4b2: Pulling fs layer
[2026-07-02T00:42:55.190Z] 423e26ddb517: Pulling fs layer
[2026-07-02T00:42:55.190Z] 27fcad4d96a6: Pulling fs layer
[2026-07-02T00:42:55.190Z] 241cbb1c2f26: Pulling fs layer
[2026-07-02T00:42:55.190Z] 4c8a34755929: Pulling fs layer
[2026-07-02T00:42:55.190Z] bf3e4cf00cd6: Pulling fs layer
[2026-07-02T00:42:55.190Z] f790d498bebb: Pulling fs layer
[2026-07-02T00:42:55.190Z] f0ef294e4f9b: Pulling fs layer
[2026-07-02T00:42:55.190Z] 5edf52847c35: Pulling fs layer
[2026-07-02T00:42:55.190Z] 4f4fb700ef54: Pulling fs layer
[2026-07-02T00:42:55.190Z] 9548298aa5f4: Pulling fs layer
[2026-07-02T00:42:55.190Z] d7df6379f9cc: Pulling fs layer
[2026-07-02T00:42:55.190Z] b09b7d04a8d9: Pulling fs layer
[2026-07-02T00:42:55.190Z] 3d56742d201d: Pulling fs layer
Output truncated.
Details
- Kill older PR Builds (2.6 sec)
- Initialize (2 hr 31 min)
- Download CI scripts (9 min 27 sec)
- Checkout Pytorch (15 min)
- Check base Docker image existence (9.3 sec)
- Pull Docker Image (5 min 17 sec)
- Build PyTorch (1 hr 59 min)
- Tests (1 hr 17 min)
- Test PyTorch (7 ms)
- Test Distributed (11 ms)
- Test Inductor (7 ms)
- Test PyTorch Slow (7 ms)
- Test PyTorch Slow (7.3 sec)
- Microbenchmark (14 sec)
- Microbenchmark (7.4 sec)
- Post Build (3.2 sec)
- Declarative: Post Actions (5.3 sec)
Loading