Skip to content

Add biweekly scheduled develop IFU workflow#3379

Open
jithunnair-amd wants to merge 3 commits into
developfrom
jnair/develop-ifu-schedule
Open

Add biweekly scheduled develop IFU workflow#3379
jithunnair-amd wants to merge 3 commits into
developfrom
jnair/develop-ifu-schedule

Update pytorch_develop_ifu_schedule.yml

34b6825
Select commit
Loading
Failed to load commit list.
ROCm Repo Management API / Jenkins failed Jun 30, 2026 in 4h 53m 43s

Tests/Test Inductor/Run pytorch_inductor_1: error in 'error' step

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-06-30T01:32:10.075Z] + ./test_pytorch_test.sh
[2026-06-30T01:32:10.075Z] + [[ -z '' ]]
[2026-06-30T01:32:10.075Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-06-30T01:32:10.075Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-06-30T01:32:10.075Z] ++ set -ex
[2026-06-30T01:32:10.075Z] ++ set -o pipefail
[2026-06-30T01:32:10.075Z] ++ copy_whls=0
[2026-06-30T01:32:10.075Z] ++ envvars=()
[2026-06-30T01:32:10.075Z] ++ [[ 13 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=-e
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e $2")
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 11 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=-e
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e $2")
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 9 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=-e
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e $2")
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 7 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=-e
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e $2")
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 5 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=-e
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e $2")
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 3 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=-e
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e $2")
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 1 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ key=--copy-whls
[2026-06-30T01:32:10.075Z] ++ case $key in
[2026-06-30T01:32:10.075Z] ++ copy_whls=1
[2026-06-30T01:32:10.075Z] ++ shift
[2026-06-30T01:32:10.075Z] ++ [[ 0 -gt 0 ]]
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-06-30T01:32:10.075Z] ++ [[ false == \t\r\u\e ]]
[2026-06-30T01:32:10.075Z] ++ [[ true == \t\r\u\e ]]
[2026-06-30T01:32:10.075Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-06-30T01:32:10.075Z] ++ docker ps -a
[2026-06-30T01:32:10.075Z] CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[2026-06-30T01:32:10.075Z] ++ docker rm -f pytorch-ci-container
[2026-06-30T01:32:10.075Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:32:11.372Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-06-30T01:32:11.777Z] cb259a83ac3d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 99e6462d03e0: Pulling fs layer
[2026-06-30T01:32:11.777Z] 4c8a34755929: Pulling fs layer
[2026-06-30T01:32:11.777Z] 95d704643610: Pulling fs layer
[2026-06-30T01:32:11.777Z] b96eaed0e751: Pulling fs layer
[2026-06-30T01:32:11.777Z] 86cc05da26d1: Pulling fs layer
[2026-06-30T01:32:11.777Z] d7df6379f9cc: Pulling fs layer
[2026-06-30T01:32:11.777Z] 0a0ffadd8b4f: Pulling fs layer
[2026-06-30T01:32:11.777Z] 3d56742d201d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 8d998c79d309: Pulling fs layer
[2026-06-30T01:32:11.777Z] 3d56742d201d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 423e26ddb517: Pulling fs layer
[2026-06-30T01:32:11.777Z] 4cfd21f31c8b: Pulling fs layer
[2026-06-30T01:32:11.777Z] 906ac4de0a2b: Pulling fs layer
[2026-06-30T01:32:11.777Z] fef65a673dc9: Pulling fs layer
[2026-06-30T01:32:11.777Z] 3d56742d201d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 8d78d529f04d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 27fcad4d96a6: Pulling fs layer
[2026-06-30T01:32:11.777Z] 8b3d007242f8: Pulling fs layer
[2026-06-30T01:32:11.777Z] 78cb638d6686: Pulling fs layer
[2026-06-30T01:32:11.777Z] d9a8b776bea7: Pulling fs layer
[2026-06-30T01:32:11.777Z] ed874654c321: Pulling fs layer
[2026-06-30T01:32:11.777Z] db0af168a1a4: Pulling fs layer
[2026-06-30T01:32:11.777Z] f5afebc61536: Pulling fs layer
[2026-06-30T01:32:11.777Z] dedc784bccba: Pulling fs layer
[2026-06-30T01:32:11.777Z] 34e29023481f: Pulling fs layer
[2026-06-30T01:32:11.777Z] c0664a7365cb: Pulling fs layer
[2026-06-30T01:32:11.777Z] 3d56742d201d: Pulling fs layer
[2026-06-30T01:32:11.777Z] 5f8dd10c9c78: Pulling fs layer
[2026-06-30T01:32:11.777Z] 4f4fb700ef54: Pulling fs layer
[2026-06-30T01:32:11.777Z] f391428cfe0f: Pulling fs layer
[2026-06-30T01:32:11.777Z] f790d498bebb: Pulling fs layer
[2026-06-30T01:32:11.777Z] 1057fba0c7df: Pulling fs layer
[2026-06-30T01:32:11.777Z] c608e00b1282: Pulling fs layer
[2026-06-30T01:32:11.777Z] cccff27d82fb: Pulling fs layer
[2026-06-30T01:32:11.777Z] eab367f1ca39: Pulling fs layer
[2026-06-30T01:32:11.777Z] 3d56742d201d: Pulling fs layer
[2026-06-30T01:32:11.777Z] b09b7d04a8d9: Pulling fs layer
[2026-06-30T01:32:11.777Z] f3058198fba5: Pulling fs layer
[2026-06-30T01:32:11.777Z] 5edf52847c35: Pulling fs layer
[2026-06-30T01:32:11.777Z] 747272797eac: Pulling fs layer
[2026-06-30T01:32:11.777Z] 9ae773ccc4b2: Pulling fs layer
[2026-06-30T01:32:11.777Z] 4f4fb700ef54: Pulling fs layer
[2026-06-30T01:32:11.777Z] 42239a12ed73: Pulling fs layer
[2026-06-30T01:32:11.777Z] 241cbb1c2f26: Pulling fs layer
[2026-06-30T01:32:11.777Z] f0ef294e4f9b: Pulling fs layer
[2026-06-30T01:32:11.777Z] 284e8cbb8a05: Pulling fs layer
[2026-06-30T01:32:11.777Z] 78a3c5370db4: Pulling fs layer
[2026-06-30T01:32:11.777Z] 0fcc3b23bd99: Pulling fs layer
[2026-06-30T01:32:11.777Z] 07b45639fc6f: Pulling fs layer
[2026-06-30T01:32:11.777Z] bf3e4cf00cd6: Pulling fs layer
[2026-06-30T01:32:11.777Z] ed86e5a5d9b1: Pulling fs layer
[2026-06-30T01:32:11.777Z] 9548298aa5f4: Pulling fs layer
[2026-06-30T01:32:11.777Z] ba1fabf878bd: Pulling fs layer
[2026-06-30T01:32:11.777Z] 65f281ce2e70: Pulling fs layer
[2026-06-30T01:32:11.777Z] 4106a853273f: Pulling fs layer
[2026-06-30T01:32:12.196Z] 69145e0460f6: Download complete
[2026-06-30T01:32:12.196Z] b96eaed0e751: Download complete
[2026-06-30T01:32:12.196Z] 86cc05da26d1: Download complete
[2026-06-30T01:32:12.583Z] f790d498bebb: Download complete
[2026-06-30T01:32:12.583Z] 906ac4de0a2b: Download complete
[2026-06-30T01:32:12.583Z] 3d56742d201d: Download complete
[2026-06-30T01:32:12.583Z] 99e6462d03e0: Download complete
[2026-06-30T01:32:12.583Z] 0a0ffadd8b4f: Download complete
[2026-06-30T01:32:12.583Z] fef65a673dc9: Download complete
[2026-06-30T01:32:12.583Z] d7df6379f9cc: Download complete
[2026-06-30T01:32:12.583Z] 8d998c79d309: Download complete
[2026-06-30T01:32:12.583Z] 423e26ddb517: Download complete
[2026-06-30T01:32:12.583Z] ed874654c321: Download complete
[2026-06-30T01:32:12.583Z] 4cfd21f31c8b: Download complete
[2026-06-30T01:32:12.583Z] 8b3d007242f8: Download complete
[2026-06-30T01:32:12.583Z] 4f4fb700ef54: Download complete
[2026-06-30T01:32:12.583Z] 27fcad4d96a6: Download complete
[2026-06-30T01:32:12.583Z] f391428cfe0f: Download complete
[2026-06-30T01:32:12.583Z] 0fcc3b23bd99: Download complete
[2026-06-30T01:32:12.583Z] f3058198fba5: Download complete
[2026-06-30T01:32:12.583Z] 07b45639fc6f: Download complete
[2026-06-30T01:32:12.583Z] 5f8dd10c9c78: Download complete
[2026-06-30T01:32:12.583Z] 78a3c5370db4: Download complete
[2026-06-30T01:32:12.583Z] 34e29023481f: Download complete
[2026-06-30T01:32:12.583Z] db0af168a1a4: Download complete
[2026-06-30T01:32:12.583Z] 284e8cbb8a05: Download complete
[2026-06-30T01:32:12.583Z] c0664a7365cb: Download complete
[2026-06-30T01:32:12.583Z] bf3e4cf00cd6: Download complete
[2026-06-30T01:32:12.583Z] 8d78d529f04d: Download complete
[2026-06-30T01:32:12.583Z] 42239a12ed73: Download complete
[2026-06-30T01:32:12.583Z] 9ae773ccc4b2: Download complete
[2026-06-30T01:32:12.583Z] b09b7d04a8d9: Download complete
[2026-06-30T01:32:12.583Z] 65f281ce2e70: Download complete
[2026-06-30T01:32:12.583Z] c608e00b1282: Download complete
[2026-06-30T01:32:12.583Z] eab367f1ca39: Download complete
[2026-06-30T01:32:12.583Z] f5afebc61536: Download complete
[2026-06-30T01:32:12.583Z] 747272797eac: Download complete
[2026-06-30T01:32:12.583Z] 241cbb1c2f26: Download complete
[2026-06-30T01:32:12.583Z] ed86e5a5d9b1: Download complete
[2026-06-30T01:32:12.583Z] f0ef294e4f9b: Download complete
[2026-06-30T01:32:12.583Z] 9548298aa5f4: Download complete
[2026-06-30T01:32:12.583Z] 0e6d9b5f5e8d: Download complete
[2026-06-30T01:32:12.963Z] 4106a853273f: Download complete
[2026-06-30T01:32:12.963Z] cb259a83ac3d: Download complete
[2026-06-30T01:32:13.345Z] 4c8a34755929: Download complete
[2026-06-30T01:32:13.345Z] ba1fabf878bd: Download complete
[2026-06-30T01:32:13.728Z] 95d704643610: Download complete
[2026-06-30T01:32:13.728Z] 99e6462d03e0: Pull complete
[2026-06-30T01:32:13.728Z] cb259a83ac3d: Pull complete
[2026-06-30T01:32:15.611Z] cccff27d82fb: Download complete
[2026-06-30T01:32:19.765Z] 1057fba0c7df: Download complete
[2026-06-30T01:32:27.594Z] 1057fba0c7df: Pull complete
[2026-06-30T01:32:27.594Z] 906ac4de0a2b: Pull complete
[2026-06-30T01:32:27.594Z] d7df6379f9cc: Pull complete
[2026-06-30T01:32:27.594Z] 34e29023481f: Pull complete
[2026-06-30T01:32:27.594Z] 8d78d529f04d: Pull complete
[2026-06-30T01:32:27.594Z] 65f281ce2e70: Pull complete
[2026-06-30T01:32:27.594Z] c608e00b1282: Pull complete
[2026-06-30T01:32:27.594Z] 0e6d9b5f5e8d: Pull complete
[2026-06-30T01:32:30.666Z] 78cb638d6686: Download complete
[2026-06-30T01:32:59.555Z] 5edf52847c35: Download complete
[2026-06-30T01:33:41.271Z] 86cc05da26d1: Pull complete
[2026-06-30T01:33:41.271Z] 3d56742d201d: Pull complete
[2026-06-30T01:33:41.271Z] 0a0ffadd8b4f: Pull complete
[2026-06-30T01:33:41.271Z] ed874654c321: Pull complete
[2026-06-30T01:33:41.271Z] db0af168a1a4: Pull complete
[2026-06-30T01:33:41.271Z] 747272797eac: Pull complete
[2026-06-30T01:33:41.271Z] 9548298aa5f4: Pull complete
[2026-06-30T01:33:41.271Z] 5edf52847c35: Pull complete
[2026-06-30T01:33:41.271Z] 8b3d007242f8: Pull complete
[2026-06-30T01:34:30.635Z] d9a8b776bea7: Download complete
[2026-06-30T01:35:55.114Z] dedc784bccba: Download complete
[2026-06-30T01:37:35.036Z] 27fcad4d96a6: Pull complete
[2026-06-30T01:37:35.036Z] bf3e4cf00cd6: Pull complete
[2026-06-30T01:37:35.036Z] dedc784bccba: Pull complete
[2026-06-30T01:37:35.036Z] 9ae773ccc4b2: Pull complete
[2026-06-30T01:37:41.409Z] 78cb638d6686: Pull complete
[2026-06-30T01:37:41.409Z] f3058198fba5: Pull complete
[2026-06-30T01:37:41.794Z] f5afebc61536: Pull complete
[2026-06-30T01:37:44.342Z] b96eaed0e751: Pull complete
[2026-06-30T01:37:44.342Z] f790d498bebb: Pull complete
[2026-06-30T01:37:44.342Z] 07b45639fc6f: Pull complete
[2026-06-30T01:37:44.342Z] c0664a7365cb: Pull complete
[2026-06-30T01:37:44.342Z] cccff27d82fb: Pull complete
[2026-06-30T01:37:45.153Z] 423e26ddb517: Pull complete
[2026-06-30T01:37:45.153Z] 5f8dd10c9c78: Pull complete
[2026-06-30T01:37:45.153Z] 4106a853273f: Pull complete
[2026-06-30T01:37:45.153Z] 95d704643610: Pull complete
[2026-06-30T01:37:45.153Z] 4cfd21f31c8b: Pull complete
[2026-06-30T01:37:45.153Z] 0fcc3b23bd99: Pull complete
[2026-06-30T01:37:45.153Z] 78a3c5370db4: Pull complete
[2026-06-30T01:37:45.153Z] 284e8cbb8a05: Pull complete
[2026-06-30T01:37:45.153Z] 42239a12ed73: Pull complete
[2026-06-30T01:37:45.153Z] b09b7d04a8d9: Pull complete
[2026-06-30T01:37:45.153Z] 241cbb1c2f26: Pull complete
[2026-06-30T01:37:45.153Z] f0ef294e4f9b: Pull complete
[2026-06-30T01:38:55.560Z] f391428cfe0f: Pull complete
[2026-06-30T01:38:55.560Z] eab367f1ca39: Pull complete
[2026-06-30T01:38:55.560Z] d9a8b776bea7: Pull complete
[2026-06-30T01:38:55.560Z] fef65a673dc9: Pull complete
[2026-06-30T01:38:55.560Z] 4f4fb700ef54: Pull complete
[2026-06-30T01:38:55.560Z] ed86e5a5d9b1: Pull complete
[2026-06-30T01:38:55.560Z] ba1fabf878bd: Pull complete
[2026-06-30T01:38:55.560Z] 8d998c79d309: Pull complete
[2026-06-30T01:38:55.560Z] 4c8a34755929: Pull complete
[2026-06-30T01:38:55.560Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-06-30T01:38:55.560Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:38:55.560Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:38:55.560Z] ++ GPU_FLAGS=
[2026-06-30T01:38:55.560Z] ++ echo 'NODE_LABELS: gfx90a k8s-mi250-1-bp7k2 mlir gfx90a k8s-mi250-1-bp7k2 mlir'
[2026-06-30T01:38:55.560Z] NODE_LABELS: gfx90a k8s-mi250-1-bp7k2 mlir gfx90a k8s-mi250-1-bp7k2 mlir
[2026-06-30T01:38:55.560Z] ++ [[ ! gfx90a k8s-mi250-1-bp7k2 mlir =~ build-only ]]
[2026-06-30T01:38:55.560Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-06-30T01:38:55.560Z] ++++ dirname prepare_docker_env.sh
[2026-06-30T01:38:55.560Z] +++ cd .
[2026-06-30T01:38:55.560Z] +++ pwd
[2026-06-30T01:38:55.560Z] ++ script_dir=/home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379
[2026-06-30T01:38:55.560Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379:/host_workspace -v /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-06-30T01:38:55.560Z] 7bcef1eae2f7cbd257415dc2404a22a216815ddefa18f03130fae7077453cb4b
[2026-06-30T01:38:55.560Z] ++ env -i python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T01:38:55.560Z] Exporting test times from test-infra
[2026-06-30T01:38:55.560Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:38:55.560Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T01:38:55.560Z] ++ python3 /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T01:38:55.560Z] Exporting test times from test-infra
[2026-06-30T01:38:55.560Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:38:55.560Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T01:38:55.560Z] ++ [[ -d /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files ]]
[2026-06-30T01:38:55.560Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:38:55.560Z] ++ [[ false != \t\r\u\e ]]
[2026-06-30T01:38:55.560Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-06-30T01:38:55.560Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T01:38:57.434Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-06-30T01:38:57.434Z] ++ [[ 1 -eq 1 ]]
[2026-06-30T01:38:57.434Z] ++ docker cp /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/agent/workspace/orch_pytorch-ci-pipeline_PR-3379/torch-2.14.0a0+git34b6825-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T01:38:57.434Z] docker: 'docker cp' requires 2 arguments
[2026-06-30T01:38:57.434Z] 
[2026-06-30T01:38:57.434Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-06-30T01:38:57.434Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-06-30T01:38:57.434Z] 
[2026-06-30T01:38:57.434Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_1 / Error signal

Error in error step, with arguments pytorch_test_1 failed.

pytorch_test_1 failed

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test.sh.

script returned exit code 1
Build log 
[2026-06-30T01:59:53.612Z] + ./test_pytorch_test.sh
[2026-06-30T01:59:53.612Z] + [[ -z '' ]]
[2026-06-30T01:59:53.612Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-06-30T01:59:53.612Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-06-30T01:59:53.612Z] ++ set -ex
[2026-06-30T01:59:53.612Z] ++ set -o pipefail
[2026-06-30T01:59:53.612Z] ++ copy_whls=0
[2026-06-30T01:59:53.612Z] ++ envvars=()
[2026-06-30T01:59:53.612Z] ++ [[ 13 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=-e
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e $2")
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 11 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=-e
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e $2")
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 9 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=-e
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e $2")
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 7 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=-e
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e $2")
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 5 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=-e
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e $2")
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 3 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=-e
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e $2")
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 1 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ key=--copy-whls
[2026-06-30T01:59:53.612Z] ++ case $key in
[2026-06-30T01:59:53.612Z] ++ copy_whls=1
[2026-06-30T01:59:53.612Z] ++ shift
[2026-06-30T01:59:53.612Z] ++ [[ 0 -gt 0 ]]
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-06-30T01:59:53.612Z] ++ [[ false == \t\r\u\e ]]
[2026-06-30T01:59:53.612Z] ++ [[ true == \t\r\u\e ]]
[2026-06-30T01:59:53.612Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-06-30T01:59:53.612Z] ++ docker ps -a
[2026-06-30T01:59:53.612Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED         STATUS                   PORTS                                                 NAMES
[2026-06-30T01:59:53.612Z] 2c83fcc423de   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   3 minutes ago   Up 3 minutes                                                                   pytorch-ci-container
[2026-06-30T01:59:53.612Z] 33af8c9e058f   dashboard-dashboard-ui                                                                                                       "/docker-entrypoint.…"   7 days ago      Up 5 hours               80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp   rocagents-dashboard-ui
[2026-06-30T01:59:53.612Z] 7c6be83a2db1   dashboard-dashboard-api                                                                                                      "uvicorn main:app --…"   7 days ago      Up 5 hours (healthy)     0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp           rocagents-dashboard-api
[2026-06-30T01:59:53.612Z] 780dc8fe5a7b   rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen                                                                                  "/scripts/run_tests_…"   3 weeks ago     Exited (1) 3 weeks ago                                                         infallible_chatelet
[2026-06-30T01:59:53.612Z] 2d7786cfd066   deployment-rocagents                                                                                                         "python src/multi_re…"   6 weeks ago     Up 5 hours (healthy)                                                           rocagents-orchestrator
[2026-06-30T01:59:53.612Z] ++ docker rm -f pytorch-ci-container
[2026-06-30T01:59:53.612Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:59:54.142Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-06-30T01:59:54.142Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-06-30T01:59:54.142Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:59:54.142Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:59:54.142Z] ++ GPU_FLAGS=
[2026-06-30T01:59:54.142Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-06-30T01:59:54.142Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-06-30T01:59:54.142Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-06-30T01:59:54.142Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-06-30T01:59:54.142Z] ++++ dirname prepare_docker_env.sh
[2026-06-30T01:59:54.142Z] +++ cd .
[2026-06-30T01:59:54.142Z] +++ pwd
[2026-06-30T01:59:54.674Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379
[2026-06-30T01:59:54.674Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=default -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-06-30T01:59:54.674Z] 1b34995a1ed0175587a63643cd5394d41e48dcf04b2508170edb948b581609b3
[2026-06-30T01:59:54.674Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T01:59:55.204Z] Exporting test times from test-infra
[2026-06-30T01:59:55.204Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:59:55.204Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T01:59:55.204Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T01:59:55.204Z] Exporting test times from test-infra
[2026-06-30T01:59:55.204Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:59:55.204Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T01:59:55.204Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files ]]
[2026-06-30T01:59:55.204Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:59:55.204Z] ++ [[ false != \t\r\u\e ]]
[2026-06-30T01:59:55.204Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-06-30T01:59:55.204Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T02:00:00.432Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-06-30T02:00:00.432Z] ++ [[ 1 -eq 1 ]]
[2026-06-30T02:00:00.432Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/torch-2.14.0a0+git34b6825-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T02:00:00.432Z] docker: 'docker cp' requires 2 arguments
[2026-06-30T02:00:00.432Z] 
[2026-06-30T02:00:00.432Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-06-30T02:00:00.432Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-06-30T02:00:00.432Z] 
[2026-06-30T02:00:00.432Z] Run 'docker cp --help' for more information

Tests / Test PyTorch / Test PyTorch / Run pytorch_test_2 / Error signal

Error in error step, with arguments pytorch_test_2 failed.

pytorch_test_2 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-06-30T01:48:57.870Z] + ./test_pytorch_test_distributed.sh
[2026-06-30T01:48:57.870Z] + [[ -z '' ]]
[2026-06-30T01:48:57.870Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-06-30T01:48:57.870Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-06-30T01:48:57.870Z] ++ set -ex
[2026-06-30T01:48:57.870Z] ++ set -o pipefail
[2026-06-30T01:48:57.870Z] ++ copy_whls=0
[2026-06-30T01:48:57.870Z] ++ envvars=()
[2026-06-30T01:48:57.870Z] ++ [[ 13 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=-e
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e $2")
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 11 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=-e
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e $2")
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 9 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=-e
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e $2")
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 7 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=-e
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e $2")
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 5 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=-e
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e $2")
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 3 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=-e
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e $2")
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 1 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ key=--copy-whls
[2026-06-30T01:48:57.870Z] ++ case $key in
[2026-06-30T01:48:57.870Z] ++ copy_whls=1
[2026-06-30T01:48:57.870Z] ++ shift
[2026-06-30T01:48:57.870Z] ++ [[ 0 -gt 0 ]]
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-06-30T01:48:57.870Z] ++ [[ false == \t\r\u\e ]]
[2026-06-30T01:48:57.870Z] ++ [[ true == \t\r\u\e ]]
[2026-06-30T01:48:57.870Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-06-30T01:48:57.870Z] ++ docker ps -a
[2026-06-30T01:48:57.870Z] CONTAINER ID   IMAGE                                           COMMAND                  CREATED       STATUS                   PORTS                                                 NAMES
[2026-06-30T01:48:57.870Z] fb60f3874c1e   rocm/pytorch-private:ubuntu20.04_py3.9_pip3.9   "bash -lc 'sleep inf…"   2 hours ago   Up 2 hours                                                                     pytorch-ci-container
[2026-06-30T01:48:57.870Z] 33af8c9e058f   dashboard-dashboard-ui                          "/docker-entrypoint.…"   7 days ago    Up 5 hours               80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp   rocagents-dashboard-ui
[2026-06-30T01:48:57.870Z] 7c6be83a2db1   dashboard-dashboard-api                         "uvicorn main:app --…"   7 days ago    Up 5 hours (healthy)     0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp           rocagents-dashboard-api
[2026-06-30T01:48:57.870Z] 780dc8fe5a7b   rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen     "/scripts/run_tests_…"   3 weeks ago   Exited (1) 3 weeks ago                                                         infallible_chatelet
[2026-06-30T01:48:57.870Z] 2d7786cfd066   deployment-rocagents                            "python src/multi_re…"   6 weeks ago   Up 5 hours (healthy)                                                           rocagents-orchestrator
[2026-06-30T01:48:57.870Z] ++ docker rm -f pytorch-ci-container
[2026-06-30T01:48:59.641Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:49:00.765Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-06-30T01:49:00.765Z] cb259a83ac3d: Already exists
[2026-06-30T01:49:00.765Z] 99e6462d03e0: Pulling fs layer
[2026-06-30T01:49:00.765Z] 1057fba0c7df: Pulling fs layer
[2026-06-30T01:49:00.765Z] 0e6d9b5f5e8d: Pulling fs layer
[2026-06-30T01:49:00.765Z] 65f281ce2e70: Pulling fs layer
[2026-06-30T01:49:00.765Z] 34e29023481f: Pulling fs layer
[2026-06-30T01:49:00.765Z] 8d78d529f04d: Pulling fs layer
[2026-06-30T01:49:00.765Z] 906ac4de0a2b: Pulling fs layer
[2026-06-30T01:49:00.765Z] c608e00b1282: Pulling fs layer
[2026-06-30T01:49:00.765Z] d7df6379f9cc: Pulling fs layer
[2026-06-30T01:49:00.765Z] 3d56742d201d: Pulling fs layer
[2026-06-30T01:49:00.765Z] 5edf52847c35: Pulling fs layer
[2026-06-30T01:49:00.765Z] 9548298aa5f4: Pulling fs layer
[2026-06-30T01:49:00.765Z] 86cc05da26d1: Pulling fs layer
[2026-06-30T01:49:00.765Z] 747272797eac: Pulling fs layer
[2026-06-30T01:49:00.765Z] db0af168a1a4: Pulling fs layer
[2026-06-30T01:49:00.765Z] ed874654c321: Pulling fs layer
[2026-06-30T01:49:00.765Z] 0a0ffadd8b4f: Pulling fs layer
[2026-06-30T01:49:00.765Z] 8b3d007242f8: Pulling fs layer
[2026-06-30T01:49:00.765Z] dedc784bccba: Pulling fs layer
[2026-06-30T01:49:00.765Z] 27fcad4d96a6: Pulling fs layer
[2026-06-30T01:49:00.765Z] bf3e4cf00cd6: Pulling fs layer
[2026-06-30T01:49:00.765Z] 9ae773ccc4b2: Pulling fs layer
[2026-06-30T01:49:00.765Z] 78cb638d6686: Pulling fs layer
[2026-06-30T01:49:00.765Z] f3058198fba5: Pulling fs layer
[2026-06-30T01:49:00.765Z] f5afebc61536: Pulling fs layer
[2026-06-30T01:49:00.765Z] cccff27d82fb: Pulling fs layer
[2026-06-30T01:49:00.765Z] c0664a7365cb: Pulling fs layer
[2026-06-30T01:49:00.765Z] 07b45639fc6f: Pulling fs layer
[2026-06-30T01:49:00.765Z] b96eaed0e751: Pulling fs layer
[2026-06-30T01:49:00.765Z] f790d498bebb: Pulling fs layer
[2026-06-30T01:49:00.765Z] 95d704643610: Pulling fs layer
[2026-06-30T01:49:00.765Z] 423e26ddb517: Pulling fs layer
[2026-06-30T01:49:00.765Z] 4106a853273f: Pulling fs layer
[2026-06-30T01:49:00.765Z] 5f8dd10c9c78: Pulling fs layer
[2026-06-30T01:49:00.765Z] b09b7d04a8d9: Pulling fs layer
[2026-06-30T01:49:00.765Z] 65f281ce2e70: Waiting
[2026-06-30T01:49:00.765Z] 4cfd21f31c8b: Pulling fs layer
[2026-06-30T01:49:00.765Z] 241cbb1c2f26: Pulling fs layer
[2026-06-30T01:49:00.765Z] f0ef294e4f9b: Pulling fs layer
[2026-06-30T01:49:00.765Z] 9548298aa5f4: Waiting
[2026-06-30T01:49:00.765Z] 4f4fb700ef54: Pulling fs layer
[2026-06-30T01:49:00.765Z] 86cc05da26d1: Waiting
[2026-06-30T01:49:00.765Z] 78a3c5370db4: Pulling fs layer
[2026-06-30T01:49:00.765Z] 34e29023481f: Waiting
[2026-06-30T01:49:00.765Z] 42239a12ed73: Pulling fs layer
[2026-06-30T01:49:00.765Z] 747272797eac: Waiting
[2026-06-30T01:49:00.765Z] 284e8cbb8a05: Pulling fs layer
[2026-06-30T01:49:00.765Z] db0af168a1a4: Waiting
[2026-06-30T01:49:00.765Z] ed874654c321: Waiting
[2026-06-30T01:49:00.765Z] 0fcc3b23bd99: Pulling fs layer
[2026-06-30T01:49:00.765Z] d9a8b776bea7: Pulling fs layer
[2026-06-30T01:49:00.765Z] f391428cfe0f: Pulling fs layer
[2026-06-30T01:49:00.765Z] d7df6379f9cc: Waiting
[2026-06-30T01:49:00.765Z] eab367f1ca39: Pulling fs layer
[2026-06-30T01:49:00.765Z] 8d78d529f04d: Waiting
[2026-06-30T01:49:00.765Z] 5edf52847c35: Waiting
[2026-06-30T01:49:00.765Z] ba1fabf878bd: Pulling fs layer
[2026-06-30T01:49:00.765Z] 906ac4de0a2b: Waiting
[2026-06-30T01:49:00.765Z] 0a0ffadd8b4f: Waiting
[2026-06-30T01:49:00.765Z] ed86e5a5d9b1: Pulling fs layer
[2026-06-30T01:49:00.765Z] c608e00b1282: Waiting
[2026-06-30T01:49:00.765Z] 3d56742d201d: Waiting
[2026-06-30T01:49:00.765Z] fef65a673dc9: Pulling fs layer
[2026-06-30T01:49:00.765Z] 8b3d007242f8: Waiting
[2026-06-30T01:49:00.765Z] 8d998c79d309: Pulling fs layer
[2026-06-30T01:49:00.765Z] 9ae773ccc4b2: Waiting
[2026-06-30T01:49:00.765Z] 4c8a34755929: Pulling fs layer
[2026-06-30T01:49:00.765Z] dedc784bccba: Waiting
[2026-06-30T01:49:00.765Z] c0664a7365cb: Waiting
[2026-06-30T01:49:00.765Z] 4106a853273f: Waiting
[2026-06-30T01:49:00.765Z] bf3e4cf00cd6: Waiting
[2026-06-30T01:49:00.765Z] 5f8dd10c9c78: Waiting
[2026-06-30T01:49:00.765Z] 07b45639fc6f: Waiting
[2026-06-30T01:49:00.765Z] cccff27d82fb: Waiting
[2026-06-30T01:49:00.765Z] b09b7d04a8d9: Waiting
[2026-06-30T01:49:00.765Z] b96eaed0e751: Waiting
[2026-06-30T01:49:00.765Z] d9a8b776bea7: Waiting
[2026-06-30T01:49:00.765Z] 27fcad4d96a6: Waiting
[2026-06-30T01:49:00.765Z] 4cfd21f31c8b: Waiting
[2026-06-30T01:49:00.765Z] f391428cfe0f: Waiting
[2026-06-30T01:49:00.765Z] f790d498bebb: Waiting
[2026-06-30T01:49:00.765Z] f3058198fba5: Waiting
[2026-06-30T01:49:00.765Z] 241cbb1c2f26: Waiting
[2026-06-30T01:49:00.765Z] 423e26ddb517: Waiting
[2026-06-30T01:49:00.765Z] 95d704643610: Waiting
[2026-06-30T01:49:00.765Z] eab367f1ca39: Waiting
[2026-06-30T01:49:00.765Z] f5afebc61536: Waiting
[2026-06-30T01:49:00.765Z] ba1fabf878bd: Waiting
[2026-06-30T01:49:00.765Z] ed86e5a5d9b1: Waiting
[2026-06-30T01:49:00.765Z] 8d998c79d309: Waiting
[2026-06-30T01:49:00.765Z] f0ef294e4f9b: Waiting
[2026-06-30T01:49:00.765Z] 284e8cbb8a05: Waiting
[2026-06-30T01:49:00.765Z] 0fcc3b23bd99: Waiting
[2026-06-30T01:49:00.765Z] 78a3c5370db4: Waiting
[2026-06-30T01:49:00.765Z] fef65a673dc9: Waiting
[2026-06-30T01:49:00.765Z] 4f4fb700ef54: Waiting
[2026-06-30T01:49:00.765Z] 4c8a34755929: Waiting
[2026-06-30T01:49:01.315Z] 99e6462d03e0: Verifying Checksum
[2026-06-30T01:49:01.315Z] 99e6462d03e0: Download complete
[2026-06-30T01:49:01.315Z] 99e6462d03e0: Pull complete
[2026-06-30T01:49:01.315Z] 0e6d9b5f5e8d: Verifying Checksum
[2026-06-30T01:49:01.315Z] 0e6d9b5f5e8d: Download complete
[2026-06-30T01:49:01.868Z] 65f281ce2e70: Verifying Checksum
[2026-06-30T01:49:01.868Z] 65f281ce2e70: Download complete
[2026-06-30T01:49:01.868Z] 34e29023481f: Verifying Checksum
[2026-06-30T01:49:01.868Z] 34e29023481f: Download complete
[2026-06-30T01:49:01.868Z] 8d78d529f04d: Verifying Checksum
[2026-06-30T01:49:01.868Z] 8d78d529f04d: Download complete
[2026-06-30T01:49:01.868Z] 906ac4de0a2b: Verifying Checksum
[2026-06-30T01:49:01.868Z] 906ac4de0a2b: Download complete
[2026-06-30T01:49:02.412Z] d7df6379f9cc: Verifying Checksum
[2026-06-30T01:49:02.412Z] d7df6379f9cc: Download complete
[2026-06-30T01:49:02.412Z] c608e00b1282: Download complete
[2026-06-30T01:49:02.412Z] 3d56742d201d: Verifying Checksum
[2026-06-30T01:49:02.412Z] 3d56742d201d: Download complete
[2026-06-30T01:49:03.307Z] 9548298aa5f4: Verifying Checksum
[2026-06-30T01:49:03.307Z] 9548298aa5f4: Download complete
[2026-06-30T01:49:03.307Z] 86cc05da26d1: Download complete
[2026-06-30T01:49:03.307Z] 747272797eac: Verifying Checksum
[2026-06-30T01:49:03.307Z] 747272797eac: Download complete
[2026-06-30T01:49:03.307Z] db0af168a1a4: Download complete
[2026-06-30T01:49:03.307Z] ed874654c321: Verifying Checksum
[2026-06-30T01:49:03.307Z] ed874654c321: Download complete
[2026-06-30T01:49:03.846Z] 0a0ffadd8b4f: Verifying Checksum
[2026-06-30T01:49:03.846Z] 0a0ffadd8b4f: Download complete
[2026-06-30T01:49:04.387Z] 8b3d007242f8: Download complete
[2026-06-30T01:49:09.717Z] 1057fba0c7df: Download complete
[2026-06-30T01:49:09.717Z] 27fcad4d96a6: Download complete
[2026-06-30T01:49:09.717Z] bf3e4cf00cd6: Verifying Checksum
[2026-06-30T01:49:09.717Z] bf3e4cf00cd6: Download complete
[2026-06-30T01:49:10.281Z] 9ae773ccc4b2: Verifying Checksum
[2026-06-30T01:49:10.281Z] 9ae773ccc4b2: Download complete
[2026-06-30T01:49:20.388Z] 1057fba0c7df: Pull complete
[2026-06-30T01:49:20.388Z] 0e6d9b5f5e8d: Pull complete
[2026-06-30T01:49:20.388Z] 65f281ce2e70: Pull complete
[2026-06-30T01:49:20.388Z] 34e29023481f: Pull complete
[2026-06-30T01:49:20.388Z] 8d78d529f04d: Pull complete
[2026-06-30T01:49:20.388Z] 906ac4de0a2b: Pull complete
[2026-06-30T01:49:20.388Z] c608e00b1282: Pull complete
[2026-06-30T01:49:20.388Z] d7df6379f9cc: Pull complete
[2026-06-30T01:49:20.388Z] 3d56742d201d: Pull complete
[2026-06-30T01:49:40.982Z] 78cb638d6686: Verifying Checksum
[2026-06-30T01:49:40.982Z] 78cb638d6686: Download complete
[2026-06-30T01:49:41.572Z] f3058198fba5: Verifying Checksum
[2026-06-30T01:49:41.572Z] f3058198fba5: Download complete
[2026-06-30T01:49:41.572Z] f5afebc61536: Verifying Checksum
[2026-06-30T01:49:41.572Z] f5afebc61536: Download complete
[2026-06-30T01:49:45.374Z] cccff27d82fb: Verifying Checksum
[2026-06-30T01:49:45.374Z] cccff27d82fb: Download complete
[2026-06-30T01:49:45.374Z] c0664a7365cb: Verifying Checksum
[2026-06-30T01:49:45.374Z] c0664a7365cb: Download complete
[2026-06-30T01:49:45.974Z] 07b45639fc6f: Download complete
[2026-06-30T01:49:45.974Z] b96eaed0e751: Verifying Checksum
[2026-06-30T01:49:45.974Z] b96eaed0e751: Download complete
[2026-06-30T01:49:46.523Z] f790d498bebb: Download complete
[2026-06-30T01:49:48.483Z] 95d704643610: Verifying Checksum
[2026-06-30T01:49:48.483Z] 95d704643610: Download complete
[2026-06-30T01:49:48.483Z] 423e26ddb517: Verifying Checksum
[2026-06-30T01:49:48.483Z] 423e26ddb517: Download complete
[2026-06-30T01:49:49.126Z] 4106a853273f: Verifying Checksum
[2026-06-30T01:49:49.126Z] 4106a853273f: Download complete
[2026-06-30T01:49:49.701Z] 5f8dd10c9c78: Download complete
[2026-06-30T01:49:49.701Z] b09b7d04a8d9: Verifying Checksum
[2026-06-30T01:49:49.701Z] b09b7d04a8d9: Download complete
[2026-06-30T01:49:50.291Z] 4cfd21f31c8b: Download complete
[2026-06-30T01:49:50.291Z] 241cbb1c2f26: Verifying Checksum
[2026-06-30T01:49:50.291Z] 241cbb1c2f26: Download complete
[2026-06-30T01:49:50.896Z] f0ef294e4f9b: Verifying Checksum
[2026-06-30T01:49:50.896Z] f0ef294e4f9b: Download complete
[2026-06-30T01:49:50.896Z] 4f4fb700ef54: Verifying Checksum
[2026-06-30T01:49:50.896Z] 4f4fb700ef54: Download complete
[2026-06-30T01:49:50.896Z] 78a3c5370db4: Verifying Checksum
[2026-06-30T01:49:50.896Z] 78a3c5370db4: Download complete
[2026-06-30T01:49:50.896Z] 42239a12ed73: Verifying Checksum
[2026-06-30T01:49:50.896Z] 42239a12ed73: Download complete
[2026-06-30T01:49:51.503Z] 284e8cbb8a05: Download complete
[2026-06-30T01:49:51.503Z] 0fcc3b23bd99: Verifying Checksum
[2026-06-30T01:49:51.503Z] 0fcc3b23bd99: Download complete
[2026-06-30T01:51:37.595Z] 5edf52847c35: Verifying Checksum
[2026-06-30T01:51:37.595Z] 5edf52847c35: Download complete
[2026-06-30T01:51:37.595Z] f391428cfe0f: Download complete
[2026-06-30T01:51:37.595Z] eab367f1ca39: Download complete
[2026-06-30T01:51:37.595Z] ba1fabf878bd: Verifying Checksum
[2026-06-30T01:51:37.595Z] ba1fabf878bd: Download complete
[2026-06-30T01:51:37.595Z] ed86e5a5d9b1: Verifying Checksum
[2026-06-30T01:51:37.595Z] ed86e5a5d9b1: Download complete
[2026-06-30T01:51:37.595Z] fef65a673dc9: Verifying Checksum
[2026-06-30T01:51:37.595Z] fef65a673dc9: Download complete
[2026-06-30T01:51:37.595Z] 8d998c79d309: Verifying Checksum
[2026-06-30T01:51:37.595Z] 8d998c79d309: Download complete
[2026-06-30T01:51:37.595Z] 4c8a34755929: Download complete
[2026-06-30T01:52:22.109Z] 5edf52847c35: Pull complete
[2026-06-30T01:52:22.109Z] 9548298aa5f4: Pull complete
[2026-06-30T01:52:22.109Z] 86cc05da26d1: Pull complete
[2026-06-30T01:52:22.109Z] 747272797eac: Pull complete
[2026-06-30T01:52:22.109Z] db0af168a1a4: Pull complete
[2026-06-30T01:52:22.109Z] ed874654c321: Pull complete
[2026-06-30T01:52:22.109Z] 0a0ffadd8b4f: Pull complete
[2026-06-30T01:52:22.109Z] 8b3d007242f8: Pull complete
[2026-06-30T01:52:22.109Z] dedc784bccba: Verifying Checksum
[2026-06-30T01:52:22.109Z] dedc784bccba: Download complete
[2026-06-30T01:52:30.145Z] d9a8b776bea7: Verifying Checksum
[2026-06-30T01:52:30.145Z] d9a8b776bea7: Download complete
[2026-06-30T01:54:44.802Z] dedc784bccba: Pull complete
[2026-06-30T01:54:44.802Z] 27fcad4d96a6: Pull complete
[2026-06-30T01:54:44.802Z] bf3e4cf00cd6: Pull complete
[2026-06-30T01:54:44.802Z] 9ae773ccc4b2: Pull complete
[2026-06-30T01:54:44.802Z] 78cb638d6686: Pull complete
[2026-06-30T01:54:44.802Z] f3058198fba5: Pull complete
[2026-06-30T01:54:44.802Z] f5afebc61536: Pull complete
[2026-06-30T01:54:47.291Z] cccff27d82fb: Pull complete
[2026-06-30T01:54:47.291Z] c0664a7365cb: Pull complete
[2026-06-30T01:54:47.291Z] 07b45639fc6f: Pull complete
[2026-06-30T01:54:47.291Z] b96eaed0e751: Pull complete
[2026-06-30T01:54:47.291Z] f790d498bebb: Pull complete
[2026-06-30T01:54:47.818Z] 95d704643610: Pull complete
[2026-06-30T01:54:47.818Z] 423e26ddb517: Pull complete
[2026-06-30T01:54:47.818Z] 4106a853273f: Pull complete
[2026-06-30T01:54:47.818Z] 5f8dd10c9c78: Pull complete
[2026-06-30T01:54:47.818Z] b09b7d04a8d9: Pull complete
[2026-06-30T01:54:47.818Z] 4cfd21f31c8b: Pull complete
[2026-06-30T01:54:47.818Z] 241cbb1c2f26: Pull complete
[2026-06-30T01:54:47.818Z] f0ef294e4f9b: Pull complete
[2026-06-30T01:54:47.818Z] 4f4fb700ef54: Pull complete
[2026-06-30T01:54:47.818Z] 78a3c5370db4: Pull complete
[2026-06-30T01:54:47.818Z] 42239a12ed73: Pull complete
[2026-06-30T01:54:47.818Z] 284e8cbb8a05: Pull complete
[2026-06-30T01:54:47.818Z] 0fcc3b23bd99: Pull complete
[2026-06-30T01:56:15.937Z] d9a8b776bea7: Pull complete
[2026-06-30T01:56:15.937Z] f391428cfe0f: Pull complete
[2026-06-30T01:56:15.937Z] eab367f1ca39: Pull complete
[2026-06-30T01:56:15.937Z] ba1fabf878bd: Pull complete
[2026-06-30T01:56:15.937Z] ed86e5a5d9b1: Pull complete
[2026-06-30T01:56:15.937Z] fef65a673dc9: Pull complete
[2026-06-30T01:56:15.937Z] 8d998c79d309: Pull complete
[2026-06-30T01:56:15.937Z] 4c8a34755929: Pull complete
[2026-06-30T01:56:15.937Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-06-30T01:56:15.937Z] Status: Downloaded newer image for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:56:15.937Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T01:56:15.937Z] ++ GPU_FLAGS=
[2026-06-30T01:56:15.937Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-06-30T01:56:15.937Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-06-30T01:56:15.937Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-06-30T01:56:15.937Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-06-30T01:56:15.937Z] ++++ dirname prepare_docker_env.sh
[2026-06-30T01:56:15.937Z] +++ cd .
[2026-06-30T01:56:15.937Z] +++ pwd
[2026-06-30T01:56:15.937Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379
[2026-06-30T01:56:15.937Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=1 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-06-30T01:56:15.937Z] 2c83fcc423de9eb0d564a6d97d0cb55924300c5e9b4c1dc7f8a64c0b17164bcf
[2026-06-30T01:56:15.937Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T01:56:17.794Z] Exporting test times from test-infra
[2026-06-30T01:56:17.794Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:56:17.794Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T01:56:17.794Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T01:56:17.794Z] Exporting test times from test-infra
[2026-06-30T01:56:17.794Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:56:17.794Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T01:56:18.350Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files ]]
[2026-06-30T01:56:18.350Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T01:56:18.350Z] ++ [[ false != \t\r\u\e ]]
[2026-06-30T01:56:18.350Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-06-30T01:56:18.350Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T01:56:22.620Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-06-30T01:56:22.620Z] ++ [[ 1 -eq 1 ]]
[2026-06-30T01:56:22.620Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/torch-2.14.0a0+git34b6825-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T01:56:22.845Z] docker: 'docker cp' requires 2 arguments
[2026-06-30T01:56:22.845Z] 
[2026-06-30T01:56:22.845Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-06-30T01:56:22.845Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-06-30T01:56:22.845Z] 
[2026-06-30T01:56:22.845Z] Run 'docker cp --help' for more information

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_1 / Error signal

Error in error step, with arguments pytorch_distributed_1 failed.

pytorch_distributed_1 failed

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Shell Script

Error in sh step, with arguments ./test_pytorch_test_distributed.sh.

script returned exit code 1
Build log 
[2026-06-30T02:04:03.924Z] + ./test_pytorch_test_distributed.sh
[2026-06-30T02:04:03.924Z] + [[ -z '' ]]
[2026-06-30T02:04:03.924Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-06-30T02:04:03.924Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-06-30T02:04:03.924Z] ++ set -ex
[2026-06-30T02:04:03.924Z] ++ set -o pipefail
[2026-06-30T02:04:03.924Z] ++ copy_whls=0
[2026-06-30T02:04:03.924Z] ++ envvars=()
[2026-06-30T02:04:03.924Z] ++ [[ 13 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=-e
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e $2")
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 11 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=-e
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e $2")
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 9 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=-e
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e $2")
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 7 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=-e
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e $2")
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 5 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=-e
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e $2")
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 3 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=-e
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e $2")
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 1 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ key=--copy-whls
[2026-06-30T02:04:03.924Z] ++ case $key in
[2026-06-30T02:04:03.924Z] ++ copy_whls=1
[2026-06-30T02:04:03.924Z] ++ shift
[2026-06-30T02:04:03.924Z] ++ [[ 0 -gt 0 ]]
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-06-30T02:04:03.924Z] ++ [[ false == \t\r\u\e ]]
[2026-06-30T02:04:03.924Z] ++ [[ true == \t\r\u\e ]]
[2026-06-30T02:04:03.924Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
[2026-06-30T02:04:03.924Z] ++ docker ps -a
[2026-06-30T02:04:03.924Z] CONTAINER ID   IMAGE                                                                                                                        COMMAND                  CREATED         STATUS                   PORTS                                                 NAMES
[2026-06-30T02:04:03.924Z] 1b34995a1ed0   rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942   "bash -lc 'sleep inf…"   4 minutes ago   Up 4 minutes                                                                   pytorch-ci-container
[2026-06-30T02:04:03.924Z] 33af8c9e058f   dashboard-dashboard-ui                                                                                                       "/docker-entrypoint.…"   7 days ago      Up 5 hours               80/tcp, 0.0.0.0:3000->3000/tcp, [::]:3000->3000/tcp   rocagents-dashboard-ui
[2026-06-30T02:04:03.924Z] 7c6be83a2db1   dashboard-dashboard-api                                                                                                      "uvicorn main:app --…"   7 days ago      Up 5 hours (healthy)     0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp           rocagents-dashboard-api
[2026-06-30T02:04:03.924Z] 780dc8fe5a7b   rocm/eigen-test:ubuntu20.04-rocm5.3.0-eigen                                                                                  "/scripts/run_tests_…"   3 weeks ago     Exited (1) 3 weeks ago                                                         infallible_chatelet
[2026-06-30T02:04:03.924Z] 2d7786cfd066   deployment-rocagents                                                                                                         "python src/multi_re…"   6 weeks ago     Up 5 hours (healthy)                                                           rocagents-orchestrator
[2026-06-30T02:04:03.924Z] ++ docker rm -f pytorch-ci-container
[2026-06-30T02:04:04.444Z] ++ docker pull rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T02:04:04.970Z] pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942: Pulling from rocm/pytorch-ci-private
[2026-06-30T02:04:04.970Z] Digest: sha256:b836aa0e221ac85a2184ec24a67b9d5089f3caa45d182cf521dee7d38112aa81
[2026-06-30T02:04:04.970Z] Status: Image is up to date for rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T02:04:04.970Z] docker.io/rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942
[2026-06-30T02:04:04.970Z] ++ GPU_FLAGS=
[2026-06-30T02:04:04.970Z] ++ echo 'NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250'
[2026-06-30T02:04:04.970Z] NODE_LABELS: multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250
[2026-06-30T02:04:04.970Z] ++ [[ ! multi_gpu rocm dlmodels gpu_8 gfx90a ixt-hq-ubb4-31 mlir MI250 =~ build-only ]]
[2026-06-30T02:04:04.970Z] ++ GPU_FLAGS='--group-add video --device=/dev/kfd --device=/dev/dri'
[2026-06-30T02:04:04.970Z] ++++ dirname prepare_docker_env.sh
[2026-06-30T02:04:04.970Z] +++ cd .
[2026-06-30T02:04:04.970Z] +++ pwd
[2026-06-30T02:04:04.970Z] ++ script_dir=/home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379
[2026-06-30T02:04:04.970Z] ++ docker run --name pytorch-ci-container -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=distributed -e SHARD_NUMBER=2 -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip -e PREBUILT=false -e TEST_CORE=true -e TEST_CORE=true -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379:/host_workspace -v /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379:/scripts --entrypoint bash -d --network=host --group-add video --device=/dev/kfd --device=/dev/dri --ipc=host --pid=host --shm-size 8G --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rocm/pytorch-ci-private:pytorch-linux-noble-rocm7.2.4-py3.12-277a2c4d916901d0ff0bee19b47924aedca4f143-gfx908_gfx90a_gfx942 -lc 'sleep infinity'
[2026-06-30T02:04:04.970Z] 79b474b283f944d8530a6299a511db1c7254f6cb36fc0adc1ead2c269d7f10eb
[2026-06-30T02:04:04.970Z] ++ env -i python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T02:04:06.075Z] Exporting test times from test-infra
[2026-06-30T02:04:06.075Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T02:04:06.075Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T02:04:06.075Z] ++ python3 /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/tools/stats/export_test_times.py
[2026-06-30T02:04:06.075Z] Exporting test times from test-infra
[2026-06-30T02:04:06.075Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T02:04:06.075Z] Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/test-class-times.json to /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-class-times.json
[2026-06-30T02:04:06.075Z] ++ [[ -d /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files ]]
[2026-06-30T02:04:06.075Z] ++ sed -i -E 's/"\S+rocm\S+"/""/g' /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch/.additional_ci_files/test-times.json
[2026-06-30T02:04:06.075Z] ++ [[ false != \t\r\u\e ]]
[2026-06-30T02:04:06.075Z] ++ docker exec -u root pytorch-ci-container bash -c 'rm -rf /var/lib/jenkins/pytorch'
[2026-06-30T02:04:06.075Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/pytorch pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T02:04:11.567Z] ++ docker exec -u root pytorch-ci-container bash -c 'chown root:root -R /var/lib/jenkins/pytorch'
[2026-06-30T02:04:11.567Z] ++ [[ 1 -eq 1 ]]
[2026-06-30T02:04:11.567Z] ++ docker cp /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/rocm_composable_kernel-7.1.1.dev0+g4266f8-py3-none-any.whl /home/jenkins/internal/workspace/orch_pytorch-ci-pipeline_PR-3379/torch-2.14.0a0+git34b6825-cp312-cp312-linux_x86_64.whl pytorch-ci-container:/var/lib/jenkins/pytorch
[2026-06-30T02:04:11.567Z] docker: 'docker cp' requires 2 arguments
[2026-06-30T02:04:11.567Z] 
[2026-06-30T02:04:11.567Z] Usage:  docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
[2026-06-30T02:04:11.567Z] 	docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
[2026-06-30T02:04:11.567Z] 
[2026-06-30T02:04:11.567Z] Run 'docker cp --help' for more information

Tests / Test Distributed / Test Distributed / Run pytorch_distributed_2 / Error signal

Error in error step, with arguments pytorch_distributed_2 failed.

pytorch_distributed_2 failed

Tests / Test Inductor / Test Inductor / Run pytorch_inductor_1 / Shell Script

Error in sh step, with arguments ./test_pytorch_inductor.sh.

script returned exit code 1
Build log 
[2026-06-30T01:39:31.861Z] + ./test_pytorch_inductor.sh
[2026-06-30T01:39:31.861Z] + [[ -z '' ]]
[2026-06-30T01:39:31.861Z] + DEFAULT_EXECUTION_OPTIONS='-e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts         -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip         --copy-whls'
[2026-06-30T01:39:31.861Z] + source prepare_docker_env.sh -e 'EXTENSION_BUILD_GFXARCH=gfx90a;gfx908;gfx942' -e CI=1 -e TEST_CONFIG=inductor -e CUSTOM_TEST_ARTIFACT_BUILD_DIR=build/custom_test_artifacts -e CUSTOM_TEST_ARTIFACTS_FILE=test_artifacts.zip --copy-whls
[2026-06-30T01:39:31.861Z] ++ set -ex
[2026-06-30T01:39:31.861Z] ++ set -o pipefail
[2026-06-30T01:39:31.861Z] ++ copy_whls=0
[2026-06-30T01:39:31.861Z] ++ envvars=()
[2026-06-30T01:39:31.861Z] ++ [[ 11 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ key=-e
[2026-06-30T01:39:31.861Z] ++ case $key in
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e $2")
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ [[ 9 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ key=-e
[2026-06-30T01:39:31.861Z] ++ case $key in
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e $2")
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ [[ 7 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ key=-e
[2026-06-30T01:39:31.861Z] ++ case $key in
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e $2")
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ [[ 5 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ key=-e
[2026-06-30T01:39:31.861Z] ++ case $key in
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e $2")
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ [[ 3 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ key=-e
[2026-06-30T01:39:31.861Z] ++ case $key in
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e $2")
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ [[ 1 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ key=--copy-whls
[2026-06-30T01:39:31.861Z] ++ case $key in
[2026-06-30T01:39:31.861Z] ++ copy_whls=1
[2026-06-30T01:39:31.861Z] ++ shift
[2026-06-30T01:39:31.861Z] ++ [[ 0 -gt 0 ]]
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e PREBUILT=${PREBUILT:-false}")
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE:-false}")
[2026-06-30T01:39:31.861Z] ++ [[ false == \t\r\u\e ]]
[2026-06-30T01:39:31.861Z] ++ [[ true == \t\r\u\e ]]
[2026-06-30T01:39:31.861Z] ++ envvars+=("-e TEST_CORE=${TEST_CORE}")
Output truncated.

Details

  • Kill older PR Builds (1.8 sec)
  • Initialize (3 hr 55 min)
    • Download CI scripts (41 sec)
    • Checkout Pytorch (1 min 54 sec)
    • Check base Docker image existence (5.3 sec)
    • Build Docker image (1 hr 7 min)
    • Build PyTorch (2 hr 43 min)
  • Tests (57 min)
    • Test PyTorch (10 ms)
      • Test PyTorch (53 min)
        • Run pytorch_test_1 (32 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_test_1 failed - logs
        • Run pytorch_test_2 (21 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_test_2 failed - logs
    • Test Distributed (7 ms)
      • Test Distributed (57 min)
        • Run pytorch_distributed_1 (49 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_1 failed - logs
        • Run pytorch_distributed_2 (7 min 48 sec)
          Error: script returned exit code 1 - logs
          Error: pytorch_distributed_2 failed - logs
    • Test Inductor (7 ms)
      • Test Inductor (38 min)
        • Run pytorch_inductor_1 (38 min)
          Error: script returned exit code 1 - logs
          Error: pytorch_inductor_1 failed - logs
    • Test PyTorch Slow (10 ms)
      • Test PyTorch Slow (13 sec)
    • Microbenchmark (26 sec)
      • Microbenchmark (13 sec)
  • Post Build (1.6 sec)
  • Declarative: Post Actions (3.3 sec)
View more details on ROCm Repo Management API