[MIGraphX EP] Wrong Resize results with the MIGraphX execution provider

[DanBmh]

Describe the issue

A Resize node (2x nearest neighbour) returns wrong results when the model is executed with the
MIGraphX execution provider, but only if the model takes its input as NHWC and transposes it to
NCHW at the beginning, which is the usual layout for exported image models.

The returned tensor contains exactly the correct set of values, sorted they are identical to the
CPU result, but the values sit at wrong positions. That looks like the resize kernel reads its
input with the wrong strides once the tensor is kept in a channels last layout.

The very same network, only fed with an NCHW input instead, returns correct results. Standalone
MIGraphX also handles the model correctly, migraphx-driver verify reports no error for it, so
the problem seems to be in the execution provider rather than in MIGraphX itself.

The effect is not a rounding issue. In our real model, an RTMDet detector, it lowers the object
scores so far that most objects fall below the score threshold and are never detected.

To reproduce

Use the attached files and run:

compare.py
make_model.py
run.sh
run_model.cpp

export ORT_DIR=/onnxruntime          # onnxruntime with the MIGraphX provider
./run.sh

run.sh builds two small models with make_model.py, runs each of them once with the CPU
provider and once with the MIGraphX provider, and compares the outputs. Both models contain the
same network, two strided convolutions followed by the Resize. They differ only in the layout
of the graph input:

• model_nhwc.onnx, input [1, 40, 40, 3], transposed to NCHW inside the model
• model_nchw.onnx, input [1, 3, 40, 40], used directly

Every run uses its own process, because creating a CPU and a MIGraphX session in the same process
corrupts the heap while tearing them down, which is an unrelated problem.

Expected behavior

Both models return the same results with both providers, within fp16 rounding.

Actual behavior

NHWC input (transposed to NCHW inside the model):
  max |cpu|                 4.69278
  max |cpu - migraphx|      7.06597
  mean|cpu - migraphx|      1.43615
  relative error            1.50571
  max diff of the sorted values      0.00245
  -> MISMATCH

NCHW input (same network, reference):
  max |cpu|                 2.92963
  max |cpu - migraphx|      0.00261
  mean|cpu - migraphx|      0.00049
  relative error            0.00089
  max diff of the sorted values      0.00261
  -> ok

The line max diff of the sorted values compares both outputs after sorting them. It stays at
rounding level for the failing case as well, so MIGraphX computes all the correct values and only
places them wrong.

Further observations

Collected while narrowing this down on a real RTMDet detector (Link):

• The graph optimization level does not matter, ORT_DISABLE_ALL, ORT_ENABLE_BASIC and
  ORT_ENABLE_EXTENDED return bit identical wrong results.

• migraphx_fp16_enable does not matter, and neither does the precision of the model. A model
  converted to fp32 fails in exactly the same way.

• migraphx-driver verify passes for the failing models, both per instruction and for the whole
  program.

• It also passes for the subgraph which the provider itself hands over to MIGraphX. Dumping it
  with ORT_MIGRAPHX_DUMP_MODEL_OPS=1 and verifying that file:

  $ ORT_MIGRAPHX_DUMP_MODEL_OPS=1 ./run_model model_nhwc.onnx migraphx out.bin
  $ migraphx-driver verify --onnx MIGraphXExecutionProvider_MGXKernel_graph_model_nhwc_*.onnx
  MIGraphX verification passed successfully.

  So MIGraphX computes exactly the graph it receives from the provider correctly, while the same
  graph returns wrong values when it is run through onnxruntime.

• Feeding the resize input as a graph input, instead of computing it inside the model, makes the
  problem disappear. Cutting the real model directly in front of the resize returns correct
  results, only the complete model starting at the NHWC input fails.

• Replacing the Resize by a depthwise ConvTranspose with a 2x2 kernel of ones and stride 2,
  which is the same nearest neighbour upsampling, returns correct results. We use that as a
  workaround.

Unrelated problems noticed on the way

Small and separate from the above, each easy to trigger:

1. The provider aborts with write_buffer: Failure opening file: ""/<hash>.mxr when
   ORT_MIGRAPHX_MODEL_CACHE_PATH is not set. It should either default to a usable path or skip
   the caching.
2. Models with several outputs can fail with
   ToInteger(std::string_view) invalid input for conversion to integer, thrown from
   compute_output_index in migraphx_execution_provider.cc, which parses the #output_N
   parameter names of the compiled program.
3. Creating a CPU session and a MIGraphX session in the same process regularly ends in
   corrupted double-linked list while the sessions are destroyed.

Urgency

Not urgent for us, the workaround above is stable.

Platform

Linux

OS Version

Ubuntu 24.04, container based on rocm/dev-ubuntu-24.04:7.2.4-complete

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

Built from source, tag v1.23.2: ./build.sh --allow_running_as_root --build_shared_lib --config Release --skip_tests \ --use_rocm --rocm_version=7.2.4 --rocm_home /opt/rocm --use_migraphx

ONNX Runtime API

C++

Architecture

X64

Execution Provider

MIGraphX

Execution Provider Library Version

• ROCm 7.2.4 * MIGraphX 2.15.0.20250912-17-221-gdd11a7555 * AMD Radeon RX 6600 XT (gfx1032, run with HSA_OVERRIDE_GFX_VERSION=10.3.0)

[liminfei-amd]

This looks like ROCm/AMDMIGraphX#4713. Could you try an ONNX Runtime build using a MIGraphX revision that includes that change?

It fixed the NHWC/NCHW Resize layout mismatch in my reproduction.

[DanBmh]

Yes, that's the same bug, and it solves my problem. Thanks for your link.

I didn't rebuild against a newer MIGraphX revision — instead I applied the equivalent change to the MIGraphX 2.15.0 I'm on (ROCm 7.2.4) and rebuilt just libmigraphx_onnx.so, keeping onnxruntime unchanged. That fixes both my minimal reproduction (relative error 1.51 → 0.0007) and the real RTMDet detector (highest score 0.209 → 0.721 against 0.721 on CPU). Since I already have a workaround for my current ROCm 7.2.4 and the fix is included in the newer ROCm 7.14 I'm fine with that.