[MIGraphX EP] Fails processing models with multiple layers of sub-graphs with "Unknown operator: MGXKernel_subgraph_tf2onnx_9255412890226604894_0"

[sdroege]

Describe the issue

This is a new issue on top of #31607 and the fix for that one (CC liminfei-amd (@liminfei-amd)).

Same model, available from https://gitlab.collabora.com/gstreamer/onnx-models/-/blob/a8de628ad5a745d3af6ba8c0ee84321605de07e9/models/ssd_mobilenet_v1_coco.onnx

Loading this model works fine after the fix for the other issue but doing actual inference fails:

./migraphx_sample migraphx onnx-models/models/ssd_mobilenet_v1_coco.onnx
Loading: onnx-models/models/ssd_mobilenet_v1_coco.onnx (provider: migraphx, device: 0)
2026-08-07 10:17:38.429816144 [W:onnxruntime:sample, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:
Session created successfully.
Input 0: image_tensor:0 shape=[1, 300, 300, 3]
Running inference...
2026-08-07 10:17:38.715384 [WARN] [AMDMIGraphX/src/onnx/parse_loop.cpp:66] PARSE_LOOP max_iterations exceeds the maximum loop iterations limit, it will be changed from 9223372036854775807 to 65535.
Warning: onnx model is not topologically sorted. Attempting to sort...
migraphx_parse_onnx_buffer: Error: AMDMIGraphX/src/onnx/onnx_parser.cpp:681: parse_graph: Unknown operator: MGXKernel_subgraph_tf2onnx_9255412890226604894_0
2026-08-07 10:17:38.717874898 [E:onnxruntime:, sequential_executor.cc:671 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_tf2onnx_9255412890226604894_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_tf2onnx_9255412890226604894_5_5' Status Message: Failed to call function
terminate called after throwing an instance of 'Ort::Exception'
  what():  Non-zero status code returned while running MGXKernel_graph_tf2onnx_9255412890226604894_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_tf2onnx_9255412890226604894_5_5' Status Message: Failed to call function
fish: Job 1, './migraphx_sample migraphx /hom…' terminated by signal SIGABRT (Abort)

AFAIU the problem here is that the execution provider is partitioning the model recursively and modifies them along the way, which then causes failures because it's working bottom-up and the upper graphs see modified sub-graphs that they can't handle anymore.

With the CPU execution provider the above works.

To reproduce

Extended minimal sample from the previous issue: migraphx_sample.cc.txt

Compile and run with the above command and it will fail.

Urgency

No response

Platform

Linux

OS Version

Fedora 44

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.28.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

MIGraphX

Execution Provider Library Version

ROCm 7.15.0 (20260803-30774164023) / MIGraphX rocm-7.14 tag

[Saad-Mallebhari]

Hi Sebastian Dröge (@sdroege)

Looking into migraphx_execution_provider.cc, when processing nested sub-graphs bottom-up (like Loop/If in ssd_mobilenet_v1_coco.onnx), the EP mutates child sub-graph nodes by replacing them with internal MGXKernel_subgraph_* operators. When the outer parent graph subsequently serializes its GraphProto sub-graph attributes into an ONNX memory buffer for migraphx_parse_onnx_buffer(), MIGraphX's parser encounters MGXKernel_subgraph_* instead of standard ONNX ops, throwing AMDMIGraphX/src/onnx/onnx_parser.cpp:681: parse_graph: Unknown operator: MGXKernel_subgraph_*.

We can fix this by ensuring sub-graph nodes retain their standard ONNX schemas during parent GraphProto buffer serialization prior to passing the buffer to migraphx_parse_onnx_buffer().

Would you like me to open a Pull Request with this fix?

[sdroege]

Sure that would be great! Thanks for looking into this so fast :)

[Saad-Mallebhari]

Hi Sebastian Dröge (Sebastian Dröge (@sdroege))

PR #31721 has been submitted to fix the MIGraphX nested control-flow sub-graph operator serialization failure:
#31721

Summary of changes:

1. In IsNodeSupported(), nodes under kMSDomain or starting with MGXKernel_ are rejected, preventing internal ORT fused kernel names from being sent to migraphx::parse_onnx_buffer().
2. In IsUnsupportedOpMode(), sub-graph block attributes (node->GetAttributeWithBlocks()) are inspected for If and Loop nodes. If an inner sub-graph already contains an MGXKernel_ node from a prior sub-graph pass, the parent If/Loop node is marked as unsupported for monolithic parent fusion.

Feel free to test PR #31721 against your ssd_mobilenet_v1_coco.onnx model pipeline!

[sdroege]

Thanks, I've commented on the PR!

[sdroege]

I've done my own version of this now at #31978

[sdroege]

Model at https://huggingface.co/onnxmodelzoo/ssd_mobilenet_v1_12 is equally affected