`eval_kernel_against_ref`: model-load phase catches only `RuntimeError`

[MattFisher]

Summary

eval_kernel_against_ref is careful to encode failures of the generated kernel in its return value rather than raising: the compilation phase catches Exception and returns KernelExecResult(compiled=False, ...), and the correctness phase catches Exception and returns KernelExecResult(compiled=True, correctness=False, ...).

The model-load phase between them, however, catches only RuntimeError:

https://github.com/ScalingIntelligence/KernelBench/blob/423217d9fda9/src/kernelbench/eval.py#L559-L577

    try:
        with torch.no_grad():
            set_seed(seed_num)
            custom_model = ModelNew(*init_inputs)
            assert hasattr(custom_model, "forward")
            original_model = original_model.to(device=device, dtype=precision)
            custom_model = custom_model.to(device=device, dtype=precision)
            torch.cuda.synchronize(device=device)
        ...
    except RuntimeError as e:
        ...
        return KernelExecResult(
            compiled=True, correctness=False, metadata=metadata
        )  # skip further steps

ModelNew(*init_inputs) executes arbitrary generated __init__ code. RuntimeError covers the anticipated CUDA cases (launch failure, illegal memory access), but if the generated code raises anything else — ValueError, TypeError, KeyError, AssertionError (including the assert hasattr(...) in the try body itself), etc. — the exception propagates out of eval_kernel_against_ref instead of being recorded.

Why it matters

Callers that follow the function's own convention — "a verdict on the kernel comes back in the KernelExecResult; an exception means the harness itself failed" — misclassify these samples. A kernel whose __init__ deterministically raises ValueError is a failure of the generated code and should grade as compiled=True, correctness=False, exactly like a RuntimeError from the same site. Instead the caller sees an unhandled exception, indistinguishable from an environment failure (e.g. CUDA/device setup problems), and may retry or error out a sample that would fail identically every time.

For comparison, the sibling phases already treat any exception from the generated code as a graded outcome:

• compilation: except Exception → compiled=False (https://github.com/ScalingIntelligence/KernelBench/blob/423217d9fda9/src/kernelbench/eval.py#L524)
• correctness: except Exception → compiled=True, correctness=False (https://github.com/ScalingIntelligence/KernelBench/blob/423217d9fda9/src/kernelbench/eval.py#L599)

The model-load phase is the only gap.

Proposed fix

Broaden the guard to match the neighbouring phases:

    except Exception as e:
        print(
            f"Failed to load custom CUDA kernel; Compiled but not able to run, count as runtime error. \nError: {e}"
        )
        graceful_eval_cleanup(context, device, tempfile)
        metadata["runtime_error"] = e
        metadata["runtime_error_name"] = get_error_name(e)
        return KernelExecResult(
            compiled=True, correctness=False, metadata=metadata
        )  # skip further steps

i.e. a one-line change from except RuntimeError as e: to except Exception as e:. metadata["runtime_error_name"] (via get_error_name) already preserves the exception class, so RuntimeError vs. other exception types remains distinguishable in the result metadata.

One consideration: this block also wraps an operation on the reference model (original_model.to(...)), and failures there arguably aren't the generated kernel's fault. That pre-exists the change (.to() failures typically raise RuntimeError, which is already caught today), but if you'd prefer, the reference-model .to() call could be hoisted out of the try so the broadened catch covers only the generated kernel's code.