Skip to content

Fix codeinsights ast_distance / asm_distance returning similarity not distance#4296

Open
Kymi808 wants to merge 1 commit into
stanford-crfm:mainfrom
Kymi808:fix/codeinsights-ast-asm-distance-inversion
Open

Fix codeinsights ast_distance / asm_distance returning similarity not distance#4296
Kymi808 wants to merge 1 commit into
stanford-crfm:mainfrom
Kymi808:fix/codeinsights-ast-asm-distance-inversion

Conversation

@Kymi808

@Kymi808 Kymi808 commented May 30, 2026

Copy link
Copy Markdown

Summary

codeinsights_code_evaluation_metrics.py imports

from Levenshtein import ratio as levenshtein_distance_ratio

then in ASTAnalyzer.calculate_ast_distance (and the assembly path in CodeInsightsCodeEvaluationMetric.evaluate_generation) returns it directly:

return levenshtein_distance_ratio(nodes1, nodes2)   # AST
asm_distance = levenshtein_distance_ratio(gen_asm, truth_asm)

Levenshtein.ratio is a similarity in [0, 1] (1.0 = identical, 0.0 = completely different), but everything around the call site assumes distance semantics:

signal says
function name calculate_ast_distance distance
docstring "a float in [0,1], where 0 means identical ASTs and 1 means completely different (or a parse failure)"
stat names MetricName("ast_distance") / "asm_distance" (contrasted with the sibling "codebert_similarity" in the same class) distance
error fallbacks (return 1.0 for parse failure, asm_distance = 1.0 for compile failure, comment "max distance for errors") distance

So a perfect AST or ASM match was reported as 1.0, indistinguishable from a parse / compile failure, and the ordering was inverted (higher score looked better than lower).

Fix

Convert similarity to distance at the two call sites:

return 1.0 - levenshtein_distance_ratio(nodes1, nodes2)
asm_distance = 1.0 - levenshtein_distance_ratio(gen_asm, truth_asm)

The legacy batch helper evaluate_ast_distances_batch calls analyzer.calculate_ast_distance(...) and so picks up the fix transitively.

Why this isn't a behavior change someone would be relying on

  • No other file in the repo references ast_distance / asm_distance / normalized_distance — no presentation/schema, no higher_is_better/lower_is_better annotation, no test.
  • The error fallbacks already use 1.0 as "max distance", so any consumer that read those error values was already assuming distance semantics.

Test plan

Added src/helm/benchmark/metrics/test_codeinsights_code_evaluation_metrics.py. Uses pytest.importorskip for the optional codeinsights extras (clang.cindex, Levenshtein) and stubs clang.cindex.Index.create + _extract_ast_features so the tests don't need a real libclang install:

  • identical inputs → 0.0 (matches docstring)
  • totally different inputs → 1.0
  • partial overlap → small positive in (0, 1)

The first two fail on main (return 1.0 and 0.0 respectively) and pass with this change. pytest -o addopts="" on the new file → 3 passed; black --check clean on both files.

… distance

In codeinsights_code_evaluation_metrics, `calculate_ast_distance` and the
assembly metric returned `Levenshtein.ratio(...)` directly. `Levenshtein.ratio`
is a similarity in [0, 1] (1.0 = identical), but the function docstring, the
public stat names `ast_distance` / `asm_distance`, and the error fallbacks
(`return 1.0` for parse / compile failure as "max distance") all assume
distance semantics (0 = identical, 1 = completely different).

Effect: a perfect AST or ASM match was reported as 1.0, identical to a parse
failure, so the metric could not distinguish a correct generation from a
broken one and the ordering was reversed (higher = better-looking).

Convert similarity to distance via `1.0 - levenshtein_distance_ratio(...)` so
the value matches the docstring, the stat name, and the error-fallback
convention. Adds regression tests for both directions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant