Fix codeinsights ast_distance / asm_distance returning similarity not distance#4296
Open
Kymi808 wants to merge 1 commit into
Open
Fix codeinsights ast_distance / asm_distance returning similarity not distance#4296Kymi808 wants to merge 1 commit into
Kymi808 wants to merge 1 commit into
Conversation
… distance In codeinsights_code_evaluation_metrics, `calculate_ast_distance` and the assembly metric returned `Levenshtein.ratio(...)` directly. `Levenshtein.ratio` is a similarity in [0, 1] (1.0 = identical), but the function docstring, the public stat names `ast_distance` / `asm_distance`, and the error fallbacks (`return 1.0` for parse / compile failure as "max distance") all assume distance semantics (0 = identical, 1 = completely different). Effect: a perfect AST or ASM match was reported as 1.0, identical to a parse failure, so the metric could not distinguish a correct generation from a broken one and the ordering was reversed (higher = better-looking). Convert similarity to distance via `1.0 - levenshtein_distance_ratio(...)` so the value matches the docstring, the stat name, and the error-fallback convention. Adds regression tests for both directions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
codeinsights_code_evaluation_metrics.pyimportsthen in
ASTAnalyzer.calculate_ast_distance(and the assembly path inCodeInsightsCodeEvaluationMetric.evaluate_generation) returns it directly:Levenshtein.ratiois a similarity in[0, 1](1.0 = identical, 0.0 = completely different), but everything around the call site assumes distance semantics:calculate_ast_distanceMetricName("ast_distance")/"asm_distance"(contrasted with the sibling"codebert_similarity"in the same class)return 1.0for parse failure,asm_distance = 1.0for compile failure, comment "max distance for errors")So a perfect AST or ASM match was reported as 1.0, indistinguishable from a parse / compile failure, and the ordering was inverted (higher score looked better than lower).
Fix
Convert similarity to distance at the two call sites:
The legacy batch helper
evaluate_ast_distances_batchcallsanalyzer.calculate_ast_distance(...)and so picks up the fix transitively.Why this isn't a behavior change someone would be relying on
ast_distance/asm_distance/normalized_distance— no presentation/schema, nohigher_is_better/lower_is_betterannotation, no test.Test plan
Added
src/helm/benchmark/metrics/test_codeinsights_code_evaluation_metrics.py. Usespytest.importorskipfor the optionalcodeinsightsextras (clang.cindex,Levenshtein) and stubsclang.cindex.Index.create+_extract_ast_featuresso the tests don't need a real libclang install:0.0(matches docstring)1.0(0, 1)The first two fail on
main(return1.0and0.0respectively) and pass with this change.pytest -o addopts=""on the new file → 3 passed;black --checkclean on both files.