Skip to content

fix(registry): merge duplicate model deployments instead of shadowing (#3709)#4316

Open
nac7 wants to merge 2 commits into
stanford-crfm:mainfrom
nac7:fix/prod-env-deployment-override
Open

fix(registry): merge duplicate model deployments instead of shadowing (#3709)#4316
nac7 wants to merge 2 commits into
stanford-crfm:mainfrom
nac7:fix/prod-env-deployment-override

Conversation

@nac7

@nac7 nac7 commented Jun 9, 2026

Copy link
Copy Markdown

What

PR #3694 changed the deployment resolution order so that prod_env/model_deployments.yaml
entries are registered after the built-in helm.config entries, and
get_default_model_deployment_for_model picks [-1] (last registered) as
the default.

As a result, a prod_env entry that is incomplete — e.g. a user who
only wanted to change client_spec but left tokenizer_name unset —
silently shadows the full built-in entry. The WindowService then
receives tokenizer_name=None and raises:

ValueError: tokenizer_name is None

The confusing part: the user's config file is correct and a standalone
diagnostic script proves it loads fine, but helm-run still fails because
the prod_env entry wins.

Fix

src/helm/benchmark/model_deployment_registry.pyregister_model_deployment

When a deployment name is already present in DEPLOYMENT_NAME_TO_MODEL_DEPLOYMENT,
merge the new entry over the existing one rather than blindly appending:

  • The new entry's non-None fields override the existing values (preserving the intended prod_env-wins behaviour from Change logic for default model deployments #3694).
  • Fields that are None in the new entry fall back to the existing value, so a sparse prod_env entry cannot accidentally wipe tokenizer_name (or max_sequence_length, etc.) that was set in the built-in config.
  • name, client_spec, and deprecated are always taken from the new entry.
  • A hwarn is emitted on every merge so that users can see which sources are competing.
# Before: blindly appended — prod_env None fields silently win
DEPLOYMENT_NAME_TO_MODEL_DEPLOYMENT[model_deployment.name] = model_deployment
ALL_MODEL_DEPLOYMENTS.append(model_deployment)

# After: merge non-None fields; None fields fall back to the existing entry
if existing := DEPLOYMENT_NAME_TO_MODEL_DEPLOYMENT.get(model_deployment.name):
    override_fields = {f.name: v for f in dataclasses.fields(model_deployment)
                       if (v := getattr(model_deployment, f.name)) is not None}
    override_fields |= {"name": ..., "client_spec": ..., "deprecated": ...}
    model_deployment = dataclasses.replace(existing, **override_fields)

Tests

src/helm/benchmark/test_model_deployment_registry.py (new file, 6 tests):

Test Checks
test_first_registration_stored_as_is Baseline: first entry stored unchanged
test_override_with_full_entry_replaces_all_fields Full prod_env entry overrides everything
test_sparse_override_inherits_none_fields_from_builtin Regression #3709: tokenizer_name=None in prod_env falls back to built-in value
test_sparse_override_none_max_sequence_inherits_from_builtin max_sequence_length=None in prod_env falls back to built-in
test_deprecated_always_taken_from_new_entry deprecated bool always uses new entry
test_only_one_entry_in_lookup_dict_after_override Dict holds exactly one entry per name

Fixes #3709

nac7 added 2 commits June 9, 2026 16:25
normalize_text removes English articles ("a", "an", "the") before
computing F1.  When the gold or pred string consists entirely of such
a word -- for example, the letter "A" in MMLU-style multiple-choice
benchmarks -- article removal empties the token set.  f_measure on an
empty set returns None, which f1_score silently converts to 0.0, so a
correct prediction scores 0 instead of 1.

Fix: after normalization, if either token set is empty, fall back to
normalization with should_remove_articles=False before calling
f_measure.  Multi-word strings that contain articles alongside other
tokens are unaffected.

Adds test_evaluate_reference_metrics.py coverage for f1_score:
  - test_f1_score_normal         (baseline behavior unchanged)
  - test_f1_score_article_gold   (regression: "A"/"An"/"The" as gold)
  - test_f1_score_article_in_longer_string (article removal still fires
                                            for multi-word strings)
  - test_f1_score_empty_pred     (empty prediction still yields 0)

Fixes stanford-crfm#2298
PR stanford-crfm#3694 changed the deployment resolution order so that prod_env entries
are registered after built-in (helm.config) entries and therefore win the
[-1] pick in get_default_model_deployment_for_model.

This means a prod_env/model_deployments.yaml entry that is incomplete
(e.g. tokenizer_name omitted because the user only wanted to change
client_spec) silently shadows the full built-in entry.  The
WindowService then receives tokenizer_name=None and raises ValueError.

Fix: when register_model_deployment encounters a name that is already in
DEPLOYMENT_NAME_TO_MODEL_DEPLOYMENT, build a merged ModelDeployment:
- the new entry's non-None field values override the existing ones
- None fields in the new entry fall back to the existing entry's values
- name, client_spec, and deprecated are always taken from the new entry

A hwarn is emitted whenever a merge occurs so that users can see which
sources are competing.

Adds test_model_deployment_registry.py with six tests covering:
  - first registration stored as-is
  - full override replaces all fields
  - sparse override inherits None fields from built-in (regression stanford-crfm#3709)
  - sparse override inherits max_sequence_length from built-in
  - deprecated flag always taken from the new entry
  - dict holds exactly one entry after override

Fixes stanford-crfm#3709
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: ValueError: tokenizer_name is None when adding a new Hugging Face model due to prod_env behavior

1 participant