`additional_special_tokens` is silently discarded when `tokenizer_config.json` also contains `extra_special_tokens`

[wagnerpatriota]

System Info

• transformers version: 5.14.1
• Platform: Linux-6.12.58-82.121.amzn2023.x86_64-x86_64-with-glibc2.39
• Python version: 3.12.11
• Huggingface_hub version: 1.26.0
• Safetensors version: 0.8.0
• Accelerate version: 1.14.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.11.0+cu130 (NA)
• Using distributed or parallel set-up in script?: N/A

Who can help?

Two v5 back-compat shims convert the deprecated additional_special_tokens kwarg into extra_special_tokens using this expression:

init_kwargs.setdefault("extra_special_tokens", init_kwargs.pop("additional_special_tokens"))

File: src/transformers/tokenization_utils_base.py — lines 1813 and 1169 (on main @ 2026-08-07)

Python evaluates a call's arguments before the call, so .pop(...) always runs. If extra_special_tokens is already present — which is the case for any tokenizer_config.json that carries both keys — setdefault then declines to overwrite it and the popped list is thrown away. The tokenizer ends up receiving neither key.

The same file already contains the correct guarded form of this conversion at line 997 (PreTrainedTokenizerBase.__init__):

if "additional_special_tokens" in kwargs and "extra_special_tokens" not in kwargs:
    kwargs["extra_special_tokens"] = kwargs.pop("additional_special_tokens")

so the three conversion sites are inconsistent with each other. Two affected sites:

Line	Enclosing method	Guard	Impact
1813	PreTrainedTokenizerBase._from_pretrained	❌ eager-pop setdefault	every from_pretrained load
1169	PreTrainedTokenizerBase.add_special_tokens	❌ eager-pop setdefault	the public add_special_tokens() API
997	PreTrainedTokenizerBase.__init__	✅ explicit not in	correct

Introduced in b76caaa88d9d ("simplify extra tokens logic in base", #43230, merged 2026-01-29), which replaced the explicit ... and "extra_special_tokens" not in init_kwargs guard with setdefault. Present in v5.1.0 through current main; absent in v5.0.0.

Two distinct user-visible consequences, both observed on real checkpoints:

1. Hard failure for tokenizers whose __init__ branches on additional_special_tokens is None — common in trust_remote_code tokenizers. They take their "no special tokens supplied, infer them" path on an ordinary load and can raise.
2. Silent correctness bug for everyone else: tokens the checkpoint explicitly declares as special are no longer special. len(tokenizer) and encode() of ordinary text are unchanged, so this passes unnoticed — only all_special_tokens / all_special_ids shrink.

Extra Environment Info

• transformers: reproduced on 5.14.1; line present in v5.1.0 … main, absent in v5.0.0
• tiktoken: 0.13.0 / tokenizers: 0.22.2

Context

• Regression?: YES — behavior changed in b76caaa88d9d (simplify extra tokens logic in base #43230, v5.1.0). The pre-simplify extra tokens logic in base #43230 code was correct.
• Affected: any checkpoint whose tokenizer_config.json has a non-empty additional_special_tokens and an extra_special_tokens key. In a local corpus of 185 cached checkpoints, 40 match — e.g. allenai/Molmo2-8B (277 tokens), bharatgenai/Param2-17B (8007), baidu/ERNIE-4.5-* (1013), mistralai/Mistral-Large-3-675B / Mistral-Small-3.1-24B (1000), tiiuae/Falcon-H1-34B (843), skt/A.X-K1 (143), stepfun-ai/Step3-VL-10B (36), zai-org/GLM-4.5 / GLM-4.5V / GLM-4.7-Flash, internlm/internlm3-8b-instruct, internlm/Intern-S1-*, mispeech/midashenglm-7b, moonshotai/Kimi-VL-A3B-Instruct / Kimi-K2.5 / Kimi-Linear-48B, xlangai/OpenCUA-7B.
• Severity: major — one class of tokenizer fails to load at all; the rest lose declared special tokens silently, which is the more dangerous outcome (no error, no warning, and ordinary-text tokenization is byte-identical, so tests that only check encode() still pass).
• Related: simplify extra tokens logic in base #43230 (the introducing PR). Also in this area but distinct — all about list-vs-dict shapes in extra_special_tokens, not this data loss: Fix _set_model_specific_special_tokens to accept list-format extra_special_tokens #44781, Accept legacy list special_tokens in _set_model_specific_special_tokens #47372 (both closed, unmerged), and special_tokens, 'list' object has no attribute 'keys' #47110.
• Why the existing v4→v5 tokenizer audit did not catch this: [vllm + v5 fix] handle TokenizersBackend fallback properly for v5 #44255 ("handle TokenizersBackend fallback properly for v5", merged 2026-03-04) audited 22 model converters and explicitly tested xlangai/OpenCUA-7B, reporting Samples compared: 32000 / Curr roundtrip OK: 32000/32000 (100.0%) / Changed samples: 0. That audit compares roundtrip_ok = (decoded == text) and the tokenizer's JSON serialization; it never inspects all_special_tokens / all_special_ids, which is the only thing this bug changes. It also ran with use_fast=True, resolving to TokenizersBackend rather than the repo's trust_remote_code class, so the hard-failure path was never exercised either. So OpenCUA passed that audit while being entirely unloadable via its own tokenizer class — the methodology, not the model, is why this stayed hidden. Downstream reports of the OpenCUA symptom, neither identifying this root cause: xlangai/OpenCUA-7B discussion #10, #11.

Suggested Fix

Restore the explicit guard at both buggy sites, so the value is only popped once it has a destination. Minimal and behavior-preserving whenever extra_special_tokens is absent:

--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ line ~1169, PreTrainedTokenizerBase.add_special_tokens
         if "additional_special_tokens" in special_tokens_dict:
-            special_tokens_dict.setdefault(
-                "extra_special_tokens", special_tokens_dict.pop("additional_special_tokens")
-            )
+            if "extra_special_tokens" not in special_tokens_dict:
+                special_tokens_dict["extra_special_tokens"] = special_tokens_dict.pop(
+                    "additional_special_tokens"
+                )
@@ line ~1813, PreTrainedTokenizerBase._from_pretrained
         # V5: Convert deprecated additional_special_tokens to extra_special_tokens
-        if "additional_special_tokens" in init_kwargs:
-            init_kwargs.setdefault("extra_special_tokens", init_kwargs.pop("additional_special_tokens"))
+        if "additional_special_tokens" in init_kwargs and "extra_special_tokens" not in init_kwargs:
+            init_kwargs["extra_special_tokens"] = init_kwargs.pop("additional_special_tokens")

This matches the already-correct site at line 997 and makes all three consistent.

Worth deciding separately (a policy question, not part of the data-loss fix): when both keys are present and non-empty, silently preferring one is still lossy — merging them, or emitting a warning that the deprecated key is being ignored, would be friendlier than dropping data.

Two notes for whoever picks this up:

• A fix that only stops the data loss but still strips the kwarg leaves consequence (1) intact for trust_remote_code tokenizers that declare additional_special_tokens in their own __init__ signature and branch on it. Consider passing the kwarg through when "additional_special_tokens" in inspect.signature(cls.__init__).parameters, or deprecating more loudly so those repos can migrate.
• A regression test should assert on all_special_tokens / all_special_ids, not on len(tokenizer) or encode() — those are unaffected by this bug, which is exactly why it went unnoticed for ~6 months. This is not hypothetical: the audit in [vllm + v5 fix] handle TokenizersBackend fallback properly for v5 #44255 checked exactly those unaffected properties on this very checkpoint and reported 0 changed samples. Extending that comparison to the special-token sets would have caught it, and would guard the whole 22-model matrix against this class of regression.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

No transformers, torch, model download, or hardware required — this is stdlib only:

d = {"additional_special_tokens": ["<|im_end|>", "<|media_placeholder|>"],
     "extra_special_tokens": {}}

d.setdefault("extra_special_tokens", d.pop("additional_special_tokens"))   # line 1813 verbatim

print(d)   # {'extra_special_tokens': {}}   <-- both special tokens are GONE

End-to-end with a real checkpoint (transformers>=5.1.0):

from transformers import AutoTokenizer

# tokenizer_config.json declares 9 additional_special_tokens AND extra_special_tokens: {}
tk = AutoTokenizer.from_pretrained("moonshotai/Kimi-VL-A3B-Instruct", trust_remote_code=True)
print(len(tk.all_special_ids))   # 9  — expected 13
for t in ("<|media_start|>", "<|media_content|>", "<|media_end|>", "<|media_pad|>"):
    print(t, t in tk.all_special_tokens)   # all False, though all 4 are declared in the config

Hard-failure case (same root cause, raises instead of degrading silently):

# tokenizer_config.json declares 20 additional_special_tokens AND extra_special_tokens: {}
AutoTokenizer.from_pretrained("xlangai/OpenCUA-7B", trust_remote_code=True)
# ValueError: unk_token should not be set in dumping mode when additional_special_tokens is None

Its remote-code TikTokenV3.__init__ guards a branch with if additional_special_tokens is None:; with the kwarg stripped, that branch runs on a normal load while unk_token/pad_token still arrive from the config, and it raises.

Stack Trace

File ".../transformers/models/auto/tokenization_auto.py", line 891, in from_pretrained
    return tokenizer_class.from_pretrained(
File ".../transformers/tokenization_utils_base.py", line 1747, in from_pretrained
    return cls._from_pretrained(
File ".../transformers/tokenization_utils_base.py", line 1943, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
File ".../transformers_modules/<hash>/tokenization_opencua.py", line 118, in __init__
    raise ValueError("unk_token should not be set in dumping mode when additional_special_tokens is None")
ValueError: unk_token should not be set in dumping mode when additional_special_tokens is None

(Line numbers are from the installed v5.14.1; the setdefault is at :1823-1824 there and :1813 on main.)

Expected behavior

A checkpoint that declares additional_special_tokens should have those tokens honored regardless of whether extra_special_tokens is also present in its config. At minimum, the conversion must never destroy the value it fails to migrate.

[ifrahgul]

Hi @wagnerpatriota, I’d like to work on this issue as a first contribution to Transformers. I’ve gone through the reproduction and the proposed fix, and I’d like to implement the guarded conversion at both affected sites and add a regression test covering "all_special_tokens" / "all_special_ids".

I’ll first verify the existing test structure and contributing guidelines, then submit a PR with the fix and regression coverage.

[Rocketknight1]

cc @itazap (looks like there are some agent PRs too)