Changes related to running benchmark experiments for the paper: Support Qwen3.5 and thinking models, Skywork, truncation tracking, benchmark changes etc#32

Open

ErlisLushtaku wants to merge 30 commits into

mainfrom

erlislushtaku/fix/support-qwen-3.5

Commits on Apr 7, 2026

Commits on Apr 13, 2026

use latest vllm with thinking tokens limits and thinking field in the output
ErlisLushtaku
committed

Commits on Apr 14, 2026

Merge remote-tracking branch 'origin/main' into erlislushtaku/fix/support-qwen-3.5
ErlisLushtaku
committed
thinking token handling improvements, mt-bench improvements, use mt-bench baseline from huggingface and update huggingface repo
ErlisLushtaku
committed

Commits on Apr 15, 2026

Revert to free form generation, and use thinking token budget with regex stripping since the structured output wasn't working for isolating thinking tokens anyway
ErlisLushtaku
committed

Commits on Apr 20, 2026

Commits on Apr 22, 2026

update system prompt with alpaca eval version, fix mismatch for expected token count for max_model_len
ErlisLushtaku
committed

Commits on Apr 28, 2026

Commits on Apr 29, 2026