Skip to content

feat: add blimp_it italian grammar evaluation tasks - #3917

Open
cristianochesi wants to merge 2 commits into
EleutherAI:mainfrom
cristianochesi:add-blimp-it-task
Open

feat: add blimp_it italian grammar evaluation tasks#3917
cristianochesi wants to merge 2 commits into
EleutherAI:mainfrom
cristianochesi:add-blimp-it-task

Conversation

@cristianochesi

Copy link
Copy Markdown

No description provided.

@CLAassistant

CLAassistant commented Jul 7, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cristianochesi

Copy link
Copy Markdown
Author

any chance to get the merge of the BLiMP-IT reviewed by end of the month?

@baberabb baberabb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! A couple of blockers though, and I think both point at the tasks not having been run end-to-end yet:

  1. _template_yaml sets validation_split: train, but NeTSlab/BLiMP-IT only has a test split.
  2. 27 task YAMLs reference dataset_names with no matching HF configs. the same 27 not listed in _blimp_it.yaml. They look like superseded duplicates (conversa_a_agreement_in_dp vs conversa_a_agreement_dp, etc.). Should those be deleted, or are the configs still to be uploaded?
  3. Can you add a row to lm_eval/tasks/README.md, with a short description as all the other tasks.

Can you do a --limit 10 run on any small model once the split is fixed? That'll confirm it end-to-end

- _template_yaml: validation_split: train -> test_split: test (dataset is test-only)
- add target_delimiter: "" for parity with blimp_nl/turblimp
- remove 27 superseded task YAMLs with no matching HF config (caused by minor naming issues in the config files, now all fixed; additional renaming applied for consistency)
- add four macro-phenomenon subgroups under blimp_it, unweighted aggregation
- add row to lm_eval/tasks/README.md and Groups/Tasks section to blimp_it/README.md
@cristianochesi

Copy link
Copy Markdown
Author

Thanks for the check! I fixed all the issues you mentioned and also grouped the phenomena to improve the readability of the results. I confirmed with a small test (lm_eval --model hf --model_args pretrained=GroNLP/gpt2-small-italian --tasks blimp_it --limit 10 --device cpu --batch_size 8) that everything runs smoothly now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants