Skip to content

Use ast.literal_eval instead of eval in hellaswag_arabic_pfn#1294

Open
jayzuccarelli wants to merge 1 commit into
huggingface:mainfrom
jayzuccarelli:fix-arabic-hellaswag-literal-eval
Open

Use ast.literal_eval instead of eval in hellaswag_arabic_pfn#1294
jayzuccarelli wants to merge 1 commit into
huggingface:mainfrom
jayzuccarelli:fix-arabic-hellaswag-literal-eval

Conversation

@jayzuccarelli

Copy link
Copy Markdown

Fixes #1293

hellaswag_arabic_pfn (the hellaswag_okapi_ar prompt function) parsed line["endings"] with the builtin eval(). That field is a string-encoded list coming from the OALL/AlGhafa-Arabic-LLM-Benchmark-Translated dataset on the Hub, so a crafted value would execute arbitrary code on the machine running the eval.

This switches it to ast.literal_eval(), which parses the same string-encoded lists and raises ValueError on anything that is not a plain literal. It's also the pattern already used elsewhere in the codebase for string-encoded dataset fields (e.g. sacrebleu.py). A repo-wide grep shows this was the only remaining eval() on dataset content.

Tests added in tests/unit/tasks/test_arabic_tasks.py:

  • legitimate string-encoded endings still parse into the same Doc (choices/gold_index)
  • a non-literal payload in endings raises ValueError instead of executing — this test fails on main (the payload runs) and passes with this change

pytest tests/unit/tasks/test_arabic_tasks.py: 2 passed. ruff check and ruff format --check clean on both touched files.

The endings field comes from a community dataset on the Hub; eval()
executes arbitrary Python from it. ast.literal_eval parses the same
string-encoded lists safely and raises on anything else, matching how
other tasks already parse such fields.

Fixes huggingface#1293
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

1 participant