Skip to content

Add script-aware language tags for FLORES 200#1272

Open
goktugozkanmd wants to merge 2 commits into
huggingface:mainfrom
goktugozkanmd:a67-issue-745-language-script
Open

Add script-aware language tags for FLORES 200#1272
goktugozkanmd wants to merge 2 commits into
huggingface:mainfrom
goktugozkanmd:a67-issue-745-language-script

Conversation

@goktugozkanmd

Copy link
Copy Markdown

Summary

  • Add Script, LanguageWithScript, and language_from_tag() to preserve script information from dataset tags such as zho_Hant, while Language.value keeps the base ISO 639 code
  • Add script-aware translation literal lookup with fallback to the base language
  • Add Traditional Chinese template literals and re-enable zho_Hant in FLORES 200

Fixes #745.

Tests

  • ruff check and ruff format --check pass on the changed files
  • pytest tests/unit/prompt tests/unit/tasks tests/unit/utils/test_language.py — 67 passed

@goktugozkanmd

Copy link
Copy Markdown
Author

Refreshed this branch against current main (64f4f5a) without dropping the existing commit, then revalidated the issue #745 path.

What is now covered:

  • keeps Language.value/ISO-639-3 lookup compatibility;
  • preserves the FLORES ISO-15924 suffix in LanguageWithScript;
  • restores zho_Hant with Traditional Chinese literals while other script variants fall back to their base-language literals;
  • retains regression coverage for duplicate-code normalization and prompt rendering.

Validation on the refreshed head e234530:

pytest -q tests/unit/utils/test_language.py tests/unit/tasks/test_flores200.py tests/unit/tasks/templates/test_multichoice.py
9 passed

ruff check <6 changed Python files>
All checks passed!

ruff format --check <6 changed Python files>
6 files already formatted

The PR currently has no hosted checks reported on the fork branch, so these are the exact local results. It is ready for maintainer direction/review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FT] Manage script and language in the Language enum

1 participant