Add self-contained AI translation engine (GlotPress replacement)

[JorgeMucientes]

Summary

Adds the GlotPress-independent translation engine and wires it into WooCommerce Android release/PR automation.

This PR now contains the previously stacked engine/CI/Fastlane hardening work that was manually folded together:

• A self-contained Ruby translation engine under fastlane/ai_translation/.
• Android XML resource parsing/writing for strings, arrays, plurals, CDATA, placeholders, formatted="false", locale plural quantities, and preserved baseline lines.
• Manifest-driven source-only delta detection with per-locale source invalidation.
• Anthropic API and local Claude CLI clients, batched JSON translation, retry/backoff, parser hardening, recoverable client-error handling, and model escalation hooks.
• Context/glossary/style inputs, hard glossary gates, placeholder/XML/key-parity/plural validation, non-translatable @string/... reference validation, and text normalization.
• Import/backfill/shadow-diff modes, metadata translation mode, and runnable bin/woo-ai-translate / bin/merge-manifests shims.
• Fastlane lanes replacing the GlotPress download flow with AI PR-time, sweep, metadata, and release/hotfix paths.
• Buildkite PR-time translation automation, including bot commits, PR comments, fork/no-secret skipping, bot-loop prevention, per-PR concurrency, hard required status, generated XML context comments for newly added strings/arrays/plurals via i18n-context-generator, and an intentional skip-ai-translation label bypass.
• Docs/runbook updates for the new workflow.

No generated locale backfill files are added in this PR. The sidecar baseline is in #15974, Play metadata content is in #15983, the 15-locale Android string backfill is in #16004, and the draft PR-time pipeline fixture is #16011.

Blocker and Review Fixes Included

• Source-change invalidation is now per (key, locale), so updating one stale locale no longer makes the other locales for that key look fresh.
• prtime --strict only fails on hard gate errors; recoverable per-key translation gaps remain non-blocking and fall back to default resources.
• The AI translation Buildkite step is no longer soft-failed, is serialized per PR, and has an explicit label bypass for intentional skips.
• New PR resources now get best-effort AI-generated XML context comments before translation for <string>, <string-array>, and <plurals> entries; failures are logged and skipped so translation can proceed.
• Manifest::SEP now uses an escaped NUL sentinel instead of a literal NUL byte in Ruby source, with a regression test that prevents NUL bytes in the AI translation Ruby files.
• European Portuguese (pt-rPT) CLDR plural coverage now includes many, matching CLDR 47 cardinal categories.
• Generated output validation now catches stale non-translatable @string/... references that no longer exist in the source resource table.
• Translation output is now written through a temp file and only replaces the destination after XML validation, so malformed generated XML cannot overwrite a good locale file.
• Generated XML context comments are sanitized defensively before being emitted.
• Metadata fallback no longer writes over-cap English text when retries still exceed the Play cap.
• merge-manifests now deep-merges metadata locale shards instead of replacing whole metadata buckets.
• Anthropic response handling now wraps malformed JSON consistently and anchors HTTP status parsing.
• The rollout doc now reflects the required PR-time check, skip label behavior, and fork/no-secret skip path.

Stack

Base: #15961.

Open stack above this PR:

• Seed manifest + sidecar baseline for the 16 existing locales #15974 - sidecar baseline for existing human locales, restored source context comments, and folded metadata char-cap test coverage.
• Backfill Play Store metadata for the 15 new locales #15983 - Play Store metadata backfill for the 15 new locales.
• Backfill 15 new locales via Claude Sonnet subagents #16004 - Android app-string backfill for the 15 new locales.
• Add PR-time AI translation pipeline fixture #16011 - draft PR-time AI translation pipeline fixture.

Merged/folded stack PR:

• Test: metadata char-cap re-prompt success path (workstream 3c) #15982 - metadata char-cap re-prompt success-path test, now included in Seed manifest + sidecar baseline for the 16 existing locales #15974/Backfill Play Store metadata for the 15 new locales #15983 history.

Full stack map: #15961.

Test Plan

• ruby fastlane/ai_translation/spec/woo_ai_translation_test.rb - 98 runs / 362 assertions / 0 failures.
• bash -n .buildkite/commands/ai-translate-pr.sh
• ruby -c fastlane/ai_translation/lib/woo_ai_translation/manifest.rb
• ruby -c fastlane/ai_translation/lib/woo_ai_translation/validators.rb
• ruby -c fastlane/ai_translation/lib/woo_ai_translation/engine.rb
• ruby -c fastlane/ai_translation/lib/woo_ai_translation/metadata_engine.rb
• ruby -c fastlane/ai_translation/lib/woo_ai_translation/anthropic_client.rb
• bundle exec rubocop fastlane/ai_translation --force-exclusion - current repo config excludes this self-contained tool, so RuboCop inspects 0 files.
• bundle exec i18n-context-generator version - i18n-context-generator 0.4.0.
• bundle exec i18n-context-generator extract --translations WooCommerce/src/main/res/values/strings.xml --source WooCommerce/src/main --keys settings_usage_tracking --dry-run
• ruby -Ifastlane/ai_translation/lib fastlane/ai_translation/spec/woo_ai_translation_test.rb --name '/comment|context/'
• git diff --check
• ruby -c fastlane/Fastfile
• Manual probes confirm per-locale stale handling and PR-time strict-mode behavior.

[dangermattic]

	9 Warnings
⚠️	This PR is larger than 300 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.
⚠️	Class Unit is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class StubClient is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class ContextProvider is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class ShadowDiff is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class SourceEncodingTest is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class GlossaryAndStyleTest is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class ClaudeCliClientTest is missing tests, but unit-tests-exemption label was set to ignore this.
⚠️	Class TranslatorJsonRepairTest is missing tests, but unit-tests-exemption label was set to ignore this.

Generated by 🚫 Danger

[wpmobilebot]

📲 You can test the changes from this Pull Request in WooCommerce Android by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce Android
	Platform	📱 Mobile
	Flavor	Jalapeno
	Build Type	Debug
	Build Number	750
	Version	24.8-rc-1
	Application ID	com.woocommerce.android.prealpha
	Commit	7aa3f96
	Installation URL	1q9a0lcco7mk8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[toupper]

Reviewed the 39 changed files (~4.6k LOC added) on top of issue/woomob-translations-p1-locales. The engine is impressively well-scoped — stdlib-only Ruby with 1.4k LOC of test coverage, a deliberate source-only invalidation contract, hard validator gates, and a layered set of rescue paths so a single LLM hiccup never blocks a PR. The "preserved-XML for glotpress-import keys" approach is a smart way to avoid re-serializing 16 locales worth of human work just to introduce the engine.

I'm not blocking on anything; the items below are suggestions and observations for follow-up.

Suggestions

• Engine#translate_locale (fastlane/ai_translation/lib/woo_ai_translation/engine.rb:84) raises on Validators.xml_well_formed failure of the just-written file. That's the one place the engine can still die mid-sweep (after writing this locale's file, before reaching the next locale's manifest.save). The PR description and b6ef4feb25 framing both lean on "never crash mid-locale"; this raise undercuts that guarantee for an admittedly rare failure mode (Writer escaping is supposed to prevent it, but adversarial model output could still produce stray controls / lone surrogates that break REXML on the round-trip). Consider turning it into a per-locale gate_error so the rest of the sweep — and the manifest persistence — survive.

• .buildkite/commands/ai-translate-pr.sh:86 sanitizes only -- in LLM-generated XML comments. '---'.gsub('--', '- -') evaluates to '- --', which still contains -- and would produce an invalid XML comment when interpolated into <!-- … --> on line 107. Likewise a description ending in - becomes <!-- foo- -->, which is invalid (XML 1.0 forbids -- and --->). Either reject any description that still matches /--/ after the first pass, loop until stable, or fall through to no-comment-added (the rest of the pipeline already handles a missing comment gracefully). The same script also doesn't escape &, <, > in the description — these are technically legal inside an XML comment, but the resulting English context would be confusing for a reviewer.

• MetadataEngine#translate_field (fastlane/ai_translation/lib/woo_ai_translation/metadata_engine.rb:91-104) only re-prompts once with the "Be more concise" hint when the field is over cap. If the second attempt is also over, it silently falls back to the unmodified English source and writes that to disk — but the English source itself may exceed the cap (the test test_metadata_engine_caps_release_notes_gating_and_reuse exercises exactly this: a 60-char title is written as English fallback to a title.txt whose cap is 50). The fallback is documented as "English fallback, never over-cap garbage" but in this case it ships an over-cap English file. Consider either keeping the previous on-disk value when fallback would overflow, hard-truncating with a logged warning, or making this a hard gate on the metadata sweep.

• Translator#system_blocks flags only the last system block for prompt caching. The current ordering puts the per-locale style block last, so the cache breakpoint only covers up to glossary/rules. A future maintainer might reorder these or add another block after the per-locale style and quietly tank the cache hit rate. Worth a comment near cacheable_system in anthropic_client.rb pinning the contract ("last block is the cache breakpoint; everything before it is shared prefix"), and ideally an assertion in the test that asserts the cache_control is on a constant block.

• bin/merge-manifests discards data['metadata']['locales'] on key collision (merged['metadata'][k] = v overwrites the whole bucket). For a sharded backfill where each shard touches a disjoint set of locales but the metadata bucket key (e.g. "title" or "10.3:release_notes") is shared, the last shard wins and earlier shards' per-locale records are lost. Mirror the per-key locales-deep-merge you already do above for the metadata buckets.

• AnthropicClient#client_error_no_retry? (fastlane/ai_translation/lib/woo_ai_translation/anthropic_client.rb:140-146) parses the status from error.message[/HTTP (\d+)/, 1]. The error string is "HTTP #{res.code}: #{res.body}", and a 5xx whose body happens to start with HTTP 4xx ... would be classified as a non-retryable client error. Vanishingly unlikely, but the regex isn't anchored. A one-line guard or a more explicit ^HTTP (\d{3})(?=:) would remove the ambiguity.

• AnthropicClient#post_messages (line 106) calls JSON.parse(res.body) outside any rescue. A 2xx with malformed JSON would propagate JSON::ParserError past with_retries (which only rescues Error and TRANSPORT_ERRORS) and crash the entire run, bypassing the Translator's split_retry. The Anthropic API doesn't do this in practice, but the assumption is worth either documenting or hardening.

• engine.rb:283-286 runs Validators.glossary_preservation unconditionally on every entry but only runs placeholder_parity when formatted != 'false'. The asymmetry is correct intent (brand-name preservation still applies to literal-percent strings) but worth a one-line code comment so the next reader doesn't "fix" it.

• docs/translation-rollout.md:60-65 describes soft_fail: true as the rollout starting point, but .buildkite/pipeline.yml:53-62 ships the job without soft_fail, and the PR description calls this out as intentional ("AI translation Buildkite step is no longer soft-failed"). If this is the deliberate cutover state, consider trimming/updating the Phase 5 wording so the runbook doesn't read as "we are about to start" when we're already there.

Positives

• Source-only invalidation contract is explicit, documented in three places (Manifest class header, docs/localization.md table, dedicated tests test_model_bump_does_not_invalidate…, test_attribute_only_change_does_not_invalidate, test_manifest_loss_trusts_existing_localized_files). The "manifest loss does not clobber human work" guarantee is exactly the right escape hatch.

• Hard gates are blocking-by-design and the failure mode is "omit the key, let Android fall back to default" rather than "ship garbage" — both the placeholder gate and glossary gate are tested for this (test_placeholder_failure_is_dropped_not_shipped, test_glossary_preservation_failure_is_dropped_not_shipped).

• CLDR plural reshaping (dup_shell_for_locale) genuinely solves the Polish/Thai Lint warnings the old GlotPress path always had, with a CLDR-47-sourced data file and tested per-locale (test_polish_plurals_get_all_four_cldr_quantities_synthesized, test_thai_plurals_drop_irrelevant_one_quantity).

• Anthropic prompt-caching is wired correctly via cache_control: ephemeral on the last system block, and the temperature-rejection self-learning (@no_temperature_models) is a pragmatic workaround for the Opus 4.7 quirk with a clean test (test_drops_temperature_and_retries_on_deprecation_400, test_remembers_model_and_skips_temperature_on_subsequent_calls).

• The Translator's recoverable-error rescue list deliberately excludes NoMethodError / SystemCallError so genuine engine bugs still surface; that boundary is captured in test_unknown_error_class_still_propagates. Same intent in the AnthropicClient#client_error_no_retry? 4xx-except-429 gate. This is exactly the right "fail loud on bugs, fail soft on flakes" line.

• PartialWriter preserves unrelated existing lines byte-for-byte and only edits selected blocks; test_only_names_preserves_existing_file_order_and_header pins the contract on a fixture that includes the original GlotPress header. That keeps PR diffs scoped and makes the engine adoption non-destructive to the 16 existing locales.

• BaselineReader + the preserved_xml path on the Writer means existing human translations can be re-emitted byte-identically as long as their manifest origin stays glotpress-import — EnginePreservedLineIntegrationTest proves a deliberately idiosyncratic baseline spacing survives a full engine run, and test_ai_origin_keys_still_route_through_render_unit proves AI-origin keys go through the canonical writer.

• Tooling-namespace attributes (tools:override) round-trip intact via expanded_name, with a test that pins the regression (test_tools_namespace_prefix_survives_parse_and_write). Easy thing to miss and would silently break Android Lint suppressions in localized files.

• bin/woo-ai-translate CliShimTest covers the very specific failure mode of the blanket bin/ gitignore rule eating the shim — small but easy to regress.

• The fork-PR / no-secret path in ai-translate-pr.sh correctly short-circuits before doing any work, and the bot-commit self-skip (BOT_SKIP_MARKER) prevents the per-PR concurrency-limited job from looping on its own commit. Skip-label bypass is a sensible operator escape hatch.

• Operational concerns are addressed: per-PR concurrency in .buildkite/pipeline.yml, manifest is saved after each locale to survive Ctrl-C, mac-metal agent restricts where the API key is needed, source encoding test catches stray NUL bytes that previously caused breakage.

• Engine version (VERSION / PROMPT_VERSION) is recorded in every manifest entry as audit metadata even though it doesn't trigger re-translation — keeps the audit trail without coupling to invalidation, exactly as docs/localization.md describes.

---

Reviewed on behalf of @toupper — drafted by Claude Code, please confirm before merging.

[JorgeMucientes]

Thanks @toupper. I pushed fixes for the critical engine issues you called out.

Changed:

• Translation XML is now written through a temp file and only replaces the destination after validation, so malformed output does not overwrite a good locale file.
• PR context comments are sanitized more defensively before being emitted as XML comments.
• Metadata fallback no longer writes over-cap English text when retries still exceed the Play cap.
• merge-manifests now deep-merges metadata locale shards instead of letting the last shard replace the whole bucket.
• Anthropic response handling now wraps malformed JSON consistently and anchors HTTP status parsing.
• The rollout doc now matches the required PR-time check and documents the skip/fork behavior.

Not changed:

• I left the prompt-caching block ordering as-is because the current code already documents the final-block cache contract, and I did not find a functional bug there.
• I also left the glossary vs placeholder behavior as-is because skipping placeholder validation for formatted="false" is intentional while glossary checks still apply.

[wpmobilebot]

Version 25.0 has now entered code-freeze, so the milestone of this PR has been updated to 25.1.

[wpmobilebot]

Version 25.1 has now entered code-freeze, so the milestone of this PR has been updated to 25.2.

[wpmobilebot]

Version 25.2 has now entered code-freeze, so the milestone of this PR has been updated to 25.3.

[wpmobilebot]

Version 25.3 has now entered code-freeze, so the milestone of this PR has been updated to 25.4.

[wpmobilebot]

Version 25.4 has now entered code-freeze, so the milestone of this PR has been updated to 25.5.