Fix terminal priorities ignored in dynamic Earley lexer for recursive rules

[deepakganesh78]

Summary

Fixes #1441

Terminal priorities (e.g. FLOAT.1) were not being respected in recursive rules when using the dynamic Earley lexer (the default for parser='earley').

Reproduction

from lark import Lark

parser = Lark(r"""
%import common.SIGNED_INT
%import common.SIGNED_FLOAT
%import common.WS
%ignore WS
FLOAT.1: SIGNED_FLOAT
INT: SIGNED_INT
start: FLOAT | INT | "[" start* "]"
""")
print(parser.parse('[1 2.0 3]').pretty())

Before fix (on master):

start
  start  1
  start  2
  start  .0
  start  3

"2.0" is split into INT("2") + FLOAT(".0") = 4 children.

After fix:

start
  start  1
  start  2.0
  start  3

"2.0" is correctly matched as FLOAT("2.0") = 3 children. Matches LALR and earley/basic behavior.

Root Cause

The dynamic Earley scanner (xearley.py) explores ALL possible terminal matches at each position, building an SPPF forest. Disambiguation is then handled by ForestSumVisitor which sums terminal priorities across packed nodes. However, for this grammar both derivations accumulate an identical priority sum of 1:

• Wrong: INT("1",p=0) + INT("2",p=0) + FLOAT(".0",p=1) + INT("3",p=0) = 1
• Correct: INT("1",p=0) + FLOAT("2.0",p=1) + INT("3",p=0) = 1

So the visitor cannot distinguish them and the wrong derivation wins by insertion order.

Fix

Apply traditional "longest match + highest priority" filtering in the scan phase of xearley.py:

• At each position i, collect all terminal matches.
• Find the highest priority among them and the longest match at that priority.
• Suppress any match from a lower-priority terminal whose length does not exceed the best high-priority match length.

Filtering rule in words: a lower-priority terminal's match is suppressed if and only if a higher-priority terminal also matches at the same position with length >= the lower-priority match. This mirrors the standard lexer behavior.

Conditions for filtering (to avoid breaking existing behavior):

1. Terminal priorities are in use (forest_sum_visitor is set)
2. complete_lex is False (dynamic_complete mode intentionally explores all sub-match lengths for full ambiguity)
3. The matched terminals have differing priorities (best_priority > min_priority). This correctly handles the negative-priority idiom: INT.-1 to de-prioritize a terminal also triggers filtering.

Safe fallback: If a terminal name is missing from terminals_by_name (e.g. %declare), its priority defaults to 0 via a helper function, avoiding AttributeError.

Validation

Per-permutation test results (all green):

Class	Result
TestEarleyDynamic	89 passed, 16 skipped
TestEarleyDynamic_complete	88 passed, 17 skipped
TestEarleyBasic	88 passed, 17 skipped
TestLalrContextual	98 passed, 7 skipped
TestFullEarleyDynamic	31 passed, 2 skipped
TestFullEarleyBasic	22 passed, 11 skipped
TestParsers	17 passed
Total	990 passed, 176 skipped, 0 failed

Priority-specific tests: 17 passed, 6 skipped (including test_priority_vs_embedded, test_ignore_carryover_with_priority)

Regression test added: test_terminal_priority_in_recursive_rule — skipped for basic (already works) and dynamic_complete (different semantics).

Compatibility

• Grammars without terminal priorities: unaffected (no forest_sum_visitor or all priorities equal).
• dynamic_complete lexer mode: unaffected (filtering disabled).
• Negative priority idiom (TERM.-1): works correctly, triggers filtering when priorities differ.
• %declare terminals: handled gracefully (default priority 0).