[core] Support snapshot-based sequence ordering for primary-key tables

[JunRuiLee]

Purpose

close #7806

Tests

• SchemaValidationTest#testSnapshotSequenceOrderingHappyPath
• SchemaValidationTest#testSnapshotSequenceOrderingRejectsSequenceField
• SchemaValidationTest#testSnapshotSequenceOrderingRejectsNonPkTable
• KeyValueWithLevelNoReusingSerializerSnapshotIdTest#testRoundTripWithSnapshotId
• KeyValueWithLevelNoReusingSerializerSnapshotIdTest#testRoundTripWithoutSnapshotId
• SortMergeSnapshotOrderingTest#testLaterSnapshotWinsOverHigherSequence
• SortMergeSnapshotOrderingTest#testFallsBackToSequenceWhenSnapshotMissing
• SortMergeSnapshotOrderingTest#testSameSnapshotFallsBackToSequence
• SortMergeSnapshotOrderingTest#testStampedAlwaysBeatsUnstamped
• PrimaryKeySimpleTableTest#testSnapshotSequenceOrdering
• PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingFallsBackToSequenceWithinSnapshot
• PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingCompactionPreservesInputSnapshotId
• PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingWithChangelogInput
• PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingWithChangelogLookup
• PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingDeleteFromLaterSnapshot

[JunRuiLee]

Hi @JingsongLi, could you help take a look? Many thanks.

[JunRuiLee]

Thanks @leaves12138 for the review! Fixed the compaction ordering issue by persisting per-record snapshotId through _SEQUENCE_NUMBER column. Added tests for the scenario you described. Old constructor removed.

PTAL, Thanks!

[leaves12138]

Thanks for the update. I took another careful pass over the snapshot-ordering implementation. I think there are still a few correctness issues to address before this can be safely merged.

[leaves12138]

This option is currently accepted for every primary-key merge engine, but the implementation only preserves snapshotId for merge functions that return an input KeyValue. For example, PartialUpdateMergeFunction and AggregateMergeFunction build a new KeyValue via replace(...), which resets snapshotId to UNKNOWN_SNAPSHOT_ID. During compaction, stampSequenceWithSnapshotId then writes -1 into _SEQUENCE_NUMBER / file sequence metadata, so later reads can order compacted records incorrectly. Could you either restrict sequence.snapshot-ordering to the supported merge engine(s) here, or propagate the winning snapshot id through all merge functions and add tests for partial-update / aggregation?

[JunRuiLee]

fixed

[leaves12138]

The lookup changelog path can still lose the snapshot id when LookupMergeFunction spills its KeyValueBuffer to the binary buffer. KeyValueBuffer.createBinaryBuffer still constructs new KeyValueWithLevelNoReusingSerializer(keyType, valueType) without includeSnapshotId, so after lookup.merge-records-threshold is exceeded, deserialized candidates have UNKNOWN_SNAPSHOT_ID and this comparator falls back to sequence-only ordering. Please thread snapshotSequenceOrdering into KeyValueBuffer's serializer and add a test that forces lookup-buffer spill, for example with a very small lookup.merge-records-threshold and an IOManager.

[JunRuiLee]

fixed

[leaves12138]

This option also looks unsafe to enable on a table that already has data written without the feature. Existing APPEND files have minSequenceNumber as the old sequence range, and existing COMPACT files have _SEQUENCE_NUMBER as the old per-record sequence number; after toggling this option on, readers will interpret those values as snapshot ids. Could this be documented and/or rejected for ALTER TABLE as a creation-only option? Otherwise an existing table can silently reorder old records.

[JunRuiLee]

this option is annotated as immutable, so enabling it via ALTER on a table with existing snapshots is rejected; empty-table ALTER remains allowed.

[JunRuiLee]

Thanks @leaves12138 for the careful review.

I fixed the first two correctness issues:

1. PartialUpdateMergeFunction and AggregateMergeFunction now preserve the winning input record’s snapshotId when returning a newly built KeyValue, so compaction no longer stamps UNKNOWN_SNAPSHOT_ID into
   _SEQUENCE_NUMBER.
2. KeyValueBuffer now preserves snapshotId when snapshot ordering is enabled, so lookup compaction buffer spill does not lose it during binary serialization/deserialization.

I also added regression coverage for merge-function snapshotId preservation, partial-update compaction, aggregate compaction, and lookup buffer spill.

For the ALTER TABLE concern: this option is annotated as immutable, so enabling it via ALTER on a table with existing snapshots is rejected; empty-table ALTER remains allowed.