Add schema-derived binary codec - #7366
Merged
Merged
Conversation
🦋 Changeset detectedLatest commit: 57f8e22 The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
This was referenced Aug 20, 2026
IMax153
reviewed
Aug 20, 2026
IMax153
reviewed
Aug 20, 2026
tim-smart
force-pushed
the
agent/codex-engineer/58cacc24
branch
from
August 21, 2026 00:41
87d2fa4 to
68df691
Compare
tim-smart
force-pushed
the
agent/codex-engineer/58cacc24
branch
from
August 21, 2026 09:06
fd1ef4f to
0778874
Compare
tim-smart
commented
Aug 22, 2026
Pack schema field ids with wire kinds so scalar values no longer need per-field extents and booleans carry their value in the tag. Apply the same header design to index-signature record pairs, including interned keys. Give compact decimals their own field kind, keep every kind skippable for schema evolution, and bump both envelopes to wire version 2. Follow-up: row shape declarations can retain scalar wire kinds so reused rows can omit per-value extents.
Arrays of structs now use row runs in fingerprint mode, so repeated strings and record keys intern across rows. A declared shape is the presence mask over layout field order instead of a field id list, and positional structs honor the handed-over key intern table. The 200-row fingerprint payload drops from 19,454 to 7,741 bytes and its decode rate matches the default mode.
A cleanup pass over the codec with no wire, API, or behavior changes; the benchmark medians are unchanged within run noise and all 229 affected tests pass. Reuse - The pooled-writer protocol lives once: encodeFrame and encodeFrames were copy-paste twins of the checkout/reset/abort/restore sequence and now share runFrames. - The two hand-rolled WeakMap caches (compileMode, compileTarget) use Function.memoize. - assignProperty imports the shared __proto__-safe helper from internal/record.ts instead of carrying a byte-for-byte copy. - The local empty ParseOptions constant is SchemaAST.defaultParseOptions. Simplification - One registry maps native declaration ids to wire kind and exactness; toBinaryASTStep, astKind, and isExact read it instead of restating the id list, and compileDeclaration's fallback now fails loudly if a registry row has no layout constructor. - The success-exit bypass predicate is defined once (isSuccessExit) instead of restated at five entry points. - withTrustedDecode and passThrough share one bypassDecode factory; the codec pipe assembly shared by toCodec and toCodecDirect lives in assembleCodec. - The required-field epilogue duplicated between decodeStruct and decodeRunRow is checkRequiredFields. - Mode carries the fingerprint only as its byte form; the lo/hi words and their dummy defaults are gone. - decodeInline was subsumed by decodeSlot; sentinel sets hash once during union compilation; the encoder's unreachable EMPTY wire case is gone; two test fixtures (Varying rows, the recursive Tree factory) are hoisted instead of repeated. Efficiency - decimalScale exits on the first out-of-range mantissa, which also rejects NaN and the infinities in one comparison instead of running all eight scales. - The decimal number-run write pass reuses the scales found during classification instead of rescanning each non-integer. - Literal sets larger than four values check membership in a Set. - The channel helpers derive their schema-dependent codecs once per channel instead of once per run, and decodeRunRow only records declared slots when the plan captures shapes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Layout-level index-signature shape cache and an unrolled uvarint fast path, from Codex Engineer.
Fixed-layout fast path for required-only struct runs and an unrolled ASCII string write, from Codex Engineer; the shared per-field write is factored out so the two row loops keep one copy of it. Benchmark numbers refreshed.
tim-smart
force-pushed
the
agent/codex-engineer/58cacc24
branch
from
August 24, 2026 00:42
8c65fcc to
f0536ef
Compare
SchemaBinary.encoder pairs with SchemaBinary.parser; dictionary: true makes them share one string table per string leaf across frames. RPC serialization enables it for envelopes, halving a typical request frame. The RPC benchmark now walks a stream in frame order instead of replaying one frame.
From Codex Engineer.
The binary layer already rejects every value for a never layout, already accepts every JSON value an unknown or any schema can hold, and validates a recursive schema in full on decode. Encoding a recursive schema keeps the pass because the cycle walk lives there.
The bypass hands the input straight back, which the shared exit says without allocating one per call. Benchmark numbers refreshed, including the adversarial raw-serializer tables.
One own-property enumeration and a lazily compiled name map replace per-field hasOwn discovery in row runs. From Codex Engineer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SchemaBinaryfromeffect/unstable/encodingand cover framing, evolution, parser behavior, native values, declarations, recursion, parse options, and error surfacesVerification
nix develop -c pnpm lintnix develop -c pnpm checknix develop -c pnpm circularnix develop -c pnpm vitest run packages/effect/test/unstable/encoding packages/effect/test/rpc packages/effect/test/clusternix develop -c pnpm test --run --project effect test/unstable/encoding/SchemaBinary.test.tsnix develop -c pnpm coverage --run --project effect test/unstable/encoding/SchemaBinary.test.ts --coverage.include=packages/effect/src/unstable/encoding/SchemaBinary.ts --coverage.reporter=textnix develop -c pnpm --dir packages/effect exec node benchmark/schema/SchemaBinary.tsnix develop -c pnpm --dir packages/effect exec node benchmark/rpc/RpcSerialization.tsCloses EFF-772
Closes EFF-773
Closes EFF-779
Closes EFF-784
Closes EFF-816
Closes EFF-827
Closes EFF-831
Closes EFF-832
Closes EFF-834
Closes EFF-840