feat(inmemory): add Rayforce grouped and parted adapter#14
Open
singaraiona wants to merge 14 commits into
Open
feat(inmemory): add Rayforce grouped and parted adapter#14singaraiona wants to merge 14 commits into
singaraiona wants to merge 14 commits into
Conversation
- src/rayforce/: native DB build via .csv.splayed, in-memory runner (load, sort by time, 3x timing, error-tolerant), assembly to the standard 21-column result PSV, row-count verification helpers, and the per-engine parameter prelude - artifacts/queries/inmemory/rayforce.psv: the 52 benchmark queries in Rayfall - generateDB.sh: DATAFORMAT=rayforce path (export to CSV -> splayed DB) - benchmarks/inmemory/queryEngines.sh: rayforce wired into the engine list
q34/q35/q36/q43 rewritten from select-by + find broadcast to the native window form (agg OVER PARTITION BY + filter), landing on rayforce dev's new `window` special form. Rowcounts identical (27/32/64397/564). vs kdb-x (small, 4 threads): q36 155->11.5ms now beats kdb (16.4); q34 ->8.7ms ~tie; q43 ->7.6ms and q35 ->52ms hugely closer (was 184x/34x losses).
Mirror of the kdb runner's addAttr step (g#sym on the time-sorted tables): after the measured load+sort, rebuild trade/quote with the sym column carrying the grouped (hash) index the engine consults for (== sym X) and (in sym [...]) point filters. Recorded as its own idx -3 "index" row, exactly like kdb's -3 addAttr row (the runner already declared indexon=sym). With the engine-side sparse-selection work (rayforce 7e6bc9fe, 5a0a7c25): q13 30.5->1.4ms, q47 25.2->1.3ms, q41 -66%, q14 -65%, q30 -60%, q05 -26%; dense-key queries (mostFreqInstr) are unaffected — the probe aborts to the scan at that density. Index build cost ~154ms, inside the measured step.
The measured sort step now orders trade/quote by (sym, time) instead of time — the same layout choice the kdbParted configuration makes (p# on a sym,time-sorted table): every symbol's rows become one contiguous block, so per-symbol filters gather sequentially and the grouped index's CSR slices are dense ranges. sortcols metadata updated to "sym,time". With the engine-side parted package (rayforce 30fb8c45): 30/52 wins vs kdb-x 4T (from 22-24 on the time-sorted prep). q05 1.14ms vs kdb 1.16, q13 0.53 vs 0.73, q06/q07/q14/q15/q18/q29/q40/q43 flip to WINs; asof q49 25.6 WIN / q50 2.95 / q51 3.01 / q52 7.8 (the one regression — the verify-first per-slice check pays ~6ms where global time order was free).
perf(rayforce): sym-index arc + parted table prep
The kdb reference query is built on `(<) prior seq`; rayforce now has the same pairwise combinator, so the translation drops the hand-rolled concat/take shift idiom (three full-column copies per shifted operand) for (prior < seq) / (prior == sym) — same results, one fused pass per mask. 2.5ms -> 1.6ms vs kdb-x 1.96ms.
perf(rayforce): q37 via the prior combinator
Align full TAQ schemas and query semantics, add grouped/parted runner integration, and support correctness output collection.\n\nValidated all 53 queries in grouped and parted layouts; grouped output matches KDB-X 53/53.
Merge the current upstream benchmark contract, implement all 84 Rayforce queries, and add date-scoped grouped/parted execution with exact KDB-X output verification.
singaraiona
marked this pull request as ready for review
July 16, 2026 13:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
time) and parted (sym,time) layoutsRayforce dependency
This adapter requires the fixes merged in RayforceDB/rayforce#326. Build Rayforce from clean release objects:
cd /path/to/rayforce make clean make releaseSet
RAYFORCE_BINwhen the Rayforce and benchmark repositories are not sibling directories:export RAYFORCE_BIN=/absolute/path/to/rayforce/rayforceData preparation
Rayforce needs the new full-schema database; databases produced by the earlier reduced-schema adapter must be regenerated. Generate the KDB database first because the Rayforce converter exports its source data from the sibling
kdbdirectory:Run Rayforce alone with:
Validation
The timing figures are from the small full-schema validation dataset and are intended as a regression/performance sanity check, not as final published benchmark numbers.