Skip to content

feat(inmemory): add Rayforce grouped and parted adapter#14

Open
singaraiona wants to merge 14 commits into
KxSystems:mainfrom
singaraiona:agent/rayforce-adapter-completion
Open

feat(inmemory): add Rayforce grouped and parted adapter#14
singaraiona wants to merge 14 commits into
KxSystems:mainfrom
singaraiona:agent/rayforce-adapter-completion

Conversation

@singaraiona

Copy link
Copy Markdown

Summary

  • add Rayforce as an in-memory NYSE TAQ query engine
  • import the full trade and quote schemas into a native Rayforce splayed database
  • implement all 53 benchmark queries
  • benchmark both grouped (time) and parted (sym,time) layouts
  • support query filtering, persisted correctness output, and standard merged PSV results

Rayforce dependency

This adapter requires the fixes merged in RayforceDB/rayforce#326. Build Rayforce from clean release objects:

cd /path/to/rayforce
make clean
make release

Set RAYFORCE_BIN when the Rayforce and benchmark repositories are not sibling directories:

export RAYFORCE_BIN=/absolute/path/to/rayforce/rayforce

Data preparation

Rayforce needs the new full-schema database; databases produced by the earlier reduced-schema adapter must be regenerated. Generate the KDB database first because the Rayforce converter exports its source data from the sibling kdb directory:

DATAFORMAT=kdb ./generateDB.sh \
  "${NYSEBENCHMARKDIR}/${SIZE}/psv" \
  "${NYSEBENCHMARKDIR}/${SIZE}/kdb" \
  "${DATE}"

DATAFORMAT=rayforce RAYFORCE_BIN=/absolute/path/to/rayforce/rayforce \
  ./generateDB.sh \
  "${NYSEBENCHMARKDIR}/${SIZE}/psv" \
  "${NYSEBENCHMARKDIR}/${SIZE}/rayforce" \
  "${DATE}"

Run Rayforce alone with:

./benchmarks/inmemory/queryEngines.sh \
  --db-dir "${NYSEBENCHMARKDIR}/${SIZE}" \
  --param-dir "./artifacts/parameters/${SIZE}" \
  --date "${DATE}" \
  --threads "0 4 16 64" \
  --engines rayforce

Validation

  • grouped output matches KDB-X for 53/53 queries
  • parted output matches the verified parted reference for 53/53 queries
  • grouped, 0-secondary-thread run: KDB-X 3.584 s, Rayforce 2.880 s (1.245x total speedup; 27/53 query wins)
  • parted, 0-secondary-thread run: KDB-X 2.931 s, Rayforce 2.490 s (1.177x total speedup; 33/53 query wins)
  • post-upstream-sync smoke test passed for grouped and parted layouts
  • shell syntax and whitespace checks pass

The timing figures are from the small full-schema validation dataset and are intended as a regression/performance sanity check, not as final published benchmark numbers.

- src/rayforce/: native DB build via .csv.splayed, in-memory runner
  (load, sort by time, 3x timing, error-tolerant), assembly to the
  standard 21-column result PSV, row-count verification helpers, and the
  per-engine parameter prelude
- artifacts/queries/inmemory/rayforce.psv: the 52 benchmark queries in Rayfall
- generateDB.sh: DATAFORMAT=rayforce path (export to CSV -> splayed DB)
- benchmarks/inmemory/queryEngines.sh: rayforce wired into the engine list
q34/q35/q36/q43 rewritten from select-by + find broadcast to the native
window form (agg OVER PARTITION BY + filter), landing on rayforce dev's new
`window` special form. Rowcounts identical (27/32/64397/564). vs kdb-x
(small, 4 threads): q36 155->11.5ms now beats kdb (16.4); q34 ->8.7ms ~tie;
q43 ->7.6ms and q35 ->52ms hugely closer (was 184x/34x losses).
Mirror of the kdb runner's addAttr step (g#sym on the time-sorted tables):
after the measured load+sort, rebuild trade/quote with the sym column
carrying the grouped (hash) index the engine consults for (== sym X) and
(in sym [...]) point filters.  Recorded as its own idx -3 "index" row,
exactly like kdb's -3 addAttr row (the runner already declared indexon=sym).

With the engine-side sparse-selection work (rayforce 7e6bc9fe, 5a0a7c25):
q13 30.5->1.4ms, q47 25.2->1.3ms, q41 -66%, q14 -65%, q30 -60%, q05 -26%;
dense-key queries (mostFreqInstr) are unaffected — the probe aborts to the
scan at that density.  Index build cost ~154ms, inside the measured step.
The measured sort step now orders trade/quote by (sym, time) instead of
time — the same layout choice the kdbParted configuration makes (p# on a
sym,time-sorted table): every symbol's rows become one contiguous block,
so per-symbol filters gather sequentially and the grouped index's CSR
slices are dense ranges.  sortcols metadata updated to "sym,time".

With the engine-side parted package (rayforce 30fb8c45): 30/52 wins vs
kdb-x 4T (from 22-24 on the time-sorted prep).  q05 1.14ms vs kdb 1.16,
q13 0.53 vs 0.73, q06/q07/q14/q15/q18/q29/q40/q43 flip to WINs; asof
q49 25.6 WIN / q50 2.95 / q51 3.01 / q52 7.8 (the one regression — the
verify-first per-slice check pays ~6ms where global time order was free).
perf(rayforce): sym-index arc + parted table prep
The kdb reference query is built on `(<) prior seq`; rayforce now has
the same pairwise combinator, so the translation drops the hand-rolled
concat/take shift idiom (three full-column copies per shifted operand)
for (prior < seq) / (prior == sym) — same results, one fused pass per
mask. 2.5ms -> 1.6ms vs kdb-x 1.96ms.
perf(rayforce): q37 via the prior combinator
Align full TAQ schemas and query semantics, add grouped/parted runner integration, and support correctness output collection.\n\nValidated all 53 queries in grouped and parted layouts; grouped output matches KDB-X 53/53.
Merge the current upstream benchmark contract, implement all 84 Rayforce queries, and add date-scoped grouped/parted execution with exact KDB-X output verification.
@singaraiona
singaraiona marked this pull request as ready for review July 16, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant