Why
Existing public human-quality datasets — KADIK10k, TID2013, CID22 — are the inputs perceptual metrics like zensim train against. They have three structural gaps that hurt us in production:
- Codec coverage is dated. KADIK and TID use synthetic distortion types (Gaussian noise, blur, JPEG, JPEG2000). Real users see AVIF, JXL, WebP, modern mozjpeg, modern x265 stills. Even CID22 (the most modern) is dominated by traditional formats.
- Viewing conditions are uniform. Almost all existing data is "calibrated lab monitor at 60cm". Web/app users actually view content at sub-mm to 2m distances on phones, tablets, laptops, OLED TVs, e-ink, projectors, with ambient light from "dark theatre" to "outdoor sun". Pixel angular size and rod/cone activation shift dramatically across these regimes; perceptual masking is condition-dependent.
- No HDR / wide-gamut signal. P3 displays are now the modal mobile screen; HDR10/Dolby Vision is shipping. Existing datasets are 8-bit sRGB. We are training a metric to ship for content we have zero ground truth on.
The result for zensim: V0_2's TID-vs-KADIK regression isn't a metric defect — it's the metric correctly capturing two different sub-populations of viewing conditions, and the operator has no good way to weight one over the other. We need a dataset that spans the actual product distribution.
Product goal
A web app + small backend that lets us crowdsource human image-pair preference data at scale across heterogeneous devices and viewing conditions, and deposits the result in a structured corpus that zensim training (and codec dev work generally) can consume directly.
Concretely:
- A user lands on a URL, opts in (consent + minimal demographic), is shown a sequence of image-pair preference tasks tuned to their device, and is rewarded with a score / completion / ranking that's intrinsically motivating (no payment scheme in V1).
- Each task captures: which of two images is closer to a reference (or which is preferred), with optional Likert-scale severity. Pairwise comparison, not absolute MOS — better signal-to-noise per rating, doesn't anchor to rater-specific scales.
- Behind the scenes, every rating is stored with a thick context envelope: device characteristics, viewing distance estimate, ambient light estimate, screen calibration proxy, codec / quality / parameters of each image, image content class, and reference image identity.
- The corpus is the artifact. It must be downloadable as a flat CSV/Parquet so we can drop it into `zensim-validate --also` or similar tooling tomorrow morning.
Data goals
What we are actually trying to capture (in roughly this priority):
Per-rating record (one row in the output corpus)
- `pair_id`: stable hash of (reference, distorted_a, distorted_b)
- `reference_id`: stable identifier for the source content
- `reference_content_class`: photo / screenshot / line-art / mixed / synthetic-test / HDR-sourced
- `distorted_a_codec_path` and `distorted_b_codec_path`: full pipeline (e.g., `mozjpeg q70 4:2:0` vs `avif speed-4 q60 yuv420` vs `jxl-lossy d=1.5 progressive`). Stored as a normalized codec config string AND as the encoded byte stream length / bytes-per-pixel for quick filtering.
- `preferred`: `A` / `B` / `tie` / `both-bad` / `neither-distinguishable`
- `severity_a`, `severity_b`: optional 0-4 perceived defect severity (only when user opts to annotate)
- `defect_tags_a`, `defect_tags_b`: optional checkboxes — blockiness, banding, ringing, blur, color-shift, hue-shift, posterization, ghosting, contour, gibbs (the "we know our codecs fail this way" set, plus an "other (free text)" escape hatch). Multi-select.
- `task_duration_ms`: how long the rater spent on this comparison (gold-standard / attention check signal)
- `session_id`: opaque, lets us model rater drift and aggregate per-rater calibration
Per-rating context envelope
The whole point of the project is that this envelope is rich. Capture EVERYTHING the browser will give us, even if we don't immediately know how to weight it:
- `device_class`: phone / tablet / laptop / desktop-monitor / external-monitor / tv / unknown
- `viewport_px_w`, `viewport_px_h`
- `device_pixel_ratio` (gives us logical-px → physical-px)
- `screen_diagonal_in`: estimated, see calibration below
- `viewing_distance_estimate_cm`: estimated, see calibration below
- `pixel_angular_size_arcmin`: derived = pixel_px / (distance_cm × tan(1 arcmin)) — the actual perceptual primitive
- `ambient_light_lux_estimate`: from AmbientLightSensor API where available, else a self-reported bucket (cave / dim / room / bright / outdoor / direct-sun)
- `color_gamut_hint`: `screen.colorGamut` (sRGB / P3 / Rec2020) where available
- `hdr_capable_hint`: `window.matchMedia('(dynamic-range: high)')`
- `color_scheme_hint`: light / dark mode
- `hardware_concurrency`, `memory_gb_hint`, `connection_downlink_hint`: rough device tier
- `user_agent`, `os`, `browser`, `browser_version`: standard fingerprint, but truncated to bucket counts at storage time (no full UA strings retained)
- `gpu_renderer_hint`: WEBGL_debug_renderer_info where exposed (often masked, that's fine)
- `brightness_setting_hint`: only available on a few platforms but capture when present
- `time_of_day_local`: hour bucket
- `session_total_tasks_completed`: detect mid-session fatigue effects
- `task_index_in_session`: ditto
Per-image-pair coverage we want spanned
We will design the task-issuing engine to make sure the corpus actually spans this grid, not just the modal user. Active-learning-flavored: prioritize pairs that are underrepresented in the captured envelope.
- Reference content: photographic (~40%), screen content (~25%), line art / text (~15%), HDR-sourced (~10%), synthetic / test patterns (~10%). All the categories that pull existing metrics in opposite directions.
- Codec coverage: mozjpeg, jpegli, AVIF (rav1e + AOM), JXL (modular + VarDCT), WebP (lossy + lossless), HEIC where licensing allows, PNG (for line art / screen content), zenjpeg / zenwebp / zenavif / zenjxl outputs once shipping. Quality grid q5–q100 in step-5 plus densified q70-95 step-2.
- Resolution: phone-thumbnail (256px), web-medium (~1024px), desktop (~2048px), 4K (~4000px). Per zensim CLAUDE.md sweep discipline, tiny + small + medium + large.
- Distortion diversity: spans the q5-q40 aggressive-compression range as densely as q60-100. The Pareto-relevant bitrate band is at the bottom, not the top.
- Image-pair distance balance: avoid "obviously different" and "perceptually identical" — sample around the JND. Use the current zensim metric to pre-filter pair candidates: pre-compute zensim distance and bias sampling toward (zensim_estimated_jnd ± Δ) so each rater is making decisions that actually carry signal.
Quality assurance / calibration
Crowdsourced perceptual data is worthless without aggressive QA. V1 must include all of these:
- Viewing distance estimation. Two-step: (a) ask the user to hold a credit card to the screen and adjust an on-screen ruler until it matches — fixes the px → mm conversion; (b) use the WebXR / face-detection-via-camera (with explicit consent) OR a simpler arm-length proxy where the user reaches out and we detect occlusion, OR fallback self-report buckets ("phone in hand at desk" / "laptop on lap" / "monitor at desk" / "TV across room"). Distance is the single most impactful context variable; we'd rather have noisy estimates than none.
- Gold-standard pairs. Every session includes ~5% gold pairs where the correct answer is unambiguous (one image is bit-identical to the reference, the other is q5 noise). Raters who fail >20% of gold pairs have their session flagged and downweighted.
- Reversal pairs. Ask the same pair twice in a session, with A/B swapped. Flag inconsistent raters.
- Attention checks. Occasionally insert a "click the green button" interrupt task. Standard.
- Rater calibration. Per-rater reliability score derived from gold + reversal performance. Stored in the corpus as a per-row weight; the consumer can choose to filter or weight.
- Image preload + measurement. Both images displayed must finish loading and be visible for ≥ 1.5 s before the user can rate. `requestIdleCallback` + load timing telemetry to detect "they decided before they actually saw it".
- Anti-bot. No CAPTCHA but: rate limit per IP, require a non-trivial JS execution + WebGL context probe, drop sessions that submit ratings faster than human-possible (100ms threshold).
Technical sketch (V1)
Just enough that another engineer can scope it. The implementation can deviate.
- Frontend: TypeScript + Preact. Single-page, CDN-hosted. Total bundle <100 KB. Image delivery via signed URLs from object storage; no encoded artifacts in the bundle.
- Backend: Either a thin Cloudflare Worker / Lambda + S3 + DynamoDB (or similar), or a small Rust service. The persistent state is small (a corpus accumulates linearly; ratings are append-only). The hot path is "given a session ID, give me the next pair to rate" + "store a rating".
- Pair sampling: A daemon (cron or stream-processor) pre-computes the next 10K pairs to issue, biased toward gaps in the captured envelope. The frontend just pulls one off the queue. Keeps the per-request server logic dumb.
- Storage: append-only row-oriented log (one Parquet shard per day or per N ratings) that aggregates into the corpus on a schedule. Public dataset releases pull from this.
- Auth: anonymous-by-default, optional account for "track my contribution / leaderboard". No user data beyond what the rater volunteers.
- Localization: English V1; strings table from day one so we can add Japanese / Spanish / German cheaply later. Asian and South American screen / viewing patterns are in the long tail that English-only would systematically miss.
- Privacy: GDPR/CCPA-grade. Show the rater what we collect, give them a delete button, retain only what's needed for the corpus. No raw camera frames stored even if camera is used for distance estimation — only derived metrics.
Reward / motivation (V1)
Without payment we need intrinsic motivation. Combine:
- A score that ramps with consistency (gold-pair pass rate, reversal-pair pass rate)
- A leaderboard segment for raters who opt in
- A "your contribution helped train [release]" callout when a corpus shipment lands
- Optionally: a small cosmetic credit on imageflow / zensim project pages for top contributors
Out of scope for V1
- Payment integration (Mechanical Turk style). V1 is volunteer-only.
- Mobile apps. Web only — covers ≥ 95% of devices we care about.
- Real-time consensus aggregation. Batch processing is fine.
- Active learning with online model updates. Pair sampling can be stratified-random in V1; AL bias is V2.
- Video. Stills only — V1 is hard enough.
- Subjective annotation beyond pairwise + a few defect tags. No "describe what you see in your own words" free text.
Future work (post-V1)
- Active-learning pair selection driven by current zensim model's uncertainty
- Triplet comparisons ("which of B or C is more similar to A?") for finer signal
- HDR pair task design (tone-mapped vs HDR-native viewing)
- Adversarial corpus (intentionally hard pairs the metric currently fails on, to focus training)
- Integration with the codec-corpus crate so ratings flow back into a shared test/training source
What we need from this issue
A V1 that ships and can deposit at minimum 50K ratings spanning the device/codec grid above. Once data starts arriving, zensim training can consume it via `zensim-validate --also`. The corpus is the deliverable; the web app is just the collection vehicle.
cc: links to existing CID22 / KADIK / TID datasets and zensim training docs in the repo.
Why
Existing public human-quality datasets — KADIK10k, TID2013, CID22 — are the inputs perceptual metrics like zensim train against. They have three structural gaps that hurt us in production:
The result for zensim: V0_2's TID-vs-KADIK regression isn't a metric defect — it's the metric correctly capturing two different sub-populations of viewing conditions, and the operator has no good way to weight one over the other. We need a dataset that spans the actual product distribution.
Product goal
A web app + small backend that lets us crowdsource human image-pair preference data at scale across heterogeneous devices and viewing conditions, and deposits the result in a structured corpus that zensim training (and codec dev work generally) can consume directly.
Concretely:
Data goals
What we are actually trying to capture (in roughly this priority):
Per-rating record (one row in the output corpus)
Per-rating context envelope
The whole point of the project is that this envelope is rich. Capture EVERYTHING the browser will give us, even if we don't immediately know how to weight it:
Per-image-pair coverage we want spanned
We will design the task-issuing engine to make sure the corpus actually spans this grid, not just the modal user. Active-learning-flavored: prioritize pairs that are underrepresented in the captured envelope.
Quality assurance / calibration
Crowdsourced perceptual data is worthless without aggressive QA. V1 must include all of these:
Technical sketch (V1)
Just enough that another engineer can scope it. The implementation can deviate.
Reward / motivation (V1)
Without payment we need intrinsic motivation. Combine:
Out of scope for V1
Future work (post-V1)
What we need from this issue
A V1 that ships and can deposit at minimum 50K ratings spanning the device/codec grid above. Once data starts arriving, zensim training can consume it via `zensim-validate --also`. The corpus is the deliverable; the web app is just the collection vehicle.
cc: links to existing CID22 / KADIK / TID datasets and zensim training docs in the repo.