Add Falsify / PRML eval-audit article to Testing, Monitoring and Main… - #177
Open
sk8ordie84 wants to merge 1 commit into
Open
Add Falsify / PRML eval-audit article to Testing, Monitoring and Main…#177sk8ordie84 wants to merge 1 commit into
sk8ordie84 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding: PRML — Pre-Registered ML Manifest (open spec) +
falsifyreference implementation.Why this fits Testing, Monitoring and Maintenance: PRML pre-registers ML evaluation claims with a SHA-256 hash before the run, then verifies them in CI. The linked piece is a methodology article on running evaluation reproducibility audits — fits the section's article-driven format. Manifests are byte-equivalent across Python, JavaScript, Go, and Rust reference implementations.
What it is: Open specification (CC BY 4.0). A SHA-256 manifest binds metric, comparator, threshold, dataset hash, seed, and producer identity before the experiment runs. Verifier deterministically emits PASS (exit 0), FAIL (exit 10), TAMPERED (exit 3), or GUARD (exit 11).
Status: v0.1 published with Zenodo DOI 10.5281/zenodo.20177839. SchemaStore catalog merged. GitHub Marketplace Action live (
prml-verify). Four reference implementations all byte-equivalent on 20 conformance vectors. v0.2 RFC open through 2026-05-22.Maintainer: Cüneyt Öztürk · Studio 11 —
hello@studio-11.co