- Info about TREC Auto-Judge: What is this about?
- Data Set: Grab the pilot data to participate!
- Starter Kit Start developing your auto-judge!
- Participant HowTo: The canonical step-by-step guide, one page per activity — setup, LLM endpoint, develop an autojudge, running, prompt cache, meta-evaluation, and submission.
- Submission guidelines: A one-page summary of the points that commonly cause problems.
- Infrastructure Status: An (still in early development) page that intends to show the status of different parts of the infrastructure.
- Meta-evaluation For local evaluation and development.
Please add additional examples.
- Starter auto-judges include a naive judge without dependencies, a minimal LLM-baed example, and a complete LLM example.
- PrefNugget-based auto-judges
- LangChain preference-nugget auto-judge — brings its own LLM client (LangChain), with an endpoint-agnostic prompt cache and on-demand preference-pair growth
- IR Axioms inspired auto-judges
- Pyterrier retrieval auto-judges
- AutoNuggetizer-based auto-judge
