The TREC AutoJudge starter kit has introduced a convention for judges that make no LLM calls: they declare this in their own workflow.yml:
uses_llm: false
Since the IR-Axioms judge scores axiomatically without calling an LLM, trec-auto-judge/workflow.yml should carry this declaration.
Background: judges on TIRA receive their LLM endpoint through injected environment variables (OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY), and the starter kit ships an endpoint-contract test that verifies each judge actually
contacts the injected endpoint — a judge with a hardcoded endpoint only fails after submission, in a remote run that is hard to debug. Judges that use no LLM are exempt from that check, and the uses_llm: false flag in the judge's
own workflow is how they say so (the workflow runner ignores the key; only the tests read it). The flag also tells the organizers at a glance which submissions need endpoint injection at all.
The change itself is one line plus a comment; see the starter kit's judges/naive/workflow.yml for the pattern:
This judge makes no LLM calls (the endpoint-contract test xfails it)
uses_llm: false
Documentation: https://github.com/trec-auto-judge/.github/blob/main/profile/howto/02-configure-llm-endpoint.md#the-endpoint-contract
The TREC AutoJudge starter kit has introduced a convention for judges that make no LLM calls: they declare this in their own workflow.yml:
uses_llm: false
Since the IR-Axioms judge scores axiomatically without calling an LLM, trec-auto-judge/workflow.yml should carry this declaration.
Background: judges on TIRA receive their LLM endpoint through injected environment variables (OPENAI_BASE_URL, OPENAI_MODEL, OPENAI_API_KEY), and the starter kit ships an endpoint-contract test that verifies each judge actually
contacts the injected endpoint — a judge with a hardcoded endpoint only fails after submission, in a remote run that is hard to debug. Judges that use no LLM are exempt from that check, and the uses_llm: false flag in the judge's
own workflow is how they say so (the workflow runner ignores the key; only the tests read it). The flag also tells the organizers at a glance which submissions need endpoint injection at all.
The change itself is one line plus a comment; see the starter kit's judges/naive/workflow.yml for the pattern:
This judge makes no LLM calls (the endpoint-contract test xfails it)
uses_llm: false
Documentation: https://github.com/trec-auto-judge/.github/blob/main/profile/howto/02-configure-llm-endpoint.md#the-endpoint-contract