Summary
holmes/checks/checks.py pins CHECK_RESPONSE_FORMAT (a strict json_schema response_format) on the LLM call, and tool_calling_llm.py forwards it into every iteration of the tool loop. Providers that implement response_format as grammar-constrained decoding (Ollama → llama.cpp GBNF, and any OpenAI-compatible local server doing the same) constrain the entire generation to the schema — tool-call tokens become unreachable, so the model can never execute the check's queries.
Observed behavior (holmes 0.36.0, Ollama 0.32.1, qwen2.5:7b-instruct)
- Every check = exactly one completion, zero tool executions.
rationale = the model's opening plan squeezed into the forced schema (e.g. "Executing the Prometheus queries to check for active anomalies.").passed = an ungrounded guess — we reproduced a hallucinated passed: true for a query that was never run. This silently defeats the purpose of the checks feature.- Reproduced identically on both the
ollama_chat/ and openai/ (Ollama /v1) LiteLLM routes.
Minimal reproduction
POST to Ollama /v1/chat/completions with a tools array and the checks-API response_format → tool_calls: null + schema-shaped content. Remove response_format from the same request → correct tool_calls emitted (streaming and non-streaming).
Suggested fix
Apply response_format only on the final iteration (when the model returned no tool calls / the loop is being finalized), not while tools are still in play. That preserves the structured verdict for all providers while restoring tool use on grammar-constrained ones.
Workaround we're running
additional_drop_params: ["response_format"] on the model entry (flows through **self.args into litellm.completion) + a prompt clause demanding a raw-JSON final answer so _parse_check_response's bare json.loads still works. Happy to PR the final-iteration-only change if you'd take it.
Summary
holmes/checks/checks.pypinsCHECK_RESPONSE_FORMAT(a strictjson_schemaresponse_format) on the LLM call, andtool_calling_llm.pyforwards it into every iteration of the tool loop. Providers that implementresponse_formatas grammar-constrained decoding (Ollama → llama.cpp GBNF, and any OpenAI-compatible local server doing the same) constrain the entire generation to the schema — tool-call tokens become unreachable, so the model can never execute the check's queries.Observed behavior (holmes 0.36.0, Ollama 0.32.1, qwen2.5:7b-instruct)
rationale= the model's opening plan squeezed into the forced schema (e.g. "Executing the Prometheus queries to check for active anomalies.").passed= an ungrounded guess — we reproduced a hallucinatedpassed: truefor a query that was never run. This silently defeats the purpose of the checks feature.ollama_chat/andopenai/(Ollama/v1) LiteLLM routes.Minimal reproduction
POST to Ollama
/v1/chat/completionswith atoolsarray and the checks-APIresponse_format→tool_calls: null+ schema-shaped content. Removeresponse_formatfrom the same request → correcttool_callsemitted (streaming and non-streaming).Suggested fix
Apply
response_formatonly on the final iteration (when the model returned no tool calls / the loop is being finalized), not while tools are still in play. That preserves the structured verdict for all providers while restoring tool use on grammar-constrained ones.Workaround we're running
additional_drop_params: ["response_format"]on the model entry (flows through**self.argsintolitellm.completion) + a prompt clause demanding a raw-JSON final answer so_parse_check_response's barejson.loadsstill works. Happy to PR the final-iteration-only change if you'd take it.