I do reliability work on the class of bug where a system reports an outcome it never confirmed, and I went looking for it here because an SRE agent's whole product is a conclusion. Filing it because the concept you'd need is already built and just doesn't reach the person reading the answer.
The short version: when a toolset fails its prerequisites it's correctly marked FAILED and excluded. The investigation then runs on what's left and produces a normal conclusion. Nothing in that conclusion, and nothing printed beside it, says which data sources were unavailable.
The path
holmes/core/tools.py:954 — check_prerequisites does the right thing. Missing env vars, failed callables, disabled prereqs all set local_status = ToolsetStatusEnum.FAILED (:991, :994, :1004, :1009, :1016, :1021). The status is real and it's accurate.
holmes/main.py:173-180 — the investigate path builds the system prompt with the enabled set:
system_prompt = build_system_prompt(
toolsets=ai.tool_executor.toolsets,
skills=None,
system_prompt_additions=investigation_additions,
...
)Same shape at :754. So the model is handed the tools that loaded and told nothing about the ones that didn't. It can't caveat an absence it has no representation of.
pretty_print_toolset_status is imported at holmes/main.py:39 and called at exactly two places, :1049 and :1063, both inside the toolset command group. It's never called on the investigate or ask path.
So both ends are blind. The model doesn't know what it wasn't given, and the operator isn't told either unless they separately run holmes toolset list — which isn't a thing anyone does in the middle of an incident, because the output they're looking at appears complete.
Why this is a wiring gap rather than a missing feature
You already built the hard part. The status enum exists, prerequisites are checked properly, and there's a formatter that prints them nicely. The distinction between checked and clean and couldn't check is fully represented inside the process and simply doesn't survive to the artifact a human reads. That's a much smaller fix than it would be in a codebase that never had the concept.
What I haven't verified, plainly
I read this on main through the API. I have not run HolmesGPT and have not reproduced an investigation with a failed toolset. In particular I haven't checked whether the web UI or the API response surfaces toolset status somewhere the CLI doesn't — if it does, this is narrower than I've framed it and I'd like to know.
I'm flagging that because I killed two of my own findings this same week by tracing one layer further, including one where the library underneath was already handling the thing I was about to report.
A possible shape for the fix
Pass the failed/disabled toolsets into the investigation alongside the enabled ones, so the model can say "I couldn't reach Prometheus" in its own analysis — and print the same list under the conclusion in the CLI. Reusing pretty_print_toolset_status on the investigate path covers the operator half in about a line.
The stronger version is a field on the result object, so automation consuming an investigation can test whether it was complete rather than inferring completeness from the absence of complaints.
Happy to open a PR if the direction is welcome.
Last bit, and I'd rather say it plainly than have it be odd later: hunting this class in unfamiliar codebases is what I do for money — guards that never fire, checks that can't come out against you, verdicts with nothing behind them. I found this one while surveying, and it stands whatever you decide about me: file it, fix it, close it, all fine.
If you want the same pass over the whole toolset and reporting layer, that's $1,500 for a day — you, me, your codebase, on a call, and every patch, repro and note is yours the same day. Not a document that arrives in a fortnight. Terms: https://unreached.dev/audit.html — where I also publish my own false-positive rates and my own retractions, including two from this week where my own checks killed my own findings before I filed them.
I do reliability work on the class of bug where a system reports an outcome it never confirmed, and I went looking for it here because an SRE agent's whole product is a conclusion. Filing it because the concept you'd need is already built and just doesn't reach the person reading the answer.
The short version: when a toolset fails its prerequisites it's correctly marked
FAILEDand excluded. The investigation then runs on what's left and produces a normal conclusion. Nothing in that conclusion, and nothing printed beside it, says which data sources were unavailable.The path
holmes/core/tools.py:954—check_prerequisitesdoes the right thing. Missing env vars, failed callables, disabled prereqs all setlocal_status = ToolsetStatusEnum.FAILED(:991,:994,:1004,:1009,:1016,:1021). The status is real and it's accurate.holmes/main.py:173-180— the investigate path builds the system prompt with the enabled set:Same shape at
:754. So the model is handed the tools that loaded and told nothing about the ones that didn't. It can't caveat an absence it has no representation of.pretty_print_toolset_statusis imported atholmes/main.py:39and called at exactly two places,:1049and:1063, both inside thetoolsetcommand group. It's never called on the investigate or ask path.So both ends are blind. The model doesn't know what it wasn't given, and the operator isn't told either unless they separately run
holmes toolset list— which isn't a thing anyone does in the middle of an incident, because the output they're looking at appears complete.Why this is a wiring gap rather than a missing feature
You already built the hard part. The status enum exists, prerequisites are checked properly, and there's a formatter that prints them nicely. The distinction between checked and clean and couldn't check is fully represented inside the process and simply doesn't survive to the artifact a human reads. That's a much smaller fix than it would be in a codebase that never had the concept.
What I haven't verified, plainly
I read this on
mainthrough the API. I have not run HolmesGPT and have not reproduced an investigation with a failed toolset. In particular I haven't checked whether the web UI or the API response surfaces toolset status somewhere the CLI doesn't — if it does, this is narrower than I've framed it and I'd like to know.I'm flagging that because I killed two of my own findings this same week by tracing one layer further, including one where the library underneath was already handling the thing I was about to report.
A possible shape for the fix
Pass the failed/disabled toolsets into the investigation alongside the enabled ones, so the model can say "I couldn't reach Prometheus" in its own analysis — and print the same list under the conclusion in the CLI. Reusing
pretty_print_toolset_statuson the investigate path covers the operator half in about a line.The stronger version is a field on the result object, so automation consuming an investigation can test whether it was complete rather than inferring completeness from the absence of complaints.
Happy to open a PR if the direction is welcome.
Last bit, and I'd rather say it plainly than have it be odd later: hunting this class in unfamiliar codebases is what I do for money — guards that never fire, checks that can't come out against you, verdicts with nothing behind them. I found this one while surveying, and it stands whatever you decide about me: file it, fix it, close it, all fine.
If you want the same pass over the whole toolset and reporting layer, that's $1,500 for a day — you, me, your codebase, on a call, and every patch, repro and note is yours the same day. Not a document that arrives in a fortnight. Terms: https://unreached.dev/audit.html — where I also publish my own false-positive rates and my own retractions, including two from this week where my own checks killed my own findings before I filed them.