Add agent observability with Monocle#287
Open
Mohammed Ansari (imohammedansari) wants to merge 1 commit into
Open
Add agent observability with Monocle#287Mohammed Ansari (imohammedansari) wants to merge 1 commit into
Mohammed Ansari (imohammedansari) wants to merge 1 commit into
Conversation
Enable Monocle (OpenTelemetry tracing for LLM apps) with one setup call plus the monocle_apptrace dependency. setup_monocle_telemetry auto-instruments the frameworks already in use and writes traces to .monocle/. Additive; no changes to application logic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Monocle observability to the agent. Monocle is an OpenTelemetry-based tracer for LLM applications. With it enabled, each run is recorded as a structured trace: the agent and graph invocations, tool calls, LLM inferences, token usage, and timings. The change is additive and does not alter application logic.
What this adds
Instrumentation is one setup call plus one dependency:
pyproject.toml: addsmonocle_apptrace.src/open_deep_research/deep_researcher.py: callssetup_monocle_telemetry(workflow_name="open-deep-research", monocle_exporters_list="file")at startup.That call auto-instruments the frameworks already in use (LangGraph and the LLM clients), so there is no per-tool or per-call wiring to maintain. Traces are written to
.monocle/by default.What you get
Each run produces a trace of all the agents that were triggered and the LLM inferences they made. In effect, it's the path the run took to answer the question. This is useful for developers building the agent, since they can see how it actually behaved on a run. The same traces are also a good basis for a behavioral test suite, an integration test that asserts on that behavior, and I've opened a companion PR that shows how that works: behavioral test suite. You can open the trace files directly, view them in the Monocle VS Code extension, or send them to Okahu for analysis across many runs.
Example trace (Okahu VS Code Extension)
The open_deep_research LangGraph workflow, with the researcher agent and its OpenAI-native web search captured as trace spans.
PS: if Monocle looks useful, a ⭐ helps the project (https://github.com/monocle2ai/monocle). And if you want to turn these traces into tests, that's the companion PR (#288).