Driver FixRecommendedSound, Wi-Fi or graphics acting up? Check drivers firstFind missing or outdated drivers fast.Check DriversOctober DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run Scan×
Skip to content
World desk7 min

Tracing AI Agent Tool Calls With OpenTelemetry: What to Capture in Production

A production trace for an AI agent turn needs invoke_agent, model, and execute_tool spans with metadata on every operation. Message bodies and tool payloads stay out unless an explicit, controlled debugging policy allows them.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

In production, record one trace for each user-facing agent turn. The root is an invoke_agent span, each model request is a child inference span, and each tool the agent runs is an execute_tool span. Even with payload capture turned off, every span should carry an operation name, a stable tool name, a duration, a status, and an error type when the operation fails. Model identity, finish reason, and token usage belong on the spans too, but only where your SDK or provider supplies them. Prompts, message bodies, tool arguments, and tool results stay out of default traces. They belong only in a narrow, opt-in debugging path with its own access and retention rules.

Build the trace around the agent turn

Treat the agent invocation as the unit the user experiences. OpenTelemetry’s GenAI semantic conventions describe an invoke_agent operation for agent invocation and recommend execute_tool spans for tool execution. Model inference gets its own span. Parent-child relationships connect them, so you can see which model request preceded a tool call, how long each step took, and where an error or retry occurred.

As an Amazon Associate I earn from qualifying purchases.

The OpenTelemetry project’s walkthrough, published May 14, 2026 and credited to James Newton-King of Microsoft, uses this layout: an invoke_agent parent with child chat spans and execute_tool spans. A single turn that calls one tool and then answers looks like this:

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
invoke_agent support-agent
├── chat                      (model request 1: decides to call a tool)
├── execute_tool lookup_order
│   └── HTTP GET              (downstream call, if HTTP instrumentation exists)
└── chat                      (model request 2: writes the final answer)

Span names and span kinds

Operation Span name pattern Span kind What the span covers
Agent invocation, running locally invoke_agent {agent-name} INTERNAL The agent run inside your process, from start to final response
Agent invocation, remote client call invoke_agent {agent-name} CLIENT Your code calling an agent hosted elsewhere
Model inference Not specified by the conventions; the OpenTelemetry walkthrough names these chat Not stated in the conventions The logical model request, from request through response, including automatic retries that belong to the same call
Tool execution execute_tool {tool-name} INTERNAL The logical tool operation, from the span boundary through the result

Keep span names low-cardinality. A name built from a user prompt, tool arguments, or a unique request value creates an unbounded set of names that is hard to query and aggregate. Use the agent name and tool name, which change only when you change your code.

Attributes every tool span should carry

For each tool execution, prioritize the following. Include them in roughly this order if you have to cut.

  • The operation name execute_tool and a stable tool name in gen_ai.tool.name.
  • A tool call ID in gen_ai.tool.call.id when the framework provides one. It links the call the model requested to the execution that followed.
  • The tool type and the agent name, when available.
  • Duration from the span boundary, the span status, and an error type when the call fails.
  • Child HTTP, RPC, database, or messaging spans for the actual downstream work, wherever instrumentation exists.

OpenTelemetry recommends following the execute-tool convention for application-owned tools that automatic instrumentation does not reliably cover. Do not add a second span for an operation that existing instrumentation already captures reliably, or every tool call will appear twice in the trace.

Metadata to capture by default

Metadata is the baseline you can keep in production without a content policy decision. Capture these fields where your stack can supply them:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Operation name
  • Tool name and tool type
  • Agent identity
  • Model and provider identity
  • Response finish reason
  • Token usage, as reported by the provider
  • Duration
  • Failure classification

With these fields you can answer practical questions. Which tool is slow or failing? Is the agent looping between model and tool calls? Which model served a given turn? Does token use rise alongside latency? The walkthrough’s example shows model identity, input and output token counts, and finish reasons on spans. That is an illustration of what is possible, not a guarantee. SDKs and frameworks differ in which fields they emit, so check the actual exported spans before you build a dashboard on a particular field.

Keep content out of default traces

OpenTelemetry flags input and output messages, system instructions, retrieval text, tool arguments, and tool results as potentially sensitive. The walkthrough’s example defaults to metadata rather than prompt content for two reasons: content can contain sensitive data, and content fields are large and awkward to render in a trace viewer.

A three-tier content policy

Tier What falls in it Controls
Never collected Raw prompts, system instructions, and full tool arguments and results for ordinary production turns Disabled in instrumentation configuration; confirm in staging that exported spans contain none of these fields
Controlled environments Full messages and tool payloads in development and staging, ideally with synthetic or sanitized data A separate telemetry pipeline, restricted access, and short retention
Scoped production sample Content for a specific debugging workflow, such as one named tool or a time-boxed incident investigation An explicit opt-in flag, filtering or truncation, access limited to named responders, a defined retention period, and redaction before export

If you enable content capture

Turn it on deliberately and document the decision. The controls to set up before the first sample is collected are these:

  • An explicit opt-in setting, not a default that someone flips for convenience.
  • Filtering or truncation, because oversized content fields degrade the trace backend and slow queries.
  • Access restricted to the people who need it, reviewed on a schedule.
  • A retention period chosen on purpose, not inherited from the backend default.
  • Redaction before data leaves your process or collector.

Moving a raw payload into a span event or a log record does not make it safer. It is still telemetry and needs the same data governance as the span attributes it sits beside.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Where spans, events, and metrics each belong

Use spans for operations that have a duration and meaningful boundaries: agent invocation, model inference, and tool execution. Put properties of the whole operation in span attributes, especially those you need at span start for sampling decisions. Use events for timestamped occurrences inside a longer operation, such as a retry, a fallback, a state transition, or a checkpoint. Events earn their place when each occurrence needs its own timestamp or its own fields. OpenTelemetry’s event conventions separate named occurrences from operation-wide attributes and recommend structured, queryable event attributes.

Aggregate metrics to pair with traces

Traces explain individual turns; metrics show trends. A practical dashboard tracks:

  • Agent-turn latency
  • Model-call latency
  • Tool latency and error rate, labeled by a bounded tool name
  • Failures grouped by error type
  • Token usage, where the provider supplies it

These are implementation suggestions. Instrument names and availability depend on your SDK and framework. Keep raw prompts, full tool arguments, and arbitrary conversation identifiers out of metric labels. They are both sensitive and unbounded, which makes them expensive and risky as dimensions. Use trace IDs to navigate from a metric spike to a specific trace instead.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Errors, retries, and coverage gaps

Record failures where they happen

Record each failure on the operation that failed, and classify it with a stable error.type where possible. Distinguish transport failures, such as a timeout or a refused connection, from a tool’s domain-level failure. A protocol can return a technically successful response that reports the tool did not accomplish its task, and the two need different handling.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Preserve the causal path from the agent operation through the model request and the tool execution to the downstream dependency. Record retries or fallbacks as events, or as child operations when they carry their own diagnostic value. The GenAI conventions defer status semantics to OpenTelemetry’s error-recording guidance, so follow the SDK guidance for your language. Then test two cases: a forced transport error and a forced domain-level error. Each should appear on the execute_tool span with a different error.type.

Know which tools are not traced

MCP tool executions may already be covered by MCP instrumentation. Manually instrument the application-owned tools that automatic instrumentation does not reliably cover. Keep an instrumentation inventory that lists each tool, whether it is covered, and by what. A missing tool span looks exactly like an agent that did nothing, so the inventory is the only reliable way to tell a gap from inactivity.

Rolling out to production

Roll out in three stages, and move to the next only when the previous one is verified:

  1. Metadata-only collection. Confirm the trace topology: an invoke_agent parent with chat and execute_tool children. Confirm tool names are stable and that failures appear on the right spans.
  2. Collector and exporter review. Validate security, retention, access policy, and telemetry volume. OpenTelemetry’s security guidance warns that telemetry may include PII, application data, or network patterns, and recommends protecting it against disclosure, tampering, and denial-of-service.
  3. Scoped payload capture, only if the debugging value is clear. Apply the controls described above before collecting any content.

Plan for change. The GenAI semantic conventions are still marked Development in the OpenTelemetry repository. Pin your SDK, framework, and semantic-convention versions, document the attributes you emit, and review schema changes during every upgrade. Do not assume every framework or exporter implements the conventions in the same way. On the backend side, the walkthrough notes that any OTLP-compatible backend can receive GenAI telemetry, so the trace design does not lock you into a single vendor.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.