To trace agent runs in Microsoft Foundry, connect an Azure Monitor Application Insights resource to the Foundry project; tracing is off by default. The project’s agents can then emit OpenTelemetry traces to that resource, where you can inspect workflow steps, inputs and outputs, tool activity, latency, token use, and errors. Use dashboards to monitor aggregated operations and evaluation results; use trace evaluation to score interactions already captured in Application Insights without replaying their requests.
How Foundry tracing moves data
Tracing is an opt-in project capability. A project owner connects an Application Insights resource, after which agents in that project send traces to it. Disconnecting the resource stops new traces; it does not delete existing data. Previously collected traces remain subject to the connected Application Insights configuration, including its retention and sampling settings. See Microsoft’s tracing and data-handling guidance.
As an Amazon Associate I earn from qualifying purchases.
The path is: agent instrumentation emits OpenTelemetry telemetry, the configured Azure Monitor exporter sends it to Application Insights, and Foundry’s observability experience and Azure Monitor tools provide ways to inspect it. Application Insights retention and billing follow that resource’s configuration; additional Azure Monitor Application Insights charges may apply. The actual cost, retention period, and sampling behavior depend on the resource and its settings, so there is no universal figure to apply to every project.
Recommended Free Tools
Traces, spans, and attributes
A trace represents a request or workflow. Its spans represent individual operations and nest to express their relationships; attributes add context to a trace or span. Depending on the instrumentation, a multi-step agent workflow might include spans for invoke_agent, invoke_workflow, plan, or execute_tool, with attributes describing tool definitions, arguments, or results.
#1 Best Overall
This structure helps answer practical debugging questions such as “Where did this response come from?” and “Which step introduced an error or latency spike?”—questions raised in the Microsoft Foundry agent tracing overview. The exact span hierarchy and detail depend on the framework and instrumentation: a trace is only as useful as the operations and context the agent emits.
What the OpenTelemetry conventions do—and do not—guarantee
OpenTelemetry provides a common structure for telemetry across agents and tools, which can make cross-framework inspection and querying more practical. Microsoft’s documentation says the OpenTelemetry GenAI semantic conventions are in Development status and may change. Treat them as an evolving convention, not a final schema: downstream queries and evaluation workflows may need adjustment as instrumentation or conventions change.
Choose instrumentation for the agent you run
| Agent setup | Documented tracing approach | What to account for |
|---|---|---|
| Microsoft Agent Framework or Semantic Kernel agent running in a Foundry project | Microsoft documents native trace emission when project tracing is enabled. | Run the agent, then check Observability > Traces. In the documented setup, traces typically appear within 2–5 minutes; this is an expected delay, not a service-level guarantee. |
| External framework or agent hosted outside the Foundry project | Instrument with the documented OpenInference packages and Microsoft OpenTelemetry distro, then export to the project’s Application Insights resource. | Setup depends on framework and hosting. The cited LangChain and LangGraph instructions are Python-only. Hosted agent server packages can configure export and add project and agent identity to spans. |
For current framework-specific setup, use Microsoft’s agent-framework tracing guide. When comparing implementations, assess hosting, framework and language support, the usefulness of emitted agent/model/tool spans, data governance, and the evaluation workflow you plan to use. There is no universal best framework: the trade-off is between a documented native integration and the instrumentation, exporter, and identity configuration required by an external setup.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Monitoring dashboards and evaluation answer different questions
The Agent Monitoring Dashboard summarizes operations over a selected time range. Microsoft lists token usage, latency, run success rate, evaluation metrics, and red-team results among its metrics. It reads telemetry from the Application Insights resource connected to the project, so the resource’s retention and billing configuration also governs that data. The dashboard documentation marks the metrics view, recurring evaluations, and red-team scans as preview features. Check the live Foundry experience for current availability and limits before designing a production workflow. Details are in the Agent Monitoring Dashboard documentation.
Recurring or scheduled evaluation
The dashboard documentation describes recurring evaluations and red-team scans as preview. This is a configured recurring workflow for an agent, distinct from scoring a selection of interactions already recorded in Application Insights. Do not assume that preview status, limits, or configuration options are the same across tenants or remain unchanged.
Evaluation of captured traces
Trace evaluation applies evaluators to production interactions already stored in Application Insights; it does not replay their requests. The Foundry SDK documentation describes the azure_ai_traces data source, which can select traces by Application Insights operation_Id or discover recent traces with an agent filter. It recommends this path for non-Foundry agents when their OpenTelemetry spans use GenAI semantic conventions and reach Application Insights. Intelligent sampling can choose a representative subset to reduce evaluation cost while preserving trace variety, according to Microsoft’s documentation. This evaluation feature is marked preview. See the guide to evaluating deployed interactions.
These are complementary paths, not interchangeable labels for one feature: recurring evaluation is configured against an agent, while trace evaluation works from interactions already captured and selected from telemetry. Decide which outcome you need—ongoing scheduled checks or analysis of recorded runs—and verify the current preview availability before relying on either in a production process.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsPermissions depend on who or what performs each task
Foundry and Azure Monitor permissions are separate. A person inspecting log-based data needs access at the relevant resource or workspace scope; a project managed identity performing evaluation needs its own assignments. The roles listed in Microsoft’s evaluation permissions guide are:
Best Value
- Foundry User: for the project managed identity to create continuous or scheduled evaluation rules.
- Reader on the connected Application Insights resource: for the project managed identity to run trace evaluations or create trace datasets.
- Log Analytics Reader: for people who view log-based data, assigned at the relevant resource or workspace scope.
- Privileged Monitoring Data Reader: an additional role required to read protected trace tables, on top of ordinary read permissions.
Grant access to the identity and scope that match the task rather than treating Foundry project access as blanket permission to read Application Insights logs.
Handle traces as potentially sensitive production data
Depending on instrumentation and configuration, traces may contain user prompts, customer content, model and agent inputs and outputs, tool calls and results, intermediate steps, timestamps, latency, token usage, and errors. Tool arguments and returned results can be as sensitive as the original prompt. Microsoft advises minimizing or redacting sensitive information, keeping secrets and credentials out of telemetry, and applying access controls and retention policies as you would for production logs. See the data-handling guidance and tracing concepts.
- Review what each framework records before enabling project-wide collection.
- Redact or omit sensitive values from prompts, tool arguments, results, and span attributes where possible.
- Do not place credentials, API keys, or other secrets in telemetry.
- Set access and retention controls on the connected monitoring resource to fit the data it receives.
- Check sampling and retention configuration when interpreting whether a trace is present or how long it remains available.
Plan for evolving capabilities
Several parts of this workflow are not documented as mature, settled features: GenAI semantic conventions are marked Development, and the cited dashboard metrics, recurring evaluation, red-team scans, and trace evaluation documentation identifies preview capabilities. OpenTelemetry makes a common data path possible, but it does not make every framework emit identical spans or guarantee that a particular preview workflow is available in every environment. Validate the current Foundry experience, framework instructions, permissions, and Azure Monitor configuration against your own deployment before making operational commitments.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
