Free tools Windows power users keep installed
One-click scans. No signup required.
To track AI agents across a SaaS, instrument each run from its starting workflow through model calls, tool use, handoffs, and failures; attach stable tenant and workflow identifiers; capture usage from provider responses; and send the resulting traces and usage records to a queryable store. Then build customer-level views from your own identifiers and reconcile estimated or reported costs with provider accounting. Traces show what happened; they do not automatically become reliable customer billing records.
What to capture: activity, usage, and customer context
Think of observability as three related but distinct records. An activity trace explains the path of a run. Usage data records consumption at model-call boundaries. Application context identifies the customer and workflow that should receive the operational or financial attribution.
As an Amazon Associate I earn from qualifying purchases.
Activity: the run and its component events
A useful trace links the end-to-end workflow to individual model calls, tools, delegated agents, handoffs, and failures. Include timestamps or durations, status, and enough input/output detail to diagnose a run, subject to your data-handling policy. OpenAI’s Agents API organizes traces into sessions, turns, and spans; its trace view can show recorded inputs and outputs, duration, status, and tool-call details. Its live session event stream and dashboard logs can also expose activity such as turns, tools, and subagents. OpenAI Agents API tracing
Recommended Free Tools
Usage: provider and model consumption
Activity is not a substitute for usage. At each provider call, capture the provider and model, request count, input and output usage, and any cached-token, reasoning-token, or modality-specific fields the response exposes. Preserve per-call records as well as run totals when available: that makes retries, nested agents, and later reconciliation easier to inspect. The OpenAI Agents SDK’s run usage aggregates requests and token totals across model calls, including calls that lead to tools or handoffs. OpenAI Agents SDK usage
#1 Best Overall
- WIFI ENABLED TO CONTROL FROM ANYWHERE – Transform your home into a smart home with the Feit Electric Smart Wi-Fi Plug. Remotely turn on or off lights, fans, coffee makers, or other home appliances from your smartphone or tablet. Works seamlessly with Alexa and Google Home, giving you effortless voice control without needing a separate hub. Manage your devices anytime, whether you’re at home, at work, or traveling.
- SIMPLE SETUP, NO HUB REQUIRED – Enjoy the convenience of smart home automation without extra equipment. The plug connects directly to your 2.4 GHz Wi-Fi network, making installation fast and easy. Plug it in, download the Feit Electric app, follow the simple steps, and your devices are instantly connected. Perfect for beginners or anyone looking to expand their smart home ecosystem with minimal hassle.
- SET YOUR ROUTINE & SAVE ENERGY – Save energy, stay organized, and automate daily routines with customizable schedules and timers. Set your lamps, heaters, or appliances to turn on and off automatically at specific times, ensuring your home is always comfortable and efficient. Ideal for morning routines, evening wind-downs, or holiday lighting, giving you peace of mind and energy savings without constant manual operation.
- ENHANCED SAFETY & CONVENIENCE – Protect your home and appliances with the Feit Electric Smart Plug’s durable design and safety features. Its compact size fits easily into standard indoor outlets without blocking other sockets. With real-time app control and notifications, you can monitor appliance activity and prevent energy waste. Ideal for families, pet owners, or anyone seeking a smarter, safer, and more convenient home setup.
- RELIABLE 2.4GHz WI-FI PERFORMANCE – Designed to work exclusively on 2.4 GHz networks, this smart plug provides stable connectivity for smooth operation of all your devices. Avoid interruptions caused by incompatible networks, ensuring your appliances respond instantly when controlled via the app or voice commands. Perfect for indoor home use, it supports up to 15 amps, handling heavy-duty appliances safely and reliably.
Context: your SaaS identifiers
Attach stable identifiers such as tenant or customer ID, user ID where appropriate, environment, workflow, and agent to the trace or associated usage record. Use IDs that remain stable, not mutable display names, as join keys. Observability products can filter or group data by available user, application, or tag fields, but those filters do not establish that your application has attached the correct tenant identity. You own that mapping and its validation.
Choose an instrumentation route
The right starting point depends on whether you need visibility mainly within one agent stack or want a shared telemetry pipeline. The options below describe documented capabilities, not an independent ranking.
Rank #2
- equipped with atom n2600 d2700 processor, compatible with many freebsd based router systems, linux distros, or win.os supported, easy configuration and management
- Please note, this is a barebone only. A system memory, a storage drive and an operating system are needed to complete this system
- 13-19 inches 1u, 50w power, with power cord, make sure to use a big brand memory and ssd/hdd with quality assurance
- Designed with console, 2 x usb, 4 x lan, vga, power switch, size at 290 x 180 x 44mm
- There are 2 inside reserved fans on chassis, which could be removed freely or be turned on in a high temperature environment to ensure the best function of the product
| Approach | Best fit | What it can provide | Check before choosing |
|---|---|---|---|
| Provider- or framework-native tracing | A stack centered on one provider or agent SDK | Low-friction capture of that framework’s events and usage fields. OpenAI’s Agents SDK documents built-in tracing and run-level usage aggregation. | Coverage of non-native tools and providers, export options, retention and policy fit, and whether the data is available when you need it. |
| OpenTelemetry-based instrumentation | A team with an existing or portable telemetry pipeline | Span-based export and integration options; Langfuse documents OpenTelemetry instrumentation, and LangSmith describes OpenTelemetry support. | Which semantic fields survive export, backend compatibility, cardinality and cost, and how model usage is attached. |
| Dedicated LLM or agent observability service | A team that wants trace exploration, usage and cost dashboards, and debugging or evaluation workflows in a product UI | Langfuse documents generation-level usage and cost reporting, dashboards, alerts, and metrics queries. LangSmith describes dashboards for usage, latency, errors, cost breakdowns, and feedback. | Data region and retention, self-hosting needs, pricing maintenance, access controls, and current plan terms. |
Compare options on framework coverage, per-call usage fidelity, tenant aggregation, export and portability, data residency and retention, cost methodology, query and alert features, and integration effort. Product documentation establishes feature claims, not comparative performance.
Native SDKs and callbacks
Framework callbacks or provider SDK tracing can get you started quickly when most activity lives in one stack. OpenAI says its Agents SDK tracing records events such as generations, tool calls, handoffs, guardrails, and custom events. OpenAI Agents SDK tracing Check whether events from external tools, non-native model providers, and your own orchestration code are also represented.
Rank #3
- Shelly Plus 1 PM is a Wi-Fi smart relay switch with 1 channel, up to 16A with power metering that can be used also as a WiFi repeater and Bluetooth gateway. Shelly Plus 1PM can be used to monitor the consumption and take control of home appliances, electric circuits, and office equipment individually.
- Automate electrical appliance and control - With Shelly Plus 1PM you can automate any electrical appliance in your home and control it remotely. Shelly Plus 1PM can control appliances with a large load which makes it perfect for kitchen appliances and domestic systems monitoring and control. You can get precise measurements of the power consumption of each appliance and switch in on/off remotely, no matter where you are.
- Set and be prepared for everything - Reveal the full potential of Shelly Plus 1PM by combining it with other devices from your home network! Set Shelly Plus 1PM to activate custom scenes based on hour, light, or various occurrences. For example, you can set Shelly Door/Window sensor to report a porch door opening and activate Shelly Plus 1PM to turn on the hot tub heaters only in the hours after 8 pm.
- Shelly Customer Service - Shelly is one of the fastest-growing Smart Home brands in the world with devices, providing solutions for the automation of private homes, buildings and businesses. We provide our customers with professional support and a 3 years device warranty.
- Shelly Smart Control App will help you control your Shelly devices remotely and will send notifications for all automated events in your home. You can easily configure devices and manage their settings individually, or you can create personalized scenes by combining Shelly devices to trigger certain actions in your home automation.
OpenTelemetry and observability products
OpenTelemetry can make sense when your organization already exports spans through a shared pipeline or wants integration flexibility. Langfuse’s current setup documentation covers SDK and framework integrations as well as OpenTelemetry; LangSmith describes support for named frameworks, custom implementations, and OpenTelemetry. These are vendor-documented capabilities, so verify the fields, export behavior, and backend fit with your actual instrumentation.
Langfuse documents usage and cost by generation, dashboards, alerts, Metrics API queries, ingested values, and inferred costs based on model definitions. Its documentation also describes custom model definitions. Langfuse token and cost tracking LangSmith describes custom dashboards for token usage, P50/P99 latency, errors, cost breakdowns, and feedback. LangSmith observability
Rank #4
- Portable 100M/1G Network TAP Appliance for remote capture of data traffic
- Integrated with a Raspberry Pi 4 module (8GB RAM and 64GB Micro SD Card)
- Can be used as a standalone 100M/1G network TAP with the external monitor port
- Dual DC power inputs for enhancing overall system availability
Implement tracking from the run outward
- Define the questions. Separate operational questions—what happened, where a run failed, and how long it took—from usage and cost questions—what model consumed which units, for which tenant and workflow.
- Instrument the full run. Use framework tracing or create spans around the top-level workflow, model requests, tools, handoffs, and meaningful custom events. Preserve root and parent-child identifiers across asynchronous work and delegation so events can be reconstructed as one run.
- Attach business context. Put tenant/customer and workflow identifiers on the trace or associated usage record. Add user, environment, and agent identifiers where useful and appropriate. Define which identifier is authoritative for billing and how it maps to your tenant records.
- Capture usage at each call boundary. Record provider, model, request and response identifiers, and the usage fields returned. Keep per-request detail in addition to aggregates if your SDK exposes it; a run total alone can conceal retries or nested calls.
- Compute and reconcile cost. Use provider-reported cost when available. Otherwise, calculate an estimate from a versioned price table that matches provider, model, relevant region, and unit type. Label inferred values as estimates, update model definitions as rates change, and reconcile them with provider statements before using them for customer billing.
- Build customer views and alerts. Start with token or other usage volume and spend by tenant, model, workflow, and time. Add latency, failures, and thresholds that prompt investigation. Langfuse documents dashboards, alerts, and metrics queries; LangSmith describes dashboard views for usage, latency, errors, cost, and feedback.
- Test accounting edge cases. Exercise failed and cancelled runs, retries, tool calls, delegated agents, streaming responses, and other billable requests such as compaction where applicable. Define how unknown usage is handled. OpenAI documentation notes that usage may be null or unknown and can change as accounting arrives; missing usage must not silently become zero.
- Review data handling before rollout. Inspect what prompts, outputs, and tool data are captured; configure redaction or sampling as needed; and check access controls, retention, export permissions, and regional requirements.
Turn telemetry into dependable customer accounting
A trace backend is an operational system, not automatically your billing ledger. To build defensible customer-level accounting, retain or export usage records with your own tenant key and enough call-level detail to explain the total. Establish how duplicate events, retries, cancellations, unknown usage, and late-arriving usage are treated. Keep the price-table version used for any inferred cost, so an estimate can be reproduced after rates change.
Use dashboards to find unusual consumption and operational problems, but make your billing process explicit about whether it uses provider-reported charges or application-calculated estimates. Reconcile against provider accounting before charging customers, and avoid treating a trace’s visible token total as a final financial amount unless your accounting rules and data source support that interpretation.
Protect prompts, traces, and exported data
Agent traces can contain sensitive prompts, model outputs, tool arguments, and results. Review the wrapper or instrumentation behavior rather than assuming that only metadata is sent. Langfuse’s setup documentation lists EU, US, Japan, and HIPAA endpoint examples; endpoint availability alone does not establish contractual suitability for a particular data type. Confirm the actual service, region, retention, access and export controls, and applicable policies before sending production data. Langfuse getting started
OpenAI documents that Agents SDK tracing is unavailable for organizations using OpenAI APIs under a Zero Data Retention policy. Separately, exporting Agents API session traces returns paginated OTLP JSON, requires trace export to be enabled and a key with appropriate read permission, and does not itself configure automatic delivery of future traces. OpenAI Agents API session trace export OpenAI Agents API trace data retention
Quick Recap
Common implementation failures to avoid
- Joining by display name: names change and can collide; use stable internal identifiers for customer and workflow attribution.
- Counting only model calls: tools, handoffs, retries, and failures are part of the run’s activity and may affect its cost or diagnosis.
- Keeping only aggregate totals: retain call-level usage where possible so nested work and retries can be traced.
- Equating missing usage with zero: preserve an unknown state and reconcile when accounting data becomes available.
- Presenting inferred cost as exact: identify estimates and their price-table basis, then reconcile against provider records.
- Assuming a tracing product knows your tenants: attach and validate your own business context before grouping or billing.
- Sending payloads without review: verify what instrumentation captures and where it is stored before enabling it on production traffic.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
