OpenLLMetry

What OpenLLMetry is

OpenLLMetry is an open-source project for monitoring and debugging how LLM applications execute, using non-intrusive tracing built on OpenTelemetry. It can send traces to Traceloop or an existing observability stack, with integrations including Datadog, Dynatrace, Honeycomb, New Relic, Sentry, Splunk, and OpenTelemetry Collector. Trace annotations can describe workflows, tasks, agents, and tools. SDK guides cover Python, JavaScript/TypeScript, Go, and Ruby; provider and framework support varies by language. Listed providers include OpenAI, Anthropic, Amazon Bedrock, and Google Gemini, while frameworks include LangChain, LlamaIndex, CrewAI, Haystack, and OpenAI Agents. It traces LLM, agent, and retrieval activity and tracks token costs. Prompts, completions, and embeddings are logged by default, but content logging can be disabled. Multi-modal traces can include images, audio, video, documents, and mixed content; visualization currently requires Traceloop. The Free Forever plan allows up to 50K spans monthly and five seats.

Who it suits

OpenLLMetry suits teams building LLM applications that need tracing across workflows, providers, or frameworks and want to export traces to an observability stack. Its self-hosting options may suit organizations with on-premise or restricted-environment requirements.

Limits

The Free Forever plan is capped at 50K spans per month, five seats, and 24 hours of data retention. Multi-modal visualization currently requires Traceloop, and media files over 10MB may be truncated or linked rather than embedded.

Getting started

1. Choose an SDK guide for Python, JavaScript/TypeScript, Go, or Ruby.
2. Add OpenLLMetry tracing to your LLM application.
3. Select Traceloop or an existing observability stack as the trace destination.
4. Use workflow annotations to describe workflows, tasks, agents, and tools.
5. Set TRACELOOP_TRACE_CONTENT=false if you want to disable content logging.

Verdict

OpenLLMetry combines tracing, token cost tracking, and integrations for LLM applications in a free, Apache-2.0-licensed project. Check the seat and retention limits, default content logging, and Traceloop requirement for multi-modal visualization against your needs.