VRUNAI

What VRUNAI is

VRUNAI evaluates LLM agents by examining their execution paths, tool calls, and final outcomes. Users can describe tools, mock data, conditional flows, and test scenarios in YAML without writing code. Repeating a scenario lets you score how consistently an agent follows the same path. VRUNAI also estimates scenario and provider costs from token usage, with model pricing and context-window statistics. Its listed providers include OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Ollama, and custom providers. The tool offers a terminal interface and a React web app, usable locally or through a hosted browser version. Local inference is available through Ollama and custom API endpoints, including OpenAI-compatible APIs. The CLI runs on your machine; web app keys are stored in browser localStorage and sent to configured provider APIs. VRUNAI is free, open source under AGPL-3.0, and has no usage limits or VRUNAI fees, though providers may charge for API calls. It is fully client-side, with no accounts, backend, or telemetry.

Who it suits

VRUNAI suits people evaluating LLM agents and comparing behavior across providers. It is also suited to users who want to define scenarios in YAML and track consistency and API costs.

Limits

VRUNAI has no fees or usage limits of its own. You still pay any configured LLM providers for their API calls, and the free plan is distributed under AGPL-3.0.

Getting started

1. Open the VRUNAI website or its GitHub project
2. Choose the terminal interface or React web app
3. Run the tool locally or try the hosted browser version
4. Define tools, mock data, flows, and scenarios in YAML
5. Configure a listed provider, Ollama, or a custom API endpoint
6. Run scenarios repeatedly and review paths, tool calls, outcomes, and costs

Verdict

VRUNAI combines agent scenario checks with consistency and cost tracking across a broad list of providers. Its free, client-side setup has no VRUNAI usage limits, but API calls can still incur provider charges.