To build an LLM agent, first define the task and how you will measure success. Then create an evaluation set, decide whether fine-tuning addresses a demonstrated problem, choose who runs the agent loop, connect tools and state, and deploy with monitoring for quality, reliability, latency, and cost. The examples below use OpenAI’s documentation; they describe one implementation path, not a universal choice for every model or workload.
1. Define the task and runtime before choosing a model
Write down what the agent must accomplish, what inputs it receives, what a successful result looks like, and which actions it is allowed to take. Be specific about where the agent may need tools, persistent state, human approval, or an execution environment. These decisions affect the agent’s design and deployment responsibilities, not just the model choice.
As an Amazon Associate I earn from qualifying purchases.
There is no single best agent architecture for every team. OpenAI’s documentation describes three integration paths: a managed Agents API, the Agents SDK running in your application, and direct Responses API integration. They differ in how much orchestration and runtime responsibility your application owns. See OpenAI’s Agents guide for its overview.
Choose who owns the agent loop
| Option | What it means for your application | Best fit when |
|---|---|---|
| Agents API | OpenAI provides a managed harness for the agent workflow. | You want a managed path and are comfortable with the controls and responsibilities it provides. |
| Agents SDK | The SDK runs in your application, giving your team an application-side runtime for orchestration and tools. | You want to implement the agent runtime within your own application and own its integration and operations. |
| Responses API directly | Your application integrates more directly with the API rather than relying on the SDK’s orchestration layer. | You want a more direct integration and are prepared to build the surrounding workflow. |
These are OpenAI-specific paths, not interchangeable labels for every provider’s offerings. Before choosing, decide how much control you need over tool implementation, state, approvals, storage, deployment, and runtime integration. The Agents SDK guide explains the SDK option and its application-side role.
2. Build evaluations before fine-tuning
Create a representative set of tasks and expected outcomes before changing the model. Include the kinds of inputs and edge cases the deployed agent is likely to encounter, then define how you will judge whether each result is acceptable. A useful evaluation lets you compare a baseline with later changes and detect regressions; without one, it is difficult to tell whether fine-tuning improved the behavior you care about.
OpenAI’s supervised fine-tuning guide puts this plainly: “Good evals first!” It recommends establishing reliable evaluations before investing in fine-tuning. Follow its guidance for preparing a dataset, uploading it, creating a fine-tuning job, and evaluating the resulting model: Supervised fine-tuning.
3. Fine-tune only when the evaluation supports it
Fine-tuning is a model-optimization step, not a substitute for a clear task, well-formed examples, or evaluation. First identify a repeatable behavior the model needs to learn, then make sure your examples demonstrate that behavior consistently. If you cannot describe the target behavior or assess it, adding training examples is unlikely to resolve the uncertainty.
How many examples should you prepare?
OpenAI’s current supervised fine-tuning guide says the minimum number of examples that can be provided is 10. It says OpenAI has seen improvements with 50–100 examples, while noting that the appropriate number varies by use case; the guide recommends starting with 50 well-crafted demonstrations and evaluating the results. Those figures are OpenAI’s guidance, not a promised outcome or a universal threshold for other providers.
Fine-tuning workflow
- Define the target behavior. Specify what a good response or action looks like for the task you evaluated.
- Prepare demonstrations. Build consistent examples that represent the behavior you want, using the dataset guidance in OpenAI’s fine-tuning guide.
- Check model eligibility. Confirm that the model you intend to use is currently eligible for fine-tuning and review the applicable limits; availability can change.
- Upload the dataset and create a job. Use the documented upload and job-creation workflow for the selected model.
- Evaluate the result against your baseline. Apply the same evaluation approach to determine whether the tuned model improves the intended task and is suitable for the agent.
4. Integrate tools, state, and approvals
An agent application combines a model with a runtime that handles the workflow around it. Decide which tools the model can call, how your application executes those calls, what information must persist between steps, and whether an action needs human approval. Match these responsibilities to the runtime choice rather than assuming the model alone supplies them.
For an SDK-based design, OpenAI describes the Agents SDK as an application-side runtime. Its documentation covers agents and their role in the SDK: OpenAI Agents SDK: Agents. If you choose a managed harness or direct API integration instead, account for the different division of orchestration and application responsibility described in OpenAI’s Agents guide.
- Tool execution: Decide how your application implements each tool and what happens when a call fails or returns an unusable result.
- State: Determine what context must persist across steps and where it is stored.
- Approvals: Identify actions that require a person to review or authorize them before execution.
- Execution environment: Choose where the agent and its tools run, and ensure that environment matches your operational and security requirements.
5. Deploy with checks for quality and operations
Before releasing the agent, run your evaluations against the deployed workflow—not just the model in isolation. Tool behavior, runtime integration, and state handling can all affect the result. OpenAI’s API deployment checklist highlights model choice, evaluation, tool calling, observability, reliability, latency, and cost as deployment considerations.
Recommended Free Tools
Pre-deployment checklist
- Model: Confirm that the selected model fits the task and that any fine-tuning eligibility still applies.
- Evaluation: Test representative inputs and review failures against the success criteria you defined.
- Tools: Verify that tool calls and their results behave as expected within the full agent workflow.
- Observability: Make the agent’s behavior visible enough for your team to investigate problems and assess operation.
- Reliability: Check how the application responds to errors in the model, tools, or runtime.
- Latency and cost: Measure these for your actual workload and decide what is acceptable before release.
Once deployed, continue measuring the same quality and operational signals. If behavior falls short, use the observed failures to decide whether the remedy belongs in the task definition, evaluation set, demonstrations, tool design, or runtime. The right trade-offs depend on the workload; the checklist is a way to make those decisions explicit, not a guarantee that one stack suits every application.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
