Deploy the model behind an Azure Machine Learning (Azure ML) managed online endpoint, then decide whether a Foundry agent should use it as its conversational model or call it as a tool. Those are separate integration paths: Foundry’s documented bring-your-own-model (BYOM) connection expects an OpenAI-compatible chat-completions API, while a task-specific Azure ML scoring endpoint is generally better exposed as a tool or called through custom agent code.
Should the custom model be the agent’s model or a tool?
Make this decision before wiring services together. It determines the API contract Foundry must reach and how the agent uses the model’s output.
As an Amazon Associate I earn from qualifying purchases.
| Decision | Connect as the agent’s model | Expose as a tool |
|---|---|---|
| Purpose | Generate the agent’s conversational responses. | Return a specialized prediction or task result for the agent to use. |
| API shape | The documented BYOM route expects OpenAI-compatible chat completions. | Can retain a task-specific request and response contract when exposed through a suitable tool or custom code. |
| Foundry integration | AI gateway and connected-model setup. | OpenAPI, function, or MCP tool; alternatively, implement the call in hosted agent code. |
| Best fit | A custom model that behaves like a chat model. | A classifier, scorer, ranker, or other callable service. |
| Primary check | Chat API compatibility, base URL, authentication, and gateway routing. | Tool schema, endpoint authentication and reachability, and handling of the returned result. |
The term “model” can be confusing here. In the first route, the custom model supplies the agent’s conversational model. In the second, the agent retains its own conversational model and calls the Azure ML endpoint for a specialized operation.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How do I deploy a custom model to an Azure ML online endpoint?
In Azure ML, an online endpoint is the serving endpoint clients call; a deployment beneath it holds the resources that run inference. As Microsoft Learn puts it, “A deployment is a set of resources required for hosting the model that does the actual inferencing.” The deployed behavior also depends on the scoring code or container, not just the model artifact. See Microsoft’s managed online endpoint guidance.
#1 Best Overall
1. Define the serving contract and prepare assets
- Specify the request fields the caller must send, the response it should receive, and how the service reports invalid input or inference errors.
- Prepare the model artifact and the inference environment. Your scoring code or container must accept the intended request shape and return a useful response.
- For production reuse and traceability, Microsoft recommends registering model and environment assets. Azure ML deployments can use local or registered assets.
A REST endpoint is not automatically a chat endpoint. A scoring script that accepts, for example, task-specific fields does not acquire chat-completions behavior merely because it is reachable over HTTP.
2. Create the endpoint and an inference deployment
- Create a managed online endpoint in an Azure region where the service is available for your requirements. Endpoint names must be region-unique.
- Choose the endpoint’s authentication mode and confirm that the eventual client has the required permissions.
- Create at least one deployment under the endpoint. Configure the model, environment, and scoring behavior that make up the inference service.
- Choose an inference implementation: use Azure ML’s supported inference and scoring configuration, or supply a compatible custom container. Microsoft’s custom-container guidance describes serving stacks such as TensorFlow Serving, TorchServe, and Triton.
For details on the deployment flow and supported configuration, consult Microsoft’s managed online endpoint documentation and custom-container deployment guide. These sources cover Azure ML CLI v2 and Python SDK v2 workflows; follow the current instructions for the interface and package versions you use.
3. Verify the endpoint before connecting Foundry
Test locally where practical, deploy to Azure, invoke the endpoint with representative requests, and inspect logs. Confirm that successful responses match the agreed schema and that invalid requests and inference failures are handled in a way a client can interpret. Azure ML’s guidance also covers monitoring the deployed endpoint.
Recommended Free Tools
How do I connect my Azure ML endpoint to a Foundry agent?
First compare the endpoint’s actual API with the role chosen above. Foundry’s documented BYOM path is not a plug-and-play connection for every Azure ML scoring endpoint: it expects an OpenAI-compatible chat-completions API behind an AI gateway. Microsoft describes the service this way: “Foundry Agent Service allows you to connect and use models hosted behind your AI gateways such as Azure API Management or other non-Azure managed AI model gateways.” See the Foundry BYOM guidance.
If the custom model should generate the agent’s responses
- Expose the model through an AI gateway with the OpenAI-compatible chat-completions interface expected by the documented BYOM flow.
- Create an admin-connected model connection with the gateway’s base URL and the authentication method it requires.
- Add the model through that connection, then create a prompt agent that uses the connected model.
Check both the interface and gateway routing before treating this route as viable. If the Azure ML endpoint has a task-specific scoring contract, add an adapter or gateway that implements the supported chat API; do not assume Foundry will translate arbitrary scoring requests into chat completions. The BYOM documentation uses “BYOM” for third-party models brought to Foundry, not for Foundry Models sold by Azure.
If the custom model should perform a specialized task
Keep the endpoint’s task-specific contract and make the operation available to the agent as a tool. Foundry documents custom tools based on functions, OpenAPI specifications, and MCP servers, as well as code-based hosted agents. The tool schema or agent code describes how to call the endpoint; the agent can decide when to call it and interpret the result. This is a tool integration, not a claim that the endpoint replaces the agent’s conversational model. See Microsoft Foundry Agent Service overview.
What should I check before choosing the route?
- API compatibility: Does the model need to answer chat-completions requests, or does it expose a task-specific prediction contract?
- Authentication: Can the gateway, tool, or hosted agent authenticate to the endpoint using an approved identity and permission model?
- Network reachability: Can the component that makes the call reach the endpoint over the intended network path?
- Governance: Do you need gateway controls such as load balancing, throttling or rate limiting, and governance? The BYOM guidance describes Azure API Management as one gateway option.
- Latency and role: Does the extra gateway or tool call suit the interaction, and is the custom model meant to converse or return a bounded result?
For managed online endpoints, Microsoft lists key-based, Azure ML token, and Microsoft Entra token authentication. Its guidance identifies Microsoft Entra token authentication as the most secure option for production managed online endpoints. Confirm the exact mode, client permissions, and gateway or agent authentication requirements for your design.
How do I test the complete Azure ML-to-agent route?
- Invoke the Azure ML endpoint directly with representative inputs and verify its request and response schemas.
- Test the agent’s call path separately: check the tool schema or chat API routing, endpoint reachability, and authentication headers or tokens.
- Exercise error cases, including invalid input, authentication failures, and endpoint errors; verify that the agent handles returned errors rather than treating them as successful predictions.
- Run representative agent turns and verify that the model is used in the intended role: as the source of conversational responses or as a callable specialist.
For gateway-backed model connections, use the authentication method and header expectations of the gateway topology you selected. The BYOM guide discusses API key and OAuth 2.0 options, with API-key header expectations that can differ by topology.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
