Azure AI Foundry Model Router: Controlling Cost and Latency Across Model Selection

Generative AI has moved rapidly from experimentation to production within many Australian organisations, and with that shift comes a harder set of operational questions. Once a service handles real workloads, leaders must account for the cost of every inference call, the latency experienced by end users, and the governance obligations attached to the data being processed. Choosing a single large model for every task tends to be expensive and slow, while choosing a small model for everything sacrifices quality on the prompts that genuinely need it.

The model router in Azure AI Foundry addresses this tension directly by treating model selection as a runtime decision rather than a fixed architectural choice. Instead of hard-coding one model into an application, teams can send prompts to the router and let it decide which underlying model should answer each request. This article explains how the capability works, how it helps control spend and latency, and how it fits within the sovereignty and guardrail requirements that government and enterprise clients cannot compromise on.

How the model router works

The model router is deployed as a single endpoint that sits in front of a set of chat completion models available in Azure AI Foundry. When an application sends a prompt, the router evaluates characteristics of that request and selects an appropriate model from the underlying pool to generate the response. Simpler prompts can be directed to smaller, faster, and less costly models, while more demanding prompts that require deeper reasoning are directed to more capable models.

From the perspective of the calling application, the interface remains consistent because the router exposes a familiar chat completions contract. Developers integrate against one deployment rather than managing conditional logic that inspects each prompt and picks a model manually. This separation of concerns matters in production because it allows the routing strategy to evolve without requiring changes to application code, and it centralises a decision that would otherwise be scattered across many services.

Controlling cost and latency

Inference cost is driven largely by the size and capability of the model handling a request, so routing lower-complexity prompts to smaller models can meaningfully reduce spend across a high-volume workload. Many enterprise scenarios involve a long tail of straightforward requests, such as short summaries, classification, or simple question answering, that do not warrant the most powerful model available. By reserving larger models for the prompts that actually benefit from them, organisations can protect response quality where it counts while avoiding paying premium rates for routine work.

Latency follows a similar pattern, because smaller models generally return responses faster than larger ones. Directing suitable requests to lighter models improves the experience for end users who expect prompt replies, particularly in interactive applications such as service desks or internal knowledge assistants. It is important to test routing behaviour against representative traffic, however, since the value of the approach depends on the mix of prompts a given workload actually receives, and teams should monitor both cost and quality metrics rather than assuming savings will materialise automatically.

Enforcing sovereignty and spend guardrails

For government and regulated enterprise clients, where a model runs is often as important as how well it performs. Azure AI Foundry deployments are provisioned within specific Azure regions, and Australian organisations can select Australian regions to keep data processing within the country to support data residency requirements. The model router itself operates within the region in which it is deployed, so establishing the correct regional placement at the outset is a prerequisite before any routing logic is considered.

Spend guardrails should be layered around the router rather than treated as an afterthought. Azure provides mechanisms such as budgets and cost alerts through Cost Management, quota and rate limiting on model deployments, and content filtering through Azure AI Content Safety, all of which can be combined to constrain both financial and safety risk. Applying role based access control, private networking, and diagnostic logging around the endpoint gives security and governance teams the visibility they need to demonstrate that generative AI is operating within approved boundaries. These controls are native to the platform, which allows them to be governed through the same policies and tooling that already protect other Azure workloads.

Operationalising the approach

Adopting the model router is best treated as an iterative programme rather than a single deployment task. Teams should begin by establishing a baseline with clear measurements of cost per request, latency, and answer quality against their existing approach, so that any change can be evaluated on evidence. From there, the underlying model pool and routing configuration can be tuned to reflect the actual distribution of prompts, with regular review as usage patterns and model options change over time.

Success also depends on strong observability and clear ownership. Capturing which model handled each request, alongside token usage and outcome quality, allows organisations to refine their strategy and to explain their decisions to auditors and stakeholders. When combined with disciplined region selection, access controls, and budget enforcement, the model router becomes a practical means of operationalising generative AI in a way that respects both the financial and the governance realities faced by Australian government and enterprise organisations.

References

• Azure AI Foundry documentation
• Microsoft Cost Management and Billing documentation
• Azure global infrastructure and geographies