# AI Model Routing: When to Use Local Models and Cloud APIs > **Published**: 16 November 2026 | **Reading Time**: 3 min read | **Author**: Mohammad Shadikur Rahman > **Tags**: AI Infrastructure, Model Routing, LLM Ops, Self-Hosted AI > **Canonical Article**: https://shadikur.com/blog/ai-model-routing-local-cloud-infrastructure --- ## Abstract / Excerpt A practical guide to routing AI workloads across self-hosted models and cloud APIs using data rules, latency budgets and safe fallbacks. --- ## Article Content A voice agent and a document summarizer can both use a language model, but they should not necessarily use the same one. A caller needs a quick response. A background summary can wait. And some requests contain data that should stay inside an approved environment. That is the problem an AI model router solves: it decides where a request can run, then picks a model that can handle the task within its time and cost budget. For a team running both self-hosted models and cloud APIs, that decision is part of the infrastructure. ## Decide where the data can go first Start with rules the application can enforce. For each request, identify the workflow, the data it is allowed to send, the required capabilities, and the deadline. Use those facts to narrow down the allowed routes before considering model quality or price. If a request is marked local-only, a cloud fallback must not become available just because the GPU is busy. Return a clear failure, hand off to a person, or defer the work if the workflow allows it. A fallback is only useful if it preserves the original constraints. ## Route by workload, not by model brand Consider three requests in the same customer support system: | Request | Starting route | If it can't finish | | --- | --- | --- | | Classify a routine call | Small approved model with a short deadline | Ask a clarifying question or hand off | | Summarize an approved call transcript | Background queue with an approved model | Retry later within a capped budget | | Analyze a restricted internal document | Approved self-hosted endpoint | Defer or report that the endpoint is unavailable | The same distinction applies to tools. If the model is drafting a CRM update, it can retry generation when a request fails. If a tool may already have committed that update, check what happened before retrying the action. Otherwise a model timeout can turn into a duplicate record. ## Measure the delay the user actually feels Time to first token measures how long the client waits before receiving the first generated token. It helps diagnose the model path, but for a voice agent the user's wait spans more than inference. Measure from the end of the caller's utterance to the first audible response, then break that delay into speech recognition, routing and queue time, model generation, and speech synthesis. A useful starting dashboard tracks p50 and p95 response time by workflow and route, queue wait, timeouts, fallback reasons, and how often the output meets the task's acceptance criteria. Track quality with examples that reflect the actual workload: noisy calls, ambiguous requests, and outputs that must follow a schema. When tracing requests, record the route and timing without collecting raw prompts or transcripts by default. ## Roll out on one workflow first Start with one workflow and one known route. Build a small evaluation set from representative examples you are permitted to use. Route a limited share of requests to a second approved model and compare quality, latency, and cost under realistic concurrency. Test what happens when the endpoint is slow or unavailable, then keep a way to revert to the previous route. For supporting metrics, see the [vLLM performance guide](https://docs.vllm.ai/projects/spyre/en/v2.4.0/user_guide/performance.html). For the broader voice pipeline, see [Building Low-Latency Voice AI Agents](https://shadikur.com/blog/building-low-latency-voice-ai-agents-webrtc-livekit). A practical first question for any team hosting AI models is: **If the preferred model is unavailable, what is this workflow allowed to do next?** Answer that before adding a second model. --- ## Author Biography Mohammad Shadikur Rahman is a Senior Infrastructure Architect and Full Stack Engineer based in Hamburg, Germany. Specializing in AWS CDK, Cloudflare R2, DevOps, microservices, and VoIP platforms. - **Blog Index**: https://shadikur.com/blog.txt - **LLM Index**: https://shadikur.com/llms.txt