Tiered LLM Routing on Azure OpenAI: Why Gentle Pills Runs Three Model Sizes, Not One

Every AI action in Gentle Pills — from extracting symptoms out of a consultation note to synthesizing a remedy comparison across multiple sources — runs on one of three Azure OpenAI deployment tiers. We call them fast, mid, and reasoning. The obvious way to assign tasks to tiers is by how hard the task looks. We didn't do that. We assign tiers by failure cost — what actually happens if the model gets it wrong — and the difference matters more than it sounds like it should.

Difficulty is the wrong axis

"Difficulty" is a proxy for how much reasoning a task seems to require, and it's tempting to route accordingly: simple-looking tasks get a cheap model, open-ended tasks get the expensive one. But difficulty says nothing about what happens downstream when the model is wrong. A task can look trivial and still be high-stakes, or look demanding and still be low-stakes, depending entirely on what catches the error afterward.

Gentle Pills never lets an AI output reach a patient record unreviewed. Every suggestion is surfaced to the doctor, who can edit, dismiss, or confirm it before anything is written. That single design constraint — human confirmation as the last gate, every time — is what makes failure cost, not difficulty, the correct routing variable. If a wrong answer gets caught before it matters, you can afford a faster, cheaper model. If a wrong answer could mislead clinical judgment before anyone catches it, you need the strongest model you have, regardless of how "hard" the task looks on paper.

The three tiers

fast is for deterministic-checkable extraction and classification against a fixed list — tasks where the output has a small, enumerable space of right answers and a doctor reviewing the result can immediately tell if it's wrong. structure_notes, which pulls candidate symptoms out of free-text case notes, runs here: the candidates are presented as a list, the doctor accepts or rejects each one, and nothing is saved without that confirmation. match_prognostic_observation is the same shape — matching a case against a fixed set of classical prognostic patterns. In both cases the review step isn't a safety net bolted on afterward; it's baked into the workflow, which is exactly why the cheapest, fastest model is the right choice.

mid is for grounded summarization and rubric mapping — tasks where the model is synthesizing across more material and the stakes are moderate. summarize_case, suggest_rubrics, remedy_relationship_narrative, and followup_summary all sit here. These outputs take more judgment to evaluate than a fast-tier list, so the model needs to be stronger, but they're still bounded by retrieved, doctor-visible source material rather than requiring open synthesis across conflicting evidence.

reasoning is reserved for open clinical judgment and multi-source evidence synthesis — the cases where the strongest model actually earns its cost. research_knowledge, compare_remedies, and isolate_miasmatic_layers run here. These are the tasks where a doctor is relying on the AI to have actually weighed something, not just extracted or summarized it, and where the output is harder to spot-check at a glance. That's where we spend the compute budget.

What's actually deployed

Today, fast maps to gpt-5.4-nano, mid maps to gpt-5.4-mini, and reasoning maps to gpt-5.4 — three separate Azure OpenAI deployments (gpt-5-4-nano, gpt-5-4-mini, gpt-5-4 in the Azure resource). A legacy gpt-4.1 deployment is kept configured as a fallback: if a tier's deployment isn't set in the environment, the system falls back to it rather than failing the request. That fallback exists for operational safety during model transitions, not as a long-term routing target.

A compatibility detail worth mentioning

GPT-5-and-later models on Azure's Chat Completions API require the max_completion_tokens parameter instead of the older max_tokens parameter — passing the legacy name to a GPT-5-class deployment gets rejected. This is exactly the kind of detail that's easy to get wrong once and then forget about, so it isn't hand-configured per deployment. The provider layer detects which parameter name a given model expects and sends the right one automatically. It's a small thing, but it's the difference between a tiered-routing system that works cleanly across model generations and one that silently breaks every time a new model family ships.

Why this is worth the complexity

Three deployments instead of one is more moving parts: three sets of capacity, three things that can be misconfigured, three places a fallback has to be correct. The payoff is that high-volume, low-stakes tasks — the kind that fire constantly as a doctor works through a case — run on the cheapest, fastest model available, while the compute budget for the strongest model goes specifically to the handful of tasks where clinical judgment quality actually changes the outcome. Latency drops where it's noticed most (the frequent small interactions), and reasoning quality concentrates where it's needed most (the infrequent, high-synthesis ones).

It's also worth being precise about what "capacity" means on Azure OpenAI here, since it's easy to assume it's a cost lever: each deployment's capacity is configured in thousands of tokens-per-minute, and it functions as a rate-limit ceiling, not a separate charge. Billing is purely per-token, consumed against whatever models actually get called. Tiered routing doesn't change what you pay per request — gpt-5.4-nano is cheap per token regardless of how you route to it — it changes *which* model handles *which* request, so the aggregate bill reflects the actual distribution of task stakes across a clinic's daily usage rather than a flat rate paid on every call regardless of whether it needed the strongest model in the fleet.