Generative AI/Level 1 · AI Foundations

Choosing the Right Model

Open vs closed, size vs latency vs cost, and how to pick a model (or a cascade) for a given workload.

Intermediate 35m interview 15m read Medium frequency Popularity 76

LLM Fundamentals Model Serving Databricks Microsoft Amazon

Introduction

Choosing the right model is one of the highest-leverage decisions in a generative AI system. The model determines quality, latency, cost, privacy posture, context capacity, modality support, and how much engineering you need around the model to make the product reliable.

In interviews, strong candidates do not answer this question with a favorite vendor or a leaderboard rank. They turn it into a decision framework: define the task, measure the actual workload, compare candidate models on representative evaluations, and choose the cheapest model that clears the quality and risk bar.

This lesson gives you that framework. You will learn the practical axes for model selection, how to compare frontier, small, reasoning, and specialized models, when open-weight hosting makes sense, and how routing, caching, batching, and cascades reduce cost without blindly sacrificing quality.

The goal is to make model choice explicit and testable. By the end, you should be able to defend a model selection in a design interview with concrete tradeoffs, token math, and an evaluation plan instead of vibes.

Where this shows up in production

Model selection appears in almost every production AI roadmap.

A support chatbot may start with a small fast model and escalate difficult conversations to a frontier model.
A coding assistant may use a specialized code model for completion, an embedding model for repository search, and a stronger reasoning model for multi-file planning.
A regulated enterprise assistant may prefer open weights in a private region because data residency matters more than quick integration.
A high-volume summarization pipeline may save more money through batching, prompt caching, and shorter outputs than through another prompt rewrite.
A multimodal product may choose a model because it supports vision or audio even if a text-only benchmark ranks another model higher.

Learning Objectives

Compare candidate models across quality, latency, token cost, context window, modalities, tool support, privacy, licensing, and fine-tuning options.
Distinguish frontier, small, reasoning, code, embedding, and other specialized model tiers.
Explain when reasoning models justify extra latency and cost, and when a standard model is enough.
Evaluate open-weight self-hosting against proprietary managed APIs using control, privacy, scale economics, reliability, and operations burden.
Design routing and cascade strategies that use cheaper models first and escalate only when needed.
Estimate monthly cost from request volume, input tokens, output tokens, and per-million-token prices.

Theory & Concepts

Model choice is a measured product decision

A model is not chosen in isolation. It is chosen for a task, a user experience, a risk tolerance, and a budget. The same model that is excellent for open-ended legal drafting may be wasteful for sentiment classification, and the model that is fast enough for autocomplete may fail on complex planning.

The interview signal is structure. Start with the job to be done, define what good output means, identify unacceptable failures, and only then compare models. This prevents a common mistake: choosing the largest or newest model before proving that the workload needs it.

Capability, latency, and cost form the first tradeoff triangle

The first screen is usually quality, latency, and cost. Capability covers accuracy, reasoning depth, instruction following, robustness, multilingual performance, and domain knowledge. Latency covers time to first token and time to complete the answer. Cost depends on both input and output token prices, because many providers charge different rates for reading tokens and generating tokens.

For many products, the right answer is not the best model. It is the cheapest and fastest model that reliably clears the quality threshold. If the task is narrow, a small model may win. If the task has high user value or high failure cost, a frontier or reasoning model may be justified.

Context, modality, tools, and output constraints define fit

A model must support the shape of the product. Long-document analysis needs a sufficient context window and strong long-context behavior, not just a large advertised limit. Image understanding needs vision input. Voice agents need audio support or a separate speech pipeline. Agentic workflows need reliable tool calling, function schemas, and structured output.

Structured-output support is especially important in production. If the downstream system expects valid JSON, a model that supports schemas, constrained decoding, or function calling can be safer than a model that scores slightly higher on general chat quality but frequently returns malformed fields.

Privacy, residency, licensing, and hosting change the answer

The best technical model may not be deployable in your environment. Some workloads require data to stay in a specific region, never leave a virtual private cloud, or avoid vendor retention. Some teams need open weights for auditability, offline operation, customization, or cost control at high scale.

Licensing also matters. Open weights do not automatically mean unrestricted commercial use, and proprietary APIs do not automatically satisfy every compliance requirement. A strong design answer names these constraints early, because they can eliminate options before any quality comparison begins.

General, reasoning, and specialized models have different jobs

Standard chat models are optimized for broad instruction following with good latency. Reasoning models spend more computation on difficult multi-step problems, which can improve math, planning, coding analysis, and tool-use decisions, but they usually cost more and respond more slowly. Specialized models focus on a narrower function such as code generation, embeddings, reranking, moderation, speech, or image understanding.

Production systems often combine several model types. A retrieval product may use an embedding model to find documents, a reranker to order passages, a standard model for normal answers, and a reasoning model only for ambiguous or high-stakes questions.

Request Flow

1
1. Define the task and failure cost
Write down the user task, output format, acceptable error rate, and impact of a bad answer. A low-stakes draft can tolerate different failures than a medical triage assistant or financial compliance tool. This step sets the quality bar before any vendor comparison.
2
2. Estimate workload shape
Measure expected requests per day, input tokens per request, output tokens per request, peak concurrency, context length, modality mix, and latency target. These numbers determine whether token cost, GPU capacity, or responsiveness will dominate the design.
3
3. List hard constraints
Filter candidates by requirements that are not negotiable: region, privacy policy, licensing, open-weight availability, supported languages, context window, vision or audio support, tool calling, structured outputs, fine-tuning, and service-level expectations.
4
4. Build a representative evaluation set
Create examples from real traffic or realistic synthetic cases. Include common requests, edge cases, adversarial prompts, long-context examples, tool-calling cases, and examples where the correct answer is intentionally uncertain. The eval set should reflect the product, not a generic leaderboard.
5
5. Benchmark quality, latency, and cost together
Run each candidate model on the same eval set and capture quality scores, refusal quality, format validity, time to first token, total latency, input tokens, output tokens, and estimated cost. A model that wins quality but doubles latency may not be a product win.
6
6. Try a routing or cascade policy
Before defaulting every request to the strongest model, test a cheap-first policy. Use a small model for easy cases, confidence checks or heuristics to detect uncertainty, and a stronger model for escalation. Measure the combined quality and cost of the system, not just each model alone.
7
7. Decide, launch behind monitoring, and revisit
Choose the model or route that clears the quality bar at the best cost and latency. Pin model versions when possible, monitor production drift, sample outputs for review, and rerun evaluations when providers release new models or pricing changes.

Deep Dive

Frontier, small, and specialized model tiers

Frontier models are the largest managed models with the broadest capability. They are best for hard reasoning, complex writing, ambiguous user intent, and high-value tasks where errors are expensive. Their drawbacks are higher latency, higher token cost, and sometimes stricter rate limits.

Small and fast models are ideal for classification, extraction, simple summarization, routing, rewriting, and low-latency chat. Specialized models should be used when the product needs a specific representation or skill: embeddings for retrieval, rerankers for search quality, code models for developer workflows, moderation models for safety, and multimodal models for image or audio inputs.

Reasoning models versus standard models

Reasoning models are worth considering when the task requires multi-step planning, careful math, difficult code analysis, constrained decision making, or tool-use sequencing where a shallow answer fails. They are often not worth it for short extraction, style rewriting, simple Q&A over retrieved context, or classification where a standard model already meets the bar.

The practical test is incremental value. Compare a reasoning model against a standard model on the hardest slice of your eval set. If quality improves only slightly while latency and cost rise sharply, route only the hard slice to the reasoning model or keep it as an escalation path.

Open-weight self-hosting versus proprietary managed APIs

Open-weight models can give stronger control over data flow, region, deployment topology, customization, and cost at steady high volume. They also let teams inspect weights, fine-tune locally, quantize, run offline, and avoid dependence on a single external API. The tradeoff is operational burden: GPU provisioning, serving stack tuning, autoscaling, reliability, security patching, monitoring, and model upgrades.

Managed APIs are usually faster to adopt and easier to operate. They provide hosted inference, model updates, high availability, safety tooling, and scaling without GPU operations. The tradeoff is less control over weights, pricing, availability, data handling choices, and sometimes version stability. Many mature teams use a hybrid: managed APIs for frontier quality and open weights for private or high-volume narrow workloads.

Routing, cascades, and distillation

A cascade sends work to a cheap model first and escalates only when the answer is low confidence, high risk, long-context, user-visible, or fails validation. A model router can use rules, a classifier, embeddings, or a learned policy to select the model by task type and difficulty. The benefit is cost control without forcing every request through the most expensive model.

Distillation is a related optimization. A strong teacher model generates labels, rationales, or target outputs that train a smaller student model for a narrow task. Distillation can reduce serving cost dramatically, but it requires evaluation discipline because the student may copy teacher mistakes or fail outside the distilled distribution.

Evaluation beyond leaderboards

Leaderboards are useful for discovery, but they are not a model-selection process. They may overrepresent tasks unlike your product, hide latency and cost, or include contaminated benchmark examples that appeared in training data. A high rank does not prove the model will follow your schema, handle your domain, or satisfy your safety constraints.

Build a representative eval set and score what matters: factuality, task success, hallucination rate, refusal quality, citation behavior, format validity, tool-call correctness, latency, token usage, and user preference. Keep a holdout set and periodically refresh examples so the model choice remains grounded in current product reality.

Token cost math and cost reducers

Monthly model cost is roughly requests per day times days per month times input tokens per request times input price, plus the same calculation for output tokens at the output price. Output tokens often cost more because generation is sequential and consumes more inference work than reading cached prompt tokens.

Common cost reducers are shorter prompts, retrieval instead of huge context stuffing, prompt caching for repeated prefixes, batching offline jobs, shorter max output lengths, streaming for perceived latency, cheaper models for easy tasks, and precomputing embeddings. Always measure cost after the full prompt template and tool context are included, not just the user message.

Production Considerations

Versioning and reproducibility

Pin model versions when the provider supports it and record prompt templates, tool schemas, sampling parameters, evaluation scores, and routing rules together. Model upgrades can change behavior even when the API name looks similar, so production systems need release notes, staged rollout, and rollback.

Observability by model route

Track quality signals, latency, token usage, cost, retries, tool-call errors, schema failures, refusals, and escalation rate separately for each model route. Without route-level metrics, a cascade can silently become expensive or a small model can absorb tasks it should have escalated.

Fallbacks and graceful degradation

Plan for rate limits, provider outages, regional failures, and model-specific regressions. Fallbacks can include a second provider, a smaller local model, cached answers, read-only mode, or escalating to human review. The fallback should preserve safety even if quality drops.

Privacy and data governance

Classify data before sending it to a model. Sensitive workloads may require redaction, tenant isolation, private networking, regional endpoints, no-retention settings, audit logs, or self-hosted open weights. The governance decision should be part of model selection, not a patch added after launch.

Interview Perspective

What interviewers look for

✓A clear framework that starts from task requirements and failure cost instead of naming a favorite model.
✓Ability to compare quality, latency, input and output token cost, context, modality, tool support, privacy, licensing, and hosting tradeoffs.
✓Understanding that routing and cascades can beat a single-model design on cost and latency.
✓A concrete evaluation plan with representative examples, not vibes or generic benchmark rank alone.

Alternative designs

Single strong default model

Use one frontier or reasoning model for every request. This is simplest and can be appropriate during an early launch or for low-volume high-value products, but it often wastes money on easy tasks and can create avoidable latency.

Task-specific model portfolio

Use different models for classification, embeddings, reranking, generation, reasoning, vision, and moderation. This improves cost and quality for mature systems, but it requires routing logic, evaluation per route, version management, and careful monitoring.

Likely follow-up questions

When would you pay for a frontier or reasoning model?

I would pay for it when the task has high value or high failure cost, requires complex multi-step reasoning, needs strong instruction following across varied inputs, or fails evaluation with smaller models. I would still measure the hard slice separately and consider routing only that slice to the expensive model rather than using it for all traffic.

How would you choose between an open-weight model and a managed API?

I would compare hard constraints first: data residency, privacy, licensing, offline needs, customization, and vendor policy. Then I would compare scale economics and operations. Open weights can win for private, customized, high-volume workloads if the team can operate GPUs reliably. Managed APIs usually win for speed to market, frontier quality, reliability, and no infrastructure burden.

How do you avoid choosing a model based on benchmark vibes?

I would build a representative eval set from product tasks, include edge cases and failure modes, score outputs with human review or calibrated graders, and record latency and token cost. Benchmarks can shortlist candidates, but the decision should come from task-specific quality, format reliability, safety, cost, and latency on the actual workload.

Common mistakes

×Choosing the newest or largest model without measuring the actual task.
×Ignoring output token price and estimating cost from input tokens only.
×Using an advertised context limit as proof of long-context quality.
×Forgetting privacy, data residency, licensing, and operations constraints until after the model is selected.
×Treating a benchmark leaderboard as a deployment evaluation.

Visual Learning

Model selection axes

Axis	What to measure	When it matters	Common trap
Capability and quality	Task success, factuality, instruction following, robustness	Open-ended, high-value, ambiguous, or safety-sensitive tasks	Using a generic leaderboard as the only quality signal
Latency	Time to first token, total completion time, tail latency	Chat, autocomplete, voice, and interactive agent workflows	Only measuring average latency instead of p95
Cost per token	Input price, output price, cache discounts, average tokens	High-volume products and long-answer workflows	Forgetting that output tokens may cost more than input tokens
Context window	Usable context length, retrieval need, long-context accuracy	Long documents, multi-turn memory, codebase analysis	Assuming a bigger window means better use of all tokens
Modality	Text, vision, audio, image generation, speech support	Products that must read screenshots, documents, voice, or media	Forcing multimodal tasks through a text-only pipeline
Tool and schema support	Function calling accuracy, structured output validity, retry rate	Agents, workflows, data extraction, and API orchestration	Accepting free-form text when downstream systems need strict fields
Customization	Fine-tuning support, adapters, prompt caching, distillation options	Domain style, private labels, or repeated narrow tasks	Fine-tuning before proving prompting and retrieval are insufficient
Privacy and licensing	Retention policy, region, open-weight license, auditability	Regulated, enterprise, offline, or private-data workloads	Treating all providers and open licenses as equivalent

Model tier tradeoffs

Tier	Strength	Weakness	Use when
Frontier or large general model	Best broad capability, strong instruction following, flexible reasoning	Highest cost, higher latency, external dependency if API-hosted	Failure is expensive or the task is broad and hard
Small fast model	Low latency, low cost, easy to route at scale	Weaker on complex reasoning and ambiguous instructions	The task is narrow, repetitive, or latency-sensitive
Reasoning model	Better multi-step planning, math, code analysis, and tool sequencing	Extra latency and cost, sometimes overkill for simple tasks	Hard eval cases fail with standard models
Specialized code model	Strong code completion, repair, and repository-aware behavior	May be less useful for general conversation or policy-heavy tasks	The product is developer-facing or code-heavy
Embedding or reranking model	Efficient semantic retrieval and ranking	Does not generate final natural-language answers	You need search, RAG, deduplication, clustering, or recommendations
Moderation or safety model	Focused policy classification and risk detection	Not a substitute for generation quality	You need scalable safety checks before or after generation

Open-weight versus managed API

Option	Advantages	Risks	Best fit
Open-weight self-hosted	Control over data path, region, quantization, fine-tuning, and scale economics	GPU operations, reliability, autoscaling, security patches, and model upgrade burden	Private or high-volume workloads with strong infrastructure support
Proprietary managed API	Fast integration, frontier quality, managed reliability, no GPU operations	Vendor dependency, external data path, pricing changes, and less weight-level control	Rapid launch, broad capability, and teams without model-serving operations
Hybrid portfolio	Use managed frontier models for hard tasks and open weights for private or cheap routes	More evaluation, routing, observability, and governance complexity	Mature products with mixed privacy, quality, and cost requirements

Decision guide

Practical model-selection checklist

1. Define the job

What user task does the model perform?
What output format is required?
What failures are unacceptable?
Is the task interactive, batch, agentic, or offline?

2. Filter by hard constraints

Context: enough usable context for the longest realistic input.
Modality: text, vision, audio, speech, or image support as needed.
Tools: reliable function calling, structured outputs, and schema adherence.
Customization: fine-tuning, distillation, adapters, or prompt caching if required.
Governance: privacy, retention, residency, audit logs, and license terms.

3. Run task-specific evals

Use a representative eval set, not vibes. Include normal traffic, edge cases, long-context inputs, adversarial prompts, tool calls, and examples where the right answer is to say uncertainty. Score quality, hallucination rate, format validity, latency, and token usage.

4. Choose the cheapest passing route

If a small model passes, use it.
If only hard cases need more capability, route easy cases to a cheap model and escalate hard cases.
If latency matters more than peak quality, prefer smaller or regional models.
If privacy or residency is strict, consider open weights or private managed deployments.
If the task is high-value and failure is expensive, pay for frontier or reasoning capability.

5. Revisit continuously

Model choice is not permanent. Re-run evals when traffic changes, prompts change, pricing changes, or providers release new versions. Keep a holdout set so improvements are real and not overfit to your examples.

Hands-on Examples

Estimate monthly model cost from token prices

This small Python helper estimates monthly cost from requests per day, average input tokens, average output tokens, and separate per-million-token prices. Use real measured token counts from logs whenever possible, because prompt templates, retrieved context, and tool results often dominate the user message.

monthly_model_cost.py

def estimate_monthly_cost(
    requests_per_day,
    input_tokens_per_request,
    output_tokens_per_request,
    input_price_per_million,
    output_price_per_million,
    days_per_month=30
):
    monthly_input_tokens = requests_per_day * input_tokens_per_request * days_per_month
    monthly_output_tokens = requests_per_day * output_tokens_per_request * days_per_month

    input_cost = monthly_input_tokens / 1000000 * input_price_per_million
    output_cost = monthly_output_tokens / 1000000 * output_price_per_million
    total_cost = input_cost + output_cost

    return {
        "monthly_input_tokens": monthly_input_tokens,
        "monthly_output_tokens": monthly_output_tokens,
        "input_cost_usd": round(input_cost, 2),
        "output_cost_usd": round(output_cost, 2),
        "total_cost_usd": round(total_cost, 2),
    }

baseline = estimate_monthly_cost(
    requests_per_day=50000,
    input_tokens_per_request=1200,
    output_tokens_per_request=400,
    input_price_per_million=2.50,
    output_price_per_million=10.00
)

shorter_prompt = estimate_monthly_cost(
    requests_per_day=50000,
    input_tokens_per_request=800,
    output_tokens_per_request=300,
    input_price_per_million=2.50,
    output_price_per_million=10.00
)

print("Baseline estimate:")
print(baseline)
print("After shorter prompts and answers:")
print(shorter_prompt)

Estimate savings from a cheap-first cascade

A cascade can lower average cost when most requests are easy. The important metric is blended cost and blended quality: if the small model handles 80 percent of requests and escalation catches most hard cases, the product can feel strong while spending far less than a frontier-only baseline.

cascade_cost.py

def blended_request_cost(cheap_cost, strong_cost, escalation_rate):
    cheap_share = 1 - escalation_rate
    return cheap_share * cheap_cost + escalation_rate * strong_cost

cheap_model_cost = 0.002
strong_model_cost = 0.030

for escalation_rate in [0.05, 0.15, 0.30, 0.50]:
    cost = blended_request_cost(cheap_model_cost, strong_model_cost, escalation_rate)
    print("Escalation rate:", escalation_rate)
    print("Blended cost per request USD:", round(cost, 4))

Quiz

0/6 answered

1.What is the best first step when choosing a model for a production feature?
2.Why should input and output token prices be estimated separately?
3.When is a reasoning model most likely worth its extra latency and cost?
4.What is the main operational downside of self-hosting an open-weight model?
5.What is a model cascade?
6.Why should you be cautious about choosing a model from leaderboard rank alone?

Flashcards

Cheat Sheet