How to choose an LLM — a practical checklist
Choosing an LLM is a product decision with a spreadsheet attached. If you start from Twitter rankings, you will buy a flagship for a classification job. If you start from the cheapest API, you will ship a bot that can’t follow instructions. The fix is an explicit order of operations.
Write the job in one sentence, pick weights, shortlist three rows on the live leaderboard, compare them, then run two real prompts in the arena. Freeze a default and an escalate model. Revisit when prices or quality index move.
Popular frontier models by quality
Live OpenRouter pricing and Arena quality, cached about an hour. Not a static blog table.
| Model | Provider | Blend $ / 1M | Quality | Context |
|---|---|---|---|---|
| Claude Opus 5 | Anthropic | 15.00 | 93.4 | 1,000,000 |
| Claude Fable 5 | Anthropic | 30.00 | 91.1 | 1,000,000 |
| Qwen3.8 27B | Qwen | 1.71 | 89.0 | 1,000,000 |
| Gemini 3.7 Flash | 2.25 | 88.4 | 1,048,576 | |
| DeepSeek V4 Pro 0423 | DeepSeek | 1.14 | 88.2 | 1,024,000 |
1. Name the job and the constraint
“Customer support RAG, p95 latency 1.5s, data must stay in the EU” is a spec. “Make it smart” is not. Constraints (region, PII, on-prem) kill more candidates than Elo does. Apply them before you look at quality index.
2. Pick weights on purpose
The leaderboard sliders exist so you cannot pretend every dimension is #1. Chat UX: speed + cost. Legal research: quality + context. Batch extraction: cost. Share the URL so the rest of the team argues with the same ranking.
3. Shortlist three, not thirty
One cheap, one mid, one flagship — or one closed, one open, one specialist (vision, long context). More than four and nobody will eval them.
4. Compare fields, then outputs
Use /compare or a static pair page for price and context. Use the arena for text. If you skip outputs, you will pick the pretty spec sheet.
5. Route, don’t marry
Default + escalate beats a single flagship. Re-run this checklist when a new catalog row is cheaper at similar quality. Hourly data is wasted if your choice is annual.
FAQ
Should I always pick the #1 model?
- No. #1 on whose weights? Set yours. #1 quality is often the wrong default for production traffic.
How often should I revisit the choice?
- When the bill hurts, when quality tickets spike, or monthly — whichever comes first.
Do I need a data scientist?
- You need 20 labeled examples and the discipline to run them. This site handles the catalog.
What if legal picks the vendor first?
- Filter to that provider on the leaderboard and still pick the right SKU inside the family.
Related guides
Also try: LLM leaderboard, compare tool, live arena.