Model Cascade for Cost and Latency Optimization
Short answer
B Use a model cascade that routes the requests to different models.
Question
Most requests are simple FAQs, but some require advanced reasoning.
You need to reduce costs and latency for common queries, without degrading the quality of the responses to complex questions.
What should you do?
Key Takeaways
Tests the ability to balance cost, latency, and quality using intelligent routing rather than uniform model selection.
Learn how to implement a model cascade in Microsoft Foundry to optimize costs and latency by routing simple queries to smaller models while reserving advanced models for complex tasks.
Common mistake: Candidates often choose option A (smaller model) or D (largest model), failing to recognize that a single model cannot satisfy both low-cost simple queries and high-quality complex reasoning requirements simultaneously.
Expert Analysis
Why the Answer Is Correct
A model cascade (Option B) is the industry-standard pattern for optimizing LLM inference costs and latency. By placing a small, fast, and inexpensive model at the front of the pipeline, simple requests like FAQs are resolved quickly without invoking expensive compute resources. Only when the smaller model determines it cannot handle the complexity (or confidence is low) does the request escalate to a larger, more capable model.
Why the Other Options Are Wrong
Option A (Route all to a smaller model) fails because smaller models lack the reasoning capabilities required for complex questions, degrading response quality. Option D (Route all to the most capable model) ignores the need for cost reduction; sending every FAQ to an expensive model wastes significant resources. Option C (Increase max_tokens) merely extends output length but does not reduce computational cost or latency, potentially increasing them.
Community Comment Notes
Community consensus strongly supports the model cascade approach. As user RICHARDALEX007 noted, "A model cascade sends simple FAQs to a small/cheap model and escalates complex queries," which directly addresses the dual requirement of reducing costs while maintaining quality. Another commenter emphasized that cascading leverages a "fast and cheap model first" to handle the bulk of traffic efficiently.
Exam Strategy
When exam questions mention mixed workload types (e.g., simple vs. complex) with constraints on both cost and quality, look for architectural patterns like 'cascading,' 'routing,' or 'tiering' rather than uniform scaling strategies.
Frequently Asked Questions
What is a model cascade?
Why not just use the smallest model?
Discussion 2 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 2
-
RICHARDALEX007 👍 2 Answered BA model cascade sends simple FAQs to a small/cheap model and escalates complex queries to a capable model, reducing cost/latency without degrading hard-question quality. Routing everything to a small model (A) hurts complex answers; routing all to the largest model (D) keeps costs high; raising max_tokens (C) does not reduce cost or latency.
-
shank23 👍 1 Answered BA Model Cascade (also known as LLM Cascading) is an optimization and reliability pattern where an AI system chains multiple language models together sequentially. Instead of sending every single user query to a highly intelligent, expensive model, the system leverages a fast and cheap model first
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All AI-103 Questions
Access 88 questions with complete answers and detailed explanations.
View Full AI-103 Practice Test →