Model Cascade for Cost and Latency Optimization
You have a Microsoft Foundry project that serves a high-volume chat app. Most requests are simple FAQs, but some require advanced reasoning. You need to reduce costs and latency for common queries, without degrading the quality of the responses to complex questions. What should you do?
Community Insight
Tests the ability to balance cost, latency, and quality using intelligent routing rather than uniform model selection.
Learn how to implement a model cascade in Microsoft Foundry to optimize costs and latency by routing simple queries to smaller models while reserving advanced models for complex tasks.
Candidates often choose option A (smaller model) or D (largest model), failing to recognize that a single model cannot satisfy both low-cost simple queries and high-quality complex reasoning requirements simultaneously.
Community Discussion (2 comments)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
A model cascade (Option B) is the industry-standard pattern for optimizing LLM inference costs and latency. By placing a small, fast, and inexpensive model at the front of the pipeline, simple requests like FAQs are resolved quickly without invoking expensive compute resources. Only when the smaller model determines it cannot handle the complexity (or confidence is low) does the request escalate to a larger, more capable model.Why the Other Options Are Wrong
Option A (Route all to a smaller model) fails because smaller models lack the reasoning capabilities required for complex questions, degrading response quality. Option D (Route all to the most capable model) ignores the need for cost reduction; sending every FAQ to an expensive model wastes significant resources. Option C (Increase max_tokens) merely extends output length but does not reduce computational cost or latency, potentially increasing them.Community Comment Notes
Community consensus strongly supports the model cascade approach. As user RICHARDALEX007 noted, "A model cascade sends simple FAQs to a small/cheap model and escalates complex queries," which directly addresses the dual requirement of reducing costs while maintaining quality. Another commenter emphasized that cascading leverages a "fast and cheap model first" to handle the bulk of traffic efficiently.Exam Strategy
When exam questions mention mixed workload types (e.g., simple vs. complex) with constraints on both cost and quality, look for architectural patterns like 'cascading,' 'routing,' or 'tiering' rather than uniform scaling strategies.
Frequently Asked Questions
What is a model cascade?
A model cascade chains multiple language models sequentially, using a small/cheap model for simple queries and escalating complex ones to larger models.
Why not just use the smallest model?
Smaller models may fail to provide accurate responses for complex reasoning tasks, leading to degraded quality despite lower costs.
Related Analysis
Practice All AI-103 Questions
Access 88 questions with complete answers and detailed explanations.
View Full AI-103 Practice Test →