Scaling Deep Learning Endpoints Using GPU Metrics
Short answer
A Update the autoscaling configuration to scale based on the GPU duty cycle metric.
Question
Key Takeaways
The question tests the ability to identify the true resource bottleneck (GPU vs. CPU) and configure autoscaling policies accordingly.
To resolve high latency in a GPU-accelerated Agent Platform endpoint, update autoscaling to monitor GPU duty cycle instead of CPU utilization, ensuring replicas scale when the actual bottleneck is reached.
Common mistake: Choosing Option B (lowering CPU target) is a common trap; it assumes CPU is the limiting factor, ignoring the explicit evidence that GPU utilization is saturated while CPU remains low.
Expert Analysis
The scenario describes a classic resource bottleneck mismatch in cloud autoscaling. The endpoint uses NVIDIA GPUs, yet the autoscaling policy is triggered by CPU utilization set at 60%. During load testing, CPU stays below 40%, so no new replicas are spawned. Meanwhile, GPU utilization exceeds 90%, indicating that the compute capacity for inference is exhausted, causing latency to spike.
Option A is correct because it addresses the actual constraint. In GPU-accelerated serving, "GPU duty cycle" (or similar metrics like GPU memory utilization) measures how busy the accelerator is. By switching the target metric to GPU duty cycle, the autoscaler will detect the high load on the accelerators and scale out the number of replicas, distributing the inference requests and reducing latency. This aligns with best practices for scaling deep learning models where the GPU, not the CPU, is the primary resource consumer.
Option B is incorrect because lowering the CPU target would trigger scaling based on a metric that is already underutilized. Since the bottleneck is the GPU, adding more CPU-idle replicas would not help and would waste money. Option C suggests changing hardware to TPUs, which is an unnecessary architectural change when the existing NVIDIA GPU infrastructure can be scaled effectively with the right metrics. Option D proposes increasing the minimum replica count to match peak load. This violates the principle of elastic scaling; it provisions resources permanently rather than scaling dynamically based on demand, leading to significant cost inefficiency during off-peak times.
Exam Strategy
When analyzing autoscaling scenarios, always look for the discrepancy between the configured metric and the observed bottleneck. If one resource is saturated while another is idle, the current metric is misaligned. Adjust the target metric to reflect the constrained resource to enable efficient, cost-effective scaling.
Frequently Asked Questions
What is GPU duty cycle?
Why not just increase the number of replicas manually?
Community Votes
100% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 2 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 2
-
taylz876 👍 1 Answered ASimilar to Q238, Q218 and Q256 Why A is right • The GPU is the bottleneck, not the CPU. CPU stays below 40% while the GPU is over 90%, so the endpoint never reaches the 60% CPU target and never adds replicas, while predictions get slower. Scaling on GPU duty cycle (the fraction of time the GPU is actively working) makes the endpoint add replicas when the GPU is busy. Why the others are wrong • B: Lowering the CPU target to 20% still tracks the wrong metric. It could add replicas when the CPU isn't the issue, and it still doesn't follow the GPU load. • C: A TPU is a different, expensive accelerator, and it doesn't fix the scaling setup. • D: A high fixed minimum keeps peak capacity running all the time, so it's costly and it doesn't scale to actual load.
-
n2183712847 👍 1 Answered ACorrect answer: A. Because the GPU is the actual bottleneck while CPU utilization remains low, configuring autoscaling based on the GPU duty cycle allows replicas to scale when GPU demand increases; B would not trigger scaling because CPU is already below the threshold, C unnecessarily changes accelerator types, and D wastes resources by permanently provisioning replicas for peak demand instead of autoscaling.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All PMLE Questions
Access 65 questions with complete answers and detailed explanations.
View Full PMLE Practice Test →