Vertex AI Endpoint Autoscaling Stalls Under Memory-Heavy Load
Short answer
D Decrease the CPU utilization target in the endpoint's autoscaling configuration so replicas are added before CPU-based scaling stalls under memory-heavy preprocessing.
Question
Key Takeaways
This tests Vertex AI's CPU-based autoscaling trigger: the trap is that a memory-bound workload looks idle to the CPU metric, so the fix is to lower the target rather than resize the machine.
Vertex AI endpoints autoscale on average CPU utilization, so memory-intensive preprocessing can keep CPU below the 60% default target and prevent scaling. Lowering the CPU utilization target (D) makes the endpoint add replicas sooner.
Common mistake: Option A (more memory) is the tempting fix because memory is the real bottleneck, but it changes the instance, not the scaling trigger — CPU-based autoscaling still won't fire under low CPU utilization.
Expert Analysis
The autoscaling signal
Vertex AI prediction endpoints scale replicas based on a CPU utilization target, not on memory usage. By default the target is 60% average CPU utilization across the endpoint's replicas, and a new replica is only provisioned when observed CPU climbs above that threshold. The endpoint in this scenario is deployed and already validated for latency, so the serving path itself works — the failure is in the scaling trigger.
Why memory hides the load
The model performs several memory-intensive preprocessing steps before predicting. Under concurrent requests, replicas can be heavily loaded on RAM while their CPU utilisation stays below the target, because the work is not CPU-bound. Traffic is uneven across replicas, so the average CPU reading stays low, the threshold is never crossed, and the autoscaler never adds capacity. This matches the question's core logic: the scaling metric (CPU) is misaligned with the actual bottleneck (memory), so the trigger never fires.
Tracing each option through the mechanism
- A. More memory per machine — enlarges the replica but leaves the scaling logic untouched. Autoscaling still keys off CPU, so a busy node can keep hogging memory while the endpoint stays under-scaled. This treats the symptom at the instance level, not the trigger.
- B. Fewer workers per machine — not a Vertex AI endpoint setting at all; endpoint replicas are provisioned per machine type, and this would reduce, not increase, capacity.
- C. Increase the CPU utilization target — moves the threshold up, so replicas must work even harder before a new one is added. This makes the under-scaling worse.
- D. Decrease the CPU utilization target — lowers the threshold, so the autoscaler reacts earlier and adds replicas while CPU is still moderate, compensating for the fact that CPU understates the real load. This is the correct action.
The takeaway: when the workload's bottleneck isn't the metric autoscaling watches, adjust the target so scaling fires sooner.
Exam Strategy
When an autoscaling question names a resource bottleneck, first identify which metric the service actually scales on — for Vertex AI endpoints it is CPU utilization, defaulting to 60%. Then ask whether the workload pushes that metric past the threshold; if not, the answer is to adjust the threshold, not the instance size. Eliminate options that are not real endpoint settings (workers per machine) and any option that moves the threshold in the wrong direction.
Frequently Asked Questions
What metric does Vertex AI use to autoscale endpoint replicas?
Why doesn't memory-intensive preprocessing trigger endpoint autoscaling?
Should I pick a bigger machine type when the endpoint won't autoscale?
Community Votes
83% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 8 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 8
-
b1a8fae 👍 11 Answered DD. The idea behind this question is getting autoscaling to handle well the fluctuating input of requests. Changing the machine (A) is not related to autoscaling, and you might not be using the full potential of the machine during the whole time, bur rather only during instances of peak traffic. You need to lower the autoscaling threshold (the target utilization metric mentioned in the options is CPU, so we will go with this) so you make use of more resources whenever too many memory-intensive requests are happening. https://cloud.google.com/compute/docs/autoscaler/scaling-cpu#scaling_based_on_cpu_utilization https://cloud.google.com/compute/docs/autoscaler#autoscaling_policy
-
pikachu007 👍 6 Answered AB. Decreasing Workers: This might reduce memory usage per machine but could also decrease overall throughput, potentially impacting performance. C. Increasing CPU Utilization Target: This wouldn't directly address the memory bottleneck and could trigger unnecessary scaling based on CPU usage, not memory requirements. D. Decreasing CPU Utilization Target: This could lead to premature scaling, potentially increasing costs without addressing the root cause.
-
guilhermebutzke 👍 3 Answered DOption D, "Decrease the CPU utilization target in the autoscaling configurations," could be a valid approach to address the issue of autoscaling and anticipate spikes in traffic. By lowering the threshold, the autoscaling system would initiate scaling actions at a lower CPU utilization level, allowing for a more proactive response to increasing demands.
-
fitri001 👍 3 Answered DD. Decrease the CPU utilization target: This is the most suitable approach. By lowering the CPU utilization target, the endpoint will scale up at a lower CPU usage level. This increases the likelihood of scaling up when the memory-intensive preprocessing tasks cause a rise in CPU utilization, even though memory is the root cause.
-
fitri001 👍 2A. Use a machine type with more memory: While this might seem logical, autoscaling in Vertex AI endpoints relies on CPU utilization as the metric, not directly on memory usage. Even with more memory, the endpoint might not scale up if CPU utilization remains below the threshold. B. Decrease the number of workers per machine (Not applicable to Vertex AI Endpoints): This option might be relevant for some serving frameworks, but Vertex AI Endpoints don't typically use a worker concept. Scaling down workers wouldn't directly address the memory bottleneck. C. Increase the CPU utilization target: This would instruct the endpoint to scale up only when CPU usage reaches a higher threshold. Since the issue is memory usage, increasing the CPU target wouldn't trigger scaling when memory is the limiting factor.
Show 3 more comments
-
b1a8fae 👍 2Addition: although memory-intensive is not directly related to CPU, for me the key is "the model does not autoscale as expected". To me this is addressing directly the settings of autoscaling, which won't change by changing the machine.
-
VinaoSilva 👍 2 Answered D"use autoscale" = deacrease cpu utilization target
-
taylz876 👍 1 Answered DD- Endpoints autoscale on CPU utilization by default, with a target of 60%. A new replica is added only when average CPU use goes above the target. This model's preprocessing is memory-intensive, not CPU-intensive, so a busy node can be struggling with memory while its CPU stays below 60%. Autoscaling never triggers. Lowering the CPU target makes the endpoint scale out at a lower CPU level, so extra replicas appear sooner as requests pile up. Today this is an Agent Platform endpoint (formerly Vertex AI). Why the others are wrong • A: A bigger machine with more memory helps each node, but it doesn't change when autoscaling triggers. The endpoint would still not scale out. • B: Workers decide how much of a node's resources get used, so fewer workers means even less CPU use, and scaling triggers even less often. • C: Raising the target means the endpoint waits for even higher CPU before it scales, which makes the problem worse.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All PMLE Questions
Access 65 questions with complete answers and detailed explanations.
View Full PMLE Practice Test →