Managing Cold Start Latency in Low-Traffic Vertex AI Endpoints

PMLE By islever Editorial Last updated

Short answer

B Turn off auto-scaling for the online prediction service of your new model. Use manual scaling with one node always available.

Question

You work for a small company that has deployed an ML model with autoscaling on Vertex AI to serve online predictions in a production environment. The current model receives about 20 prediction requests per hour with an average response time of one second. You have retrained the same model on a new batch of data, and now you are canary testing it, sending ~10% of production traffic to the new model. During this canary test, you notice that prediction requests for your new model are taking between 30 and 180 seconds to complete. What should you do?
  1. Submit a request to raise your project quota to ensure that multiple prediction services can run concurrently.
  2. Turn off auto-scaling for the online prediction service of your new model. Use manual scaling with one node always available. Correct Answer
  3. Remove your new model from the production environment. Compare the new model and existing model codes to identify the cause of the performance bottleneck.
  4. Remove your new model from the production environment. For a short trial period, send all incoming prediction requests to BigQuery. Request batch predictions from your new model, and then use the Data Labeling Service to validate your model’s performance before promoting it to production.

Key Takeaways

Tests recognition of autoscale-to-zero behavior versus model performance issues, trapping candidates who assume code defects instead of infrastructure provisioning delays.

This question evaluates how to troubleshoot high latency in Vertex AI online prediction services under low request volumes, with community consensus favoring manual scaling to prevent cold starts. Candidates must distinguish between infrastructure provisioning delays and actual model performance regressions.

Common mistake: Option C is commonly selected due to the dramatic latency spike, but it incorrectly attributes predictable cold-start delays to a hidden code or data bottleneck.

Expert Analysis

Why the Answer Is Correct

Vertex AI automatically scales prediction service replicas down to zero during periods of inactivity to optimize costs. With only twenty requests per hour, the new model instance consistently spins down, forcing a lengthy cold start whenever traffic arrives. Disabling auto-scaling and configuring a single dedicated node ensures the environment remains warm, instantly eliminating the thirty to one hundred eighty second initialization delay. This approach directly addresses the infrastructure bottleneck while maintaining the integrity of the canary test.

Why the Other Options Are Wrong

Raising project quotas does not influence runtime provisioning speed or eliminate cold start overhead. Removing the model to compare code ignores the well-documented autoscaling mechanics of managed ML platforms and prematurely halts valid performance validation. Diverting traffic to BigQuery for batch processing fundamentally changes the serving architecture and defeats the purpose of real-time online prediction evaluation. Each alternative fails to resolve the immediate latency trigger during active canary deployment.

Community Comment Notes

Candidates heavily debated options B and C, with many initially suspecting a code regression. Comment [1] accurately identifies that sparse traffic triggers scale-to-zero behavior, making persistent nodes essential for consistent latency. Comment [2] reinforces this by citing official documentation that mandates a minimum replica count, validating manual scaling as the prescribed remediation. While some users advocate for debugging code first, the predictable infrastructure pattern strongly supports prioritizing deployment configuration over premature code inspection.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

Always distinguish between application-level bottlenecks and cloud infrastructure provisioning limits when latency correlates with low traffic patterns. In certification scenarios, recognizing autoscaling trade-offs like cost optimization versus cold start latency frequently determines the correct architectural decision.

Community Votes

B
53%
C
47%

53% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 13 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 13

  1. sonicclasps 👍 4 Answered B
    sounds to me that the new model has too few requests per hour and therefore scales downs to 0. Which means it has to create the an instance every time it serves a request, and this takes time. By manually setting the number of nodes, the nodes will always be running, whether or not they are serving predictions
  2. YushiSato 👍 3
    I don't see B as the right answer. The Vertex AI Endpoint cannot scale to 0 for newer version of the model. > When you configure a DeployedModel, you must set dedicatedResources.minReplicaCount to at least 1. In other words, you cannot configure the DeployedModel to scale to 0 prediction nodes when it is unused. https://cloud.google.com/vertex-ai/docs/general/deployment#scaling
  3. guilhermebutzke 👍 2 Answered C
    Choose C. The significant increase in response time from 1 second to between 30 and 180 seconds indicates a performance issue with the new model. Before making any further changes or decisions, it's crucial to identify the root cause of this performance bottleneck. By comparing the code of the new model with the existing model, you can pinpoint any differences that might be causing the slowdown. In A, This may not be the root cause and could incur unnecessary costs without addressing the performance issue. In B, it doesn't address the underlying issue causing the significant increase in response time observed during canary testing. in D, This would significantly increase latency and hinder real-time predictions, negatively impacting user experience.
  4. AnnaR 👍 2 Answered B
    B can be effective in controlling resources available to the new model, ensuring that it is not delayed by the autoscaling trying to scale up from 0. Not A: there is no indication in the description that quota limits cause the slowdown and does not address issue where new model is performing poorly on canary testing. Not C : when you pull the new model from prod environment, you could affect end-user experience Not D: Same as C plus you rely on batch predictions which does not align with the need for online, real-time predictions in the prod environemnt. Data Labeling Service is more about assessing accuracy and less about resolving latency issues.
  5. desertlotus1211 👍 2 Answered C
    You're performing 20 predictions an hour - so scaling isn’t the root issue. Code issue.
Show 8 more comments
  1. b1a8fae 👍 1
    Unsure on this one, but I would go with A. B. Turning off auto-scaling is a good measure when dealing with datasets with steep spikes of requests traffic (here we are dealing with avg. 20 request per hour) "The service may not be able to bring nodes online fast enough to keep up with large spikes of request traffic." https://cloud.google.com/blog/products/ai-machine-learning/scaling-machine-learning-predictions C. You retrain the SAME model on a different batch of data. It is implied that the code is the same too? D. Actual quality of the model is not in question here, but rather the long prediction time per request. Even if the requests traffic is very low, I can only consider option A: the selected quota cannot deal with the amount of concurrent prediction requests.
  2. potomeek 👍 1 Answered C
    Removing the new model from production to debug and address the root cause of the latency issue is the most efficient and logical course of action. This ensures minimal disruption to production services and lays the groundwork for a smooth rollout after fixing the bottleneck
  3. kalle_balle 👍 1 Answered C
    Option B or D is completely wrong. Option A to raise the quota might be necessary in some situations but doesn't necessarily deal with the performance issue at the test. Option C seems like the most suitable option.
  4. vini123 👍 1 Answered B
    Since the same model is being used and the only change is the data, it's likely that the latency issue is caused by how Vertex AI is scaling the prediction service.
  5. VipinSingla 👍 1 Answered B
    bottleneck seems to be start of node as there are very low number of requests so having one node always available will help in this case.
  6. pinimichele01 👍 1 Answered B
    You have retrained the same model on a new batch of data
  7. Carlose2108 👍 1 Answered C
    I went C. Diagnosing the root cause.
  8. Aastha_Vashist 👍 1 Answered C
    went with c

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

How to preprocess BigQuery data for TensorFlow?

S-grade

How to Track Model and Prediction Lineage in Vertex AI?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide