Reducing ML Training Costs with Preemptible VMs and Checkpoints

PMLE By islever Editorial Last updated

Building models given the task considering cost, complexity, latency, and scalability. Training models.

Short answer

C Migrate to training with Kubeflow on Google Kubernetes Engine, and use preemptible VMs with checkpoints.

Question

You lead a data science team at a large international corporation. Most of the models your team trains are large-scale models using high-level TensorFlow APIs on AI Platform with GPUs. Your team usually takes a few weeks or months to iterate on a new version of a model. You were recently asked to review your team’s spending. How should you reduce your Google Cloud compute costs without impacting the model’s performance?
  1. Use AI Platform to run distributed training jobs with checkpoints.
  2. Use AI Platform to run distributed training jobs without checkpoints.
  3. Migrate to training with Kuberflow on Google Kubernetes Engine, and use preemptible VMs with checkpoints. Correct Answer
  4. Migrate to training with Kuberflow on Google Kubernetes Engine, and use preemptible VMs without checkpoints.

Key Takeaways

The main trap is ignoring the short lifetime of preemptible VMs; without checkpoints, long training jobs will fail repeatedly, negating cost benefits.

Learn how to reduce Google Cloud ML training costs by using Kubeflow on GKE with preemptible VMs and checkpoints to handle long-running model iterations.

Common mistake: Choosing Option A because it mentions checkpoints but fails to utilize Preemptible/Spot VMs, which are the primary source of cost reduction for long-running jobs.

Expert Analysis

The scenario involves training large-scale models that take weeks or months to complete. The primary objective is to reduce Google Cloud compute costs without impacting performance.

Preemptible VMs (now known as Spot VMs) offer significant cost savings compared to standard on-demand VMs. However, they can be terminated by the cloud provider with little notice, typically after a maximum lifetime of 24 hours. This makes them unsuitable for long-running jobs unless the job is resilient to interruptions.

To use Preemptible/Spot VMs effectively for long training runs, you must implement checkpointing. Checkpointing saves the model's state at regular intervals. If a VM is preempted, the training job can resume from the last saved checkpoint rather than starting over. This ensures that the investment in compute time is not lost and the overall training duration remains manageable despite interruptions.

Option A suggests using AI Platform with checkpoints but does not mention Preemptible VMs, missing the key cost-saving mechanism of spot pricing. Option B lacks both checkpointing and spot pricing. Option D uses Preemptible VMs but lacks checkpointing, which would result in frequent restarts and potentially longer total training times due to wasted work, thus impacting efficiency if not managed carefully, though technically it might still finish. However, the combination in C is the industry best practice for this specific constraint: migrating to a flexible orchestration platform like Kubeflow on GKE allows for better management of distributed training across spot instances, and using Preemptible VMs with checkpoints is the definitive strategy for reducing costs while maintaining progress integrity.

Therefore, migrating to Kubeflow on GKE and utilizing Preemptible VMs with checkpoints (Option C) is the correct approach to minimize costs while ensuring the model training completes successfully without data loss.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When faced with 'reduce cost' questions involving long-duration tasks, look for 'Preemptible' or 'Spot' instances combined with 'Checkpointing'. Always verify if the solution addresses the interruption risk inherent in spot pricing. If a question implies stability is key, ensure the mechanism for resuming interrupted jobs is explicitly stated.

Frequently Asked Questions

Why are checkpoints necessary when using Preemptible VMs?

Preemptible VMs can be terminated at any time, usually within 24 hours. Checkpoints allow the training job to resume from the last saved state instead of restarting from scratch, preventing loss of computed progress.

Can I use Preemptible VMs for any length of training job?

Yes, provided the job is designed to handle interruptions via checkpointing. Without checkpoints, long jobs would fail repeatedly, making them impractical.

What is the benefit of using Kubeflow on GKE for this scenario?

Kubeflow provides robust orchestration for distributed training pipelines, making it easier to manage complex workflows across multiple nodes, including handling failures and resumptions efficiently.

More PMLE FAQ →

Community Votes

C
77%
A
23%

77% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 20 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 20

  1. seifou 👍 11 Answered C
    https://cloud.google.com/blog/products/ai-machine-learning/reduce-the-costs-of-ml-workflows-with-preemptible-vms-and-gpus?hl=en
  2. MultiCloudIronMan 👍 5 Answered C
    Pre-emptive VMs are cheaper and checkpoints will enable termination if the result is acceptable
  3. joaquinmenendez 👍 4
    C is the best approach because it allows you to reduce your compute costs without impacting the model's performance. Preemptible VMs are much cheaper than standard VMs, but they can be terminated at any time. By using checkpoints, you can ensure that your training job can be resumed if a preemptible VM is terminated. Also, even if training takes days, the checkpoints will prevent lossing the progress if preemtible VM are down.
  4. hiromi 👍 4 Answered C
    It's seem C - https://www.kubeflow.org/docs/distributions/gke/pipelines/preemptible/ - https://cloud.google.com/optimization/docs/guide/checkpointing
  5. ares81 👍 3
    "A Preemptible VM (PVM) is a Google Compute Engine (GCE) virtual machine (VM) instance that can be purchased for a steep discount as long as the customer accepts that the instance will terminate after 24 hours." This excludes C and D. Checkpoints are needed for long processing, so A.
Show 15 more comments
  1. neochaotic 👍 3 Answered C
    C - Reduce cost with preemptive instances and add checkpoints to snapshot intermediate results
  2. _learner_ 👍 2 Answered A
    preemtible vm are valid for 24hrs. Hence training needs months to complete which is mentioned in question that makes A is answer.
  3. LearnSodas 👍 2 Answered A
    Saving checkpoints avoids re-run from scratch
  4. Liting 👍 2 Answered C
    Optimize cost then should use kubeflow
  5. ares81 👍 2 Answered A
    It's A.
  6. taylz876 👍 1 Answered C
    The team trains large models on GPUs, and each iteration takes weeks or months. That's a lot of compute time, so the biggest saving is to use preemptible VMs, which are much cheaper. The catch is that Google can reclaim them at any time, which would otherwise lose days of work. Checkpoints solve that by saving progress regularly, so an interrupted job resumes from the last checkpoint instead of starting over. That gives you the savings without hurting the model's performance. The question asks for Kubeflow on GKE because at that time AI Platform Training didn't offer preemptible VMs. Current names: preemptible VMs are now Spot VMs. Today you can also use Spot VMs in Agent Platform custom training, so you wouldn't need to move to GKE.
  7. tavva_prudhvi 👍 1
    I think it’s a. By using distributed training jobs with checkpoints, you can train your models on multiple GPUs simultaneously, which reduces the training time. Checkpoints allow you to save the progress of your training jobs regularly, so if the training job gets interrupted or fails, you can restart it from the last checkpoint instead of starting from scratch. This saves time and resources, which reduces costs. Additionally, AI Platform's autoscaling feature can automatically adjust the number of resources used based on the workload, further optimizing costs.
  8. CloudKida 👍 1 Answered C
    https://cloud.google.com/ai-platform/prediction/docs/ai-explanations/overview AI Explanations helps you understand your model's outputs for classification and regression tasks. Whenever you request a prediction on AI Platform, AI Explanations tells you how much each feature in the data contributed to the predicted result. You can then use this information to verify that the model is behaving as expected, recognize bias in your models, and get ideas for ways to improve your model and your training data.
  9. tavva_prudhvi 👍 1
    Additionally, AI Platform's autoscaling feature can automatically adjust the number of resources used based on the workload, further optimizing costs.
  10. John_Pongthorn 👍 1
    C is out of date ? AI Platform is Vertex-AI ,so , this is a simple scenario that would accommodate infrastructure for this case.
  11. YangG 👍 1
    I think it should be A https://cloud.google.com/ai-platform/training/docs/overview
  12. libo1985 👍 1
    I guess distributed training is not cheap. So C.
  13. PhilipKoku 👍 1 Answered C
    C) Preemptible VMs with Check points
  14. M25 👍 1 Answered C
    Went with C
  15. sashimii14 👍 1 Answered C
    C for me

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide