How to configure distributed training with minimal code changes?

PMLE By islever Editorial Last updated

Short answer

A Create a Vertex AI custom training job with GPU accelerators for the second worker pool. Use tf.distribute.MultiWorkerMirroredStrategy for distribution.

Question

You developed a Transformer model in TensorFlow to translate text. Your training data includes millions of documents in a Cloud Storage bucket. You plan to use distributed training to reduce training time. You need to configure the training job while minimizing the effort required to modify code and to manage the cluster’s configuration. What should you do?
  1. Create a Vertex AI custom training job with GPU accelerators for the second worker pool. Use tf.distribute.MultiWorkerMirroredStrategy for distribution. Correct Answer
  2. Create a Vertex AI custom distributed training job with Reduction Server. Use N1 high-memory machine type instances for the first and second pools, and use N1 high-CPU machine type instances for the third worker pool.
  3. Create a training job that uses Cloud TPU VMs. Use tf.distribute.TPUStrategy for distribution.
  4. Create a Vertex AI custom training job with a single worker pool of A2 GPU machine type instances. Use tf.distribute.MirroredStrategv for distribution.

Key Takeaways

The question tests the selection of a managed distributed training solution that balances performance with low operational overhead, specifically identifying MultiWorkerMirroredStrategy as the low-effort standard for multi-GPU setups.

To minimize code modification and cluster management effort for distributed TensorFlow training, use a Vertex AI custom training job with GPU accelerators and the tf.distribute.MultiWorkerMirroredStrategy.

Common mistake: Selecting Cloud TPU VMs (Option C) is a frequent error because, while TPUs are fast for Transformers, they typically require significant code changes compared to the minimal refactoring needed for MultiWorkerMirroredStrategy on GPUs.

Expert Analysis

Why the Answer Is Correct

Option A is the correct choice because Vertex AI custom training jobs provide a managed environment that handles cluster provisioning and scaling, significantly reducing management effort. Utilizing tf.distribute.MultiWorkerMirroredStrategy allows for synchronous data-parallel training across multiple GPU workers, which is ideal for Transformer models, and generally requires minimal code changes—mostly just wrapping the model creation and compilation in a strategy scope.

Why the Other Options Are Wrong

Option B is incorrect because introducing a Reduction Server adds configuration complexity and is typically reserved for specific asynchronous training scenarios or extreme parameter server loads, rather than standard synchronous training. Option C is incorrect because migrating to Cloud TPUs often necessitates substantial code alterations, such as adjusting input pipelines and enabling XLA compilation, which violates the requirement to minimize code changes. Option D is incorrect because tf.distribute.MirroredStrategy is designed for single-machine, multi-GPU training and does not support distributed training across a cluster of machines, which is necessary to efficiently process millions of documents.

Community Comment Notes

Community comments strongly validate Option A, emphasizing its ability to leverage managed infrastructure for reduced overhead and its compatibility with existing TensorFlow code. Commenters specifically ruled out Option C due to the code modification requirements for TPUs and Option B due to its complex setup involving different machine types and Reduction Servers.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When exam questions emphasize minimizing code changes and management effort for distributed training, always prefer Vertex AI Custom Jobs with MultiWorkerMirroredStrategy over TPUs or complex parameter server architectures, unless the workload explicitly demands TPU-specific optimizations.

Community Votes

A
100%

100% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 4 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 4

  1. fitri001 👍 3 Answered A
    Vertex AI custom training job: This leverages a managed service within GCP, reducing cluster configuration and management overhead. GPU accelerators for the second worker pool: This allows for distributed training across multiple GPUs, significantly speeding up training compared to a single worker pool. tf.distribute.MultiWorkerMirroredStrategy: This is a TensorFlow strategy specifically designed for distributed training on multiple machines. It minimizes code changes as it handles data parallelization and model replication across devices.
  2. pikachu007 👍 3 Answered A
    Minimizes code modification: MultiWorkerMirroredStrategy often requires minimal code changes to distribute training across multiple workers, aligning with the goal of minimizing effort. Simplifies cluster management: Vertex AI handles cluster configuration and scaling for custom training jobs, reducing the need for manual management. Effective distributed training: MultiWorkerMirroredStrategy is well-suited for large models and datasets, efficiently distributing training across GPUs.
  3. guilhermebutzke 👍 2 Answered A
    My Answer: A - Distributed training: Utilizes GPUs in 2nd worker pool for speedup. - Minimal code changes: Vertex AI custom job for ease of use. - Managed cluster: No manual configuration needed. Other options: - B: Complex setup with different machine types and Reduction Server. - C: TPUs may not be optimal for Transformers and require code changes. - D: Lacks distributed training, limiting speed improvement.
  4. Carlose2108 👍 2
    Why not C?

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

How to preprocess BigQuery data for TensorFlow?

S-grade

How to Track Model and Prediction Lineage in Vertex AI?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide