Resolving 175B LLM OOM on a 32-Chip TPU v5p Pod

PMLE By islever Editorial Last updated

Short answer

C Shard the 175-billion-parameter model's parameters and optimizer states across the 32-chip TPU v5p Pod while retaining data parallelism, eliminating per-chip OOM without adding hardware.

Question

You are optimizing the training of a 175-billion parameter LLM on Gemini Enterprise Agent Platform. You have provisioned a TPU v5p Pod slice with 32 chips for the job. You initially ran the training using standard data parallelism, but the job immediately failed with an out-of-memory (OOM) error. You need to implement a training strategy that resolves the memory issue and minimizes training latency. What should you do?
  1. Configure the training to use a parameter server architecture where the model weights are stored in the host CPU memory.
  2. Enable gradient checkpointing, and continue using the data parallelism strategy.
  3. Configure the training to shard the model parameters and optimizer states, and continue using the data parallelism strategy. Correct Answer
  4. Scale up the TPU Pod slice to include more chips, and continue using the data parallelism strategy.

Key Takeaways

This question tests whether you can distinguish parameter and optimizer-state memory from activation memory and choose sharded data parallelism over checkpointing or hardware scaling.

For a 175-billion-parameter LLM on a 32-chip TPU v5p Pod, standard data parallelism OOMs because every chip holds the full model and optimizer state; the fix is sharded data parallelism (ZeRO/FSDP), not gradient checkpointing or more chips.

Common mistake: Choosing B (gradient checkpointing) is tempting because it reduces memory, but it primarily cuts activation memory and still leaves each chip holding the full 175B parameters and optimizer states, so the OOM persists.

Expert Analysis

The boundary: model-state memory vs. activation memory A 175-billion-parameter model needs roughly 350 GB just for bf16 weights, before gradients and optimizer states, and standard data parallelism replicates that full state on every TPU chip. The OOM here is therefore driven by per-chip model-state replication, not by an activation-memory explosion. The key distinction is between pure data parallelism, which keeps a complete model copy on each chip, and sharded data parallelism such as ZeRO or FSDP, which retains data-parallel training while partitioning parameters, gradients and optimizer state across ranks. This scenario falls on the sharded side of that boundary.

Why B does not cross the boundary Gradient checkpointing trades compute for memory by recomputing activations during the backward pass. It reduces activation memory, but it does not reduce the per-chip parameter and optimizer-state footprint that dominates a 175B model. Continuing pure data parallelism leaves the fundamental replication problem intact.

Why A and D fail A parameter-server design that stores weights in host CPU memory avoids per-chip replication but introduces CPU-memory capacity and host-to-device communication bottlenecks, hurting latency. Adding chips (D) does not fix pure data parallelism either: every new chip still stores a full model replica, so the OOM remains and cost rises.

Correct choice C shards model parameters and optimizer states while retaining data parallelism, shrinking the per-chip memory footprint and using the TPU pod interconnect efficiently. That resolves the OOM without the latency penalty of host offload or unnecessary extra hardware. The answer is C.

Official Reference

https://cloud.google.com/tpu/docs/v5p

This link opens the official vendor documentation in a new tab.

Exam Strategy

When an LLM training question reports OOM immediately at startup, before many steps, suspect model-state memory rather than activation memory. Compare options by whether they reduce per-chip parameter and optimizer storage: sharding, ZeRO or FSDP does, while checkpointing and adding chips do not. On TPU pods, prefer strategies that keep the fast interconnect busy with data-parallel gradients instead of offloading weights to host CPU memory.

Frequently Asked Questions

Why doesn't gradient checkpointing solve OOM for a 175B model on TPU v5p?

Gradient checkpointing reduces stored activation memory, but it does not reduce the per-chip copy of 175B parameters and optimizer states under pure data parallelism. Each chip still needs the full model state, so the OOM remains.

Does adding more TPU chips fix standard data parallelism OOM?

No. Pure data parallelism replicates the full model on every chip, so adding chips creates more replicas rather than reducing per-chip memory. The 175B model state still exceeds each chip's capacity, and cost increases.

When should sharded data parallelism be used instead of tensor or pipeline parallelism?

Use sharded data parallelism when the model state exceeds single-chip memory but you want to preserve data-parallel training. ZeRO or FSDP partitions parameters, gradients and optimizer states across data-parallel ranks, avoiding full replication.

More PMLE FAQ →

Community Votes

C
100%

100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 2 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 2

  1. taylz876 👍 1 Answered C
    Similar to Q64 and Q209 Why C is right • Standard data parallelism keeps a full copy of the model on every chip. A 175-billion-parameter model, plus its optimizer states, is far too big for one chip's memory, so every chip runs out of memory (OOM). • Sharding splits the model's parameters and optimizer states across the chips, so each chip holds only a slice. Data parallelism still splits the training data, so the work stays efficient and training latency stays low. Why the others are wrong • A: A parameter server in host CPU memory moves weights back and forth between the CPU and the chips, which adds a lot of latency. • B: Gradient checkpointing saves memory on activations, but the weights and optimizer states are still too big for each chip. • D: More chips with full copies of the model on each still run out of memory.
  2. n2183712847 👍 1 Answered C
    Correct answer: C. Sharding the model parameters and optimizer states across the TPU chips reduces the per-chip memory footprint while preserving data parallelism, resolving the OOM issue efficiently; A introduces CPU-memory and communication bottlenecks, B mainly reduces activation memory and does not sufficiently address the huge parameter/optimizer memory footprint, and D adds more hardware and cost rather than using memory-efficient sharding.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide