Resolving 175B LLM OOM on a 32-Chip TPU v5p Pod
Short answer
C Shard the 175-billion-parameter model's parameters and optimizer states across the 32-chip TPU v5p Pod while retaining data parallelism, eliminating per-chip OOM without adding hardware.
Question
Key Takeaways
This question tests whether you can distinguish parameter and optimizer-state memory from activation memory and choose sharded data parallelism over checkpointing or hardware scaling.
For a 175-billion-parameter LLM on a 32-chip TPU v5p Pod, standard data parallelism OOMs because every chip holds the full model and optimizer state; the fix is sharded data parallelism (ZeRO/FSDP), not gradient checkpointing or more chips.
Common mistake: Choosing B (gradient checkpointing) is tempting because it reduces memory, but it primarily cuts activation memory and still leaves each chip holding the full 175B parameters and optimizer states, so the OOM persists.
Expert Analysis
The boundary: model-state memory vs. activation memory A 175-billion-parameter model needs roughly 350 GB just for bf16 weights, before gradients and optimizer states, and standard data parallelism replicates that full state on every TPU chip. The OOM here is therefore driven by per-chip model-state replication, not by an activation-memory explosion. The key distinction is between pure data parallelism, which keeps a complete model copy on each chip, and sharded data parallelism such as ZeRO or FSDP, which retains data-parallel training while partitioning parameters, gradients and optimizer state across ranks. This scenario falls on the sharded side of that boundary.
Why B does not cross the boundary Gradient checkpointing trades compute for memory by recomputing activations during the backward pass. It reduces activation memory, but it does not reduce the per-chip parameter and optimizer-state footprint that dominates a 175B model. Continuing pure data parallelism leaves the fundamental replication problem intact.
Why A and D fail A parameter-server design that stores weights in host CPU memory avoids per-chip replication but introduces CPU-memory capacity and host-to-device communication bottlenecks, hurting latency. Adding chips (D) does not fix pure data parallelism either: every new chip still stores a full model replica, so the OOM remains and cost rises.
Correct choice C shards model parameters and optimizer states while retaining data parallelism, shrinking the per-chip memory footprint and using the TPU pod interconnect efficiently. That resolves the OOM without the latency penalty of host offload or unnecessary extra hardware. The answer is C.
Official Reference
https://cloud.google.com/tpu/docs/v5p
This link opens the official vendor documentation in a new tab.
Exam Strategy
When an LLM training question reports OOM immediately at startup, before many steps, suspect model-state memory rather than activation memory. Compare options by whether they reduce per-chip parameter and optimizer storage: sharding, ZeRO or FSDP does, while checkpointing and adding chips do not. On TPU pods, prefer strategies that keep the fast interconnect busy with data-parallel gradients instead of offloading weights to host CPU memory.
Frequently Asked Questions
Why doesn't gradient checkpointing solve OOM for a 175B model on TPU v5p?
Does adding more TPU chips fix standard data parallelism OOM?
When should sharded data parallelism be used instead of tensor or pipeline parallelism?
Community Votes
100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 2 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 2
-
taylz876 👍 1 Answered CSimilar to Q64 and Q209 Why C is right • Standard data parallelism keeps a full copy of the model on every chip. A 175-billion-parameter model, plus its optimizer states, is far too big for one chip's memory, so every chip runs out of memory (OOM). • Sharding splits the model's parameters and optimizer states across the chips, so each chip holds only a slice. Data parallelism still splits the training data, so the work stays efficient and training latency stays low. Why the others are wrong • A: A parameter server in host CPU memory moves weights back and forth between the CPU and the chips, which adds a lot of latency. • B: Gradient checkpointing saves memory on activations, but the weights and optimizer states are still too big for each chip. • D: More chips with full copies of the model on each still run out of memory.
-
n2183712847 👍 1 Answered CCorrect answer: C. Sharding the model parameters and optimizer states across the TPU chips reduces the per-chip memory footprint while preserving data parallelism, resolving the OOM issue efficiently; A introduces CPU-memory and communication bottlenecks, B mainly reduces activation memory and does not sufficiently address the huge parameter/optimizer memory footprint, and D adds more hardware and cost rather than using memory-efficient sharding.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All PMLE Questions
Access 65 questions with complete answers and detailed explanations.
View Full PMLE Practice Test →