Deploying Gemma on GKE for Low Latency

PMLE By islever Editorial Last updated

Serving models. Scaling online model serving.

Short answer

B Deploy the model on a GKE cluster using Model Garden to gain full infrastructure control and minimize inference time with optimized serving templates.

Question

You work as an ML researcher at an investment bank, and you are experimenting with the Gemma large language model (LLM). You plan to deploy the model for an internal use case. You need to have full control of the mode's underlying infrastructure and minimize the model's inference time. Which serving configuration should you use for this task?
  1. Deploy the model on a Vertex AI endpoint manually by creating a custom inference container.
  2. Deploy the model on a Google Kubernetes Engine (GKE) cluster by using the deployment options in Model Garden. Correct Answer
  3. Deploy the model on a Vertex AI endpoint by using one-click deployment in Model Garden.
  4. Deploy the model on a Google Kubernetes Engine (GKE) cluster manually by cresting a custom yaml manifest.

Key Takeaways

This question tests the trade-off between infrastructure control and inference optimization, requiring GKE for the former and Model Garden templates for the latter.

Deploying the Gemma LLM on a GKE cluster using Model Garden provides full control over the underlying infrastructure and minimizes inference time through pre-optimized serving configurations.

Common mistake: Choosing Option D because a custom YAML offers maximum manual control, but failing to recognize that Model Garden provides pre-optimized serving engines (like vLLM) that minimize inference time better than a scratch-built manifest.

Expert Analysis

Underlying Infrastructure Control The requirement to have "full control of the model's underlying infrastructure" eliminates Vertex AI endpoints (Options A and C). Vertex AI is a fully managed service where Google provisions and manages the underlying compute nodes, operating system, and cluster configuration. In contrast, deploying to a Google Kubernetes Engine (GKE) cluster (Options B and D) provides direct administrative access to node pools, GPU driver configurations, and cluster networking, satisfying the strict infrastructure control requirement for an internal bank use case.

Minimizing Inference Time To minimize inference time for a large language model like Gemma, the serving engine must be highly optimized (e.g., utilizing vLLM, Text Generation Inference, or JetStream). Option D, manually creating a custom YAML manifest, requires building the serving stack from scratch. This approach is highly susceptible to suboptimal configurations that result in higher latency. Option B leverages the deployment options in Model Garden, which provide pre-configured, optimized serving containers specifically tuned for low latency and high throughput. These templates implement best practices for inference acceleration out of the box.

Conclusion Option B correctly combines the infrastructure control of GKE with the inference optimization of Model Garden. As noted in the comments, while custom YAML (D) offers maximum manual control, Model Garden's GKE deployment uses an optimized serving engine that minimizes inference speed compared to a generic manual setup, making B the superior choice for this dual requirement.

Official Reference

https://cloud.google.com/kubernetes-engine/docs/concepts/deployment

This link opens the official vendor documentation in a new tab.

Exam Strategy

When a question demands both "full control of underlying infrastructure" and "optimized performance," separate the requirements. "Full control" rules out fully managed endpoints (Vertex AI), pointing to self-managed compute (GKE). "Optimized performance" rules out manual configurations (custom YAML), pointing to vendor-provided templates (Model Garden).

Community Votes

B
63%
D
25%
A
12%

63% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 8 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 8

  1. vamgcp 👍 3 Answered D
    Deploying the model on GKE with a custom YAML manifest allows maximum control over infrastructure and latency, aligning with the need for low inference time and internal model use. Vertex AI's one-click deployment (Option A) limits control, and deploying on Vertex AI (Option C) doesn't allow for as much customization as a GKE setup.
  2. Fer660 👍 3 Answered B
    Not A or C: Vertex AI deployment will not give you full control of the underlying infra. Not D: because B is faster path to deployment B:in order to minimize to deployment, start with a standard deployment. Then tweak the deployment to meet your latency requirements.
  3. hit_cloudie 👍 3 Answered A
    Provides full control over the container and environment while benefiting from Vertex AI’s optimized serving infrastructure. You can control dependencies, runtime, and hardware (e.g., GPUs), which is ideal for internal LLM serving with low latency.
  4. tmpuserx 👍 2 Answered D
    Model Garden deployment options typically use default configurations not optimized for low latency
  5. taylz876 👍 1 Answered B
    Why B is right • Full control of the underlying infrastructure: "Need full control over the serving infrastructure → deploy to GKE." On GKE you choose the node pools, the accelerators (GPUs) and the configuration. A Vertex AI endpoint (now an Agent Platform endpoint) hides the cluster layer from you. • Minimize inference time: Model Garden's GKE deployment options come with serving configurations already tuned for open LLMs such as Gemma, so you get fast inference without hand-tuning. Why the others are wrong • A and C: Both run on a managed endpoint, so you don't control the underlying infrastructure. • D: GKE does give control, but a hand-written manifest means you have to build and tune the serving setup yourself, with the risk of slower inference. Pattern: a Model Garden open model with full infrastructure control and the fastest inference means GKE using Model Garden's deployment options. A one-click endpoint is for ease, not control, and a manual manifest isn't tuned.
Show 3 more comments
  1. lance_yu 👍 1 Answered B
    Deploying on GKE provides: Full control over: Node types GPUs Autoscaling Networking Storage Runtime configuration Ability to optimize serving performance for low latency Support for Model Garden deployment templates that reduce operational effort compared to building everything from scratch Model Garden provides deployment options specifically for open models like Gemma, allowing deployment to GKE while retaining infrastructure control.
  2. RogerRabid 👍 1 Answered B
    A and C are out because you don't have control over the infra. B over D because Model Garden uses optimized serving engine and inference speed is minimized in comparison to generic GKE/Manual YAML
  3. Johnb9682 👍 1 Answered B
    1. Full control of the model's underlying infrastructure 2. Minimize the model's inference time

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

Building CLV Models with BigQuery ML in Vertex AI Workbench

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide