How to Optimize Vertex AI Pipeline Execution for Algorithm Testing?

You have created a Vertex AI pipeline that includes two steps. The first step preprocesses 10 TB data completes in about 1 hour, and saves the result in a Cloud Storage bucket. The second step uses the processed data to train a model. You need to update the model’s code to allow you to test different algorithms. You want to reduce pipeline execution time and cost while also minimizing pipeline changes. What should you do?

  1. Add a pipeline parameter and an additional pipeline step. Depending on the parameter value, the pipeline step conducts or skips data preprocessing, and starts model training.
  2. Create another pipeline without the preprocessing step, and hardcode the preprocessed Cloud Storage file location for model training.
  3. Configure a machine with more CPU and RAM from the compute-optimized machine family for the data preprocessing step.
  4. Enable caching for the pipeline job, and disable caching for the model training step. Source Reference Answer

Community Votes

D
100%

100% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

Tests understanding of pipeline caching mechanics, with the common trap being the misconception that caching applies uniformly across all steps rather than allowing granular enable/disable settings.

Leverage Vertex AI pipeline caching to skip redundant preprocessing while enabling rapid algorithm iteration. The community strongly agrees that configuring task-level caching minimizes both runtime and costs without restructuring the pipeline.

Option B is frequently chosen because creating a separate pipeline seems efficient, but it violates the requirement to minimize pipeline changes and increases maintenance overhead.

Community Discussion (6 comments)

lunalongo 👍 1 Selected: B
B) The preprocessing step is already complete and its output is stored is in GCS, so a separate, smaller pipeline just for training is the most efficient solution. *A) Conditional logic still performs prepocessing steps when the logic points to not skipping it, increasing costs; C) While reducing preprocessing time, this solution would increase this step's cost; D) Would still include unnecessary preprocessing for each algorithm test before it's cached.
fitri001 👍 1 Selected: D
Caching Preprocessed Data: Since the preprocessed data (10 TB) is the same for different model training runs, enabling caching allows Vertex AI to reuse it for subsequent pipeline executions. This significantly reduces execution time and cost, especially for large datasets. Disabling Model Training Cache: Model training is typically non-deterministic due to factors like random initialization. Caching the model training step could lead to stale models and inaccurate results. Disabling caching ensures the model is re-trained each time with potentially updated code for different algorithms.
gscharly 👍 1 Selected: D
agree with guilhermebutzke
guilhermebutzke 👍 3 Selected: D
According to this documentation cited: https://cloud.google.com/vertex-ai/docs/pipelines/configure-caching it is possible to write a pipeline setting True or False for each task component, like this: # Model training step with caching disabled train_model_task = train_model_op() train_model_task.set_caching_options(False) # Disable caching for this step # Model training step depends on the preprocessing step train_model_task.after(preprocess_task) So, with this, letter D is the best option. Furthermore, letter A and, Adding a pipeline parameter and an additional pipeline step introduces unnecessary complexity when caching can handle conditional execution efficiently and in letter C, configuring a machine with more CPU and RAM for preprocessing does not address the goal of minimizing pipeline changes and reducing execution time/cost effectively.
b1a8fae 👍 4 Selected: D
Not A. Adding a pipeline parameter and new pipeline steps does not minimise pipeline changes. Not C. The idea is not to re-run the preprocessing step at all. Not B. Creating a whole new pipeline implies a significant investment of effort. I opt for D: Enabling caching only for preprocessing job (although it says “pipeline job” in the option, I think that is a typo). Quoting Vertex AI docs: “If there is a matching execution in Vertex ML Metadata, the outputs of that execution are used and the step is skipped. This helps to reduce costs by skipping computations that were completed in a previous pipeline run.” https://cloud.google.com/vertex-ai/docs/pipelines/configure-caching
pikachu007 👍 1 Selected: A
The pipeline already generates the preprocessed dataset and stores, there's no need to preprocess again for another model

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Submitted for moderation before publishing. Keep it helpful and respectful.

Expert Analysis

Why the Answer Is Correct

Enabling caching for the preprocessing step allows Vertex AI to store and reuse the output when inputs have not changed, instantly skipping the 1-hour data processing phase. Disabling caching for the training step ensures each algorithm run executes fresh, which is necessary since model training involves stochastic elements like random weight initialization. This granular control directly satisfies the goals of reducing time, cutting costs, and avoiding structural pipeline modifications.

Why the Other Options Are Wrong

Option A introduces conditional branching that still risks re-executing preprocessing if parameters are not perfectly managed, violating the minimize changes constraint. Option B requires building and maintaining an entirely new pipeline, which contradicts the explicit goal of minimizing pipeline changes. Option C merely scales up compute resources, which reduces duration but significantly increases costs without addressing the fundamental redundancy of reprocessing identical data.

Community Comment Notes

Candidates consistently highlight that Vertex AI supports per-task caching configuration via .set_caching_options(), as noted in comment [2]. Multiple users emphasize that disabling caching for the training step prevents stale results from non-deterministic algorithms, aligning with official documentation guidance. Several contributors correctly dismiss option A and B as overly complex or contradictory to the prompt's constraints.

Official Reference

https://cloud.google.com/vertex-ai/docs/pipelines/configure-caching

Exam Strategy

When optimizing ML pipelines, always evaluate whether task outputs are deterministic before enabling caching. Use granular caching controls to skip expensive upstream steps while ensuring downstream components execute freshly for accurate experimentation.

Related Analysis

You work for a hospital that wants to optimize how it schedu

S-Grade

You work for a company that captures live video footage of c

S-Grade

You need to train an XGBoost model on a small dataset. Your

S-Grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide