How to Optimize Vertex AI Pipeline Caching for Cost Reduction?

You developed a Vertex AI pipeline that trains a classification model on data stored in a large BigQuery table. The pipeline has four steps, where each step is created by a Python function that uses the KubeFlow v2 API. The components have the following names: You launch your Vertex AI pipeline as the following: You perform many model iterations by adjusting the code and parameters of the training step. You observe high costs associated with the development, particularly the data export and preprocessing steps. You need to reduce model development costs. What should you do? - -

Change the components’ YAML filenames to export.yaml, preprocess,yaml, f "train- Source Reference Answer
Add the {"kubeflow.v1.caching": True} parameter to the set of params provided to your PipelineJob.
Move the first step of your pipeline to a separate step, and provide a cached path to Cloud Storage as an input to the main pipeline.
Change the name of the pipeline to f"my-awesome-pipeline-{dt}".

Community Votes

52%

29%

19%

52% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

Assesses knowledge of Kubeflow caching mechanics, where dynamic filenames invalidate cache keys and force expensive re-execution.

This scenario evaluates optimizing Vertex AI Pipelines by leveraging Kubeflow’s automatic caching to eliminate redundant compute expenses. Industry consensus confirms that static component naming is required to trigger successful cache hits.

Test-takers frequently select option B or C, incorrectly assuming v1 caching syntax applies to v2 APIs or that manual storage refactoring replaces native caching.

Community Discussion (14 comments)

guilhermebutzke 👍 6 Selected: A

My Answer: A From what I understood, it's about optimizing the process of adjusting code while utilizing previously processed results from the pipeline. Kubeflow inherently caches these steps, eliminating the need to explicitly store results in a designated path. However, the original filenames include a timestamp (-dt), suggesting that by removing this timestamp, the pipeline steps might not rerun as expected. Option C could be an approach, but it would require more effort to implement (since Kubeflow handles it automatically). Additionally, the beginning of the option only mentions moving the first step, which is the export, and doesn't say anything about preprocessing (which could be one of the more expensive steps). So, considering all of these factors, I think A is the best choice."

HaroonRaizada01 👍 1 Selected: B

Sivaram06 👍 1 Selected: B

Adding caching to your pipeline by setting the parameter {"kubeflow.v1.caching": True} is the most efficient and effective approach to reduce model development costs, particularly for steps like data export and preprocessing, which are often time-consuming and costly to repeat during multiple iterations. This will help you avoid unnecessary re-computation and save on resource usage.

lunalongo 👍 2 Selected: C

Option A is a superficial change with no significant impact on cost optimization. Option C is the correct approach for effectively leveraging caching to reduce costs. C strategically uses the caching mechanism by separating the expensive preprocessing steps and storing their outputs in Cloud Storage, thus reducing costs by reusing the preprocessed data across multiple pipeline runs. Changing filenames could affect caching only if the caching mechanism relies on exact filename matching, which is unlikely. Besides, Kubeflow and Vertex AI Pipelines do not automatically handle caching of intermediate results; it is not inherent to the pipeline steps themselves; it's a feature that needs to be explicitly managed and leveraged.

f084277 👍 1 Selected: A

A. The dynamic filename is causing kubeflow to be unable to cache the export and preprocess steps, causing the problems mentioned in the question.

Foxy2021 👍 1

I select C: By leveraging a Dataproc cluster, you can maintain compatibility with your existing PySpark jobs, minimize management overhead, and create a scalable proof of concept quickly and efficiently.

Foxy2021 👍 1

I select B. A: Changing the YAML filenames does not affect caching behavior or cost reduction. The pipeline's efficiency and cost effectiveness are primarily governed by how it handles inputs and outputs rather than the filenames of the components. C: Moving the first step to a separate pipeline may help with organization but doesn’t directly address the cost incurred by repeated data exports and preprocessing. Also, simply providing a cached path does not guarantee that the preprocessing step itself won’t be executed multiple times. D: Changing the name of the pipeline to include a timestamp or other identifier does not influence caching or resource usage. It merely alters the identification of the pipeline runs without any impact on the efficiency of the operations being performed.

gscharly 👍 1 Selected: A

see guilhermebutzke

pinimichele01 👍 1 Selected: A

see guilhermebutzke

Yan_X 👍 1 Selected: C

C Caching should be enabled for all steps, e.g., export, preprocessing and training.

shadz10 👍 2 Selected: C

Not A - Changing file names does not help with reducing costs Not B - you cannot directly use kubeflow.v1.caching on a pipeline that uses the KubeFlow v2 API. Version Incompatibility: The kubeflow.v1.caching module is specifically designed for KubeFlow Pipelines v1, and its structure and functionality are not directly compatible with KubeFlow Pipelines v2. so best option here is C

b1a8fae 👍 1 Selected: C

I considered B but a search of "kubeflow.v1.caching" on Google only produces 1 result, which is this very question on this very website. Thus, I rule it out as non-existent (please share a resource if there is any that proves it exists) and opt for C.

BlehMaks 👍 2 Selected: A

i think it's A. 1)if we want to use the same results several times we shouldn't rename them. so we need to delete {dt} from the first two components names. 2)we already have this option enable_caching = True, why do we need kubeflow.v1.caching then? 3)i'm not sure but may be it does metter

pikachu007 👍 2 Selected: B

Enables caching: Setting this parameter instructs Vertex AI Pipelines to cache the outputs of pipeline steps that have successfully completed. This means that if a step's inputs haven't changed, its execution can be skipped, reusing the cached output instead. Targets costly steps: The prompt highlights that data export and preprocessing steps are particularly expensive. Caching these steps can significantly reduce costs during model iterations.

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

Kubeflow Pipelines automatically cache outputs based on component names and input hashes. When YAML filenames contain dynamic timestamps, the cache key changes on every run, forcing redundant execution of costly steps. Standardizing the filenames allows the cache to activate seamlessly, reusing previous results and significantly lowering development costs.

Why the Other Options Are Wrong

Option B references kubeflow.v1.caching, which is incompatible with the KubeFlow v2 API explicitly mentioned in the scenario. Option C suggests restructuring the pipeline architecture unnecessarily, as caching can be enabled natively without moving steps. Option D introduces another timestamp, exacerbating the caching failure rather than resolving it.

Community Comment Notes

Multiple voters highlighted that dynamic naming breaks caching logic, aligning with platform behavior (Comments 1, 4, 7). Users also correctly noted that v1 caching modules do not apply to v2 pipelines, effectively dismissing option B (Comments 3, 9). Several candidates initially favored manual cloud storage paths but recognized that native caching is the intended exam solution.

Official Reference

https://cloud.google.com/vertex-ai/docs/pipelines/cache

Exam Strategy

Always identify dynamic variables in pipeline configurations, as they routinely disable deterministic caching and inflate resource usage. Favor built-in optimization features over manual architectural changes unless explicitly required.

Related Analysis

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide