Reducing Vertex AI Pipeline Costs by Decoupling Data Export

PMLE By islever Editorial Last updated

Developing end-to-end ML pipelines.

Short answer

C Move the first step of your pipeline to a separate step, and provide a cached path to Cloud Storage as an input to the main pipeline.

Question

You developed a Vertex AI pipeline that trains a classification model on data stored in a large BigQuery table. The pipeline has four steps, where each step is created by a Python function that uses the KubeFlow v2 API. The components have the following names:
Exhibit: Reducing Vertex AI Pipeline Costs by Decoupling Data Export
You launch your Vertex AI pipeline as the following:
Exhibit: Reducing Vertex AI Pipeline Costs by Decoupling Data Export
You perform many model iterations by adjusting the code and parameters of the training step. You observe high costs associated with the development, particularly the data export and preprocessing steps. You need to reduce model development costs. What should you do?
  1. Change the components’ YAML filenames to export.yaml, preprocess,yaml, f "train- {dt}.yaml", f"calibrate-{dt).vaml".
  2. Add the {"kubeflow.v1.caching": True} parameter to the set of params provided to your PipelineJob.
  3. Move the first step of your pipeline to a separate step, and provide a cached path to Cloud Storage as an input to the main pipeline. Correct Answer
  4. Change the name of the pipeline to f"my-awesome-pipeline-{dt}".

Key Takeaways

The trap is using KFP v1 caching parameters on a v2 pipeline; the real solution is architectural decoupling to avoid redundant BigQuery exports.

Learn how to reduce model development costs in Vertex AI pipelines by decoupling expensive data export steps from iterative training loops using Cloud Storage caching.

Common mistake: Choosing Option B because it mentions 'caching', but failing to notice that `kubeflow.v1.caching` is incompatible with the KubeFlow v2 API specified in the question.

Expert Analysis

The core challenge here is optimizing a Vertex AI pipeline built with the KubeFlow v2 API to reduce costs during iterative model development. The question highlights that the data export and preprocessing steps are particularly expensive.

Let's analyze why the most attractive wrong answers fail:

Option B (Add caching parameter): This is a strong distractor because caching reduces cost. However, the comment section correctly identifies a critical technical constraint: kubeflow.v1.caching is specific to KFP v1. The question explicitly states the pipeline uses the KubeFlow v2 API. In KFP v2, caching is handled differently (often via the @component decorator or pipeline job settings in Vertex AI), and using a v1-specific parameter would likely cause an error or be ignored. Furthermore, even if caching were enabled, it doesn't solve the issue of re-exporting* data from BigQuery if the input signature changes or if the cache is invalidated by dynamic names.

Option A (Change YAML filenames): This option suggests removing timestamps (-dt) from filenames to enable caching. While stable filenames help with caching, simply renaming files doesn't address the root cause: the pipeline is re-running the export from BigQuery. Even with caching, if the pipeline structure forces a re-execution of the export step due to how inputs/outputs are defined, or if the cache isn't properly leveraged for the data artifact*, you still pay for the BigQuery read/export. More importantly, Option C offers a structural optimization that bypasses this step entirely for iterations.

  • Why Option C is correct: Moving the first step (data export/preprocessing) to a separate pipeline or decoupling it allows you to run this expensive operation once. You store the result in Cloud Storage. Subsequent iterations of the model training only need to read from Cloud Storage, which is cheap and fast. This avoids the recurring cost of querying and exporting large datasets from BigQuery for every single model iteration. This architectural change directly targets the identified cost driver (data export) and enables efficient iteration.

Therefore, decoupling the static data preparation from the iterative training loop is the most effective way to reduce costs.

Exam Strategy

Always verify the API version mentioned in the question (e.g., KFP v1 vs. v2). Many exam questions test compatibility between features and versions. If a feature is version-specific (like v1 caching), it cannot be used in a v2 pipeline regardless of its conceptual appropriateness. Look for architectural solutions that eliminate redundant expensive operations rather than just enabling features.

Frequently Asked Questions

Why can't I use kubeflow.v1.caching in a KFP v2 pipeline?

KFP v1 and v2 have different APIs and structures. The kubeflow.v1.caching module is specific to the older v1 framework. In KFP v2, caching is managed through different mechanisms, often integrated into the component definition or pipeline job configuration within Vertex AI.

Does changing file names enable caching?

Stable file names can help caching work correctly by ensuring inputs/outputs match across runs. However, simply changing names does not stop the pipeline from re-executing steps like BigQuery exports if the logic requires it. It is a supporting factor, not a primary cost-saving architecture.

More PMLE FAQ →

Community Votes

A
43%
C
36%
B
21%

43% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 16 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 16

  1. guilhermebutzke 👍 8 Answered A
    My Answer: A From what I understood, it's about optimizing the process of adjusting code while utilizing previously processed results from the pipeline. Kubeflow inherently caches these steps, eliminating the need to explicitly store results in a designated path. However, the original filenames include a timestamp (-dt), suggesting that by removing this timestamp, the pipeline steps might not rerun as expected. Option C could be an approach, but it would require more effort to implement (since Kubeflow handles it automatically). Additionally, the beginning of the option only mentions moving the first step, which is the export, and doesn't say anything about preprocessing (which could be one of the more expensive steps). So, considering all of these factors, I think A is the best choice."
  2. mfounta 👍 4 Answered A
    A "When caching is enabled for a component, KFP will reuse the component’s outputs if the component is executed again with the same inputs and parameters (and the output is still available)." With the dynamic file names, inputs will vary per execution. Caching will only work if the inputs are constant. https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/caching/#:~:text=Caching%20is%20enabled%20by%20default,False)%20on%20a%20task%20object.&text=You%20can%20also%20enable%20or,submitting%20a%20pipeline%20for%20execution.
  3. f084277 👍 3 Answered A
    A. The dynamic filename is causing kubeflow to be unable to cache the export and preprocess steps, causing the problems mentioned in the question.
  4. lunalongo 👍 2 Answered C
    Option A is a superficial change with no significant impact on cost optimization. Option C is the correct approach for effectively leveraging caching to reduce costs. C strategically uses the caching mechanism by separating the expensive preprocessing steps and storing their outputs in Cloud Storage, thus reducing costs by reusing the preprocessed data across multiple pipeline runs. Changing filenames could affect caching only if the caching mechanism relies on exact filename matching, which is unlikely. Besides, Kubeflow and Vertex AI Pipelines do not automatically handle caching of intermediate results; it is not inherent to the pipeline steps themselves; it's a feature that needs to be explicitly managed and leveraged.
  5. pikachu007 👍 2 Answered B
    Enables caching: Setting this parameter instructs Vertex AI Pipelines to cache the outputs of pipeline steps that have successfully completed. This means that if a step's inputs haven't changed, its execution can be skipped, reusing the cached output instead. Targets costly steps: The prompt highlights that data export and preprocessing steps are particularly expensive. Caching these steps can significantly reduce costs during model iterations.
Show 11 more comments
  1. shadz10 👍 2 Answered C
    Not A - Changing file names does not help with reducing costs Not B - you cannot directly use kubeflow.v1.caching on a pipeline that uses the KubeFlow v2 API. Version Incompatibility: The kubeflow.v1.caching module is specifically designed for KubeFlow Pipelines v1, and its structure and functionality are not directly compatible with KubeFlow Pipelines v2. so best option here is C
  2. BlehMaks 👍 2 Answered A
    i think it's A. 1)if we want to use the same results several times we shouldn't rename them. so we need to delete {dt} from the first two components names. 2)we already have this option enable_caching = True, why do we need kubeflow.v1.caching then? 3)i'm not sure but may be it does metter
  3. Foxy2021 👍 1
    I select B. A: Changing the YAML filenames does not affect caching behavior or cost reduction. The pipeline's efficiency and cost effectiveness are primarily governed by how it handles inputs and outputs rather than the filenames of the components. C: Moving the first step to a separate pipeline may help with organization but doesn’t directly address the cost incurred by repeated data exports and preprocessing. Also, simply providing a cached path does not guarantee that the preprocessing step itself won’t be executed multiple times. D: Changing the name of the pipeline to include a timestamp or other identifier does not influence caching or resource usage. It merely alters the identification of the pipeline runs without any impact on the efficiency of the operations being performed.
  4. lance_yu 👍 1 Answered C
    The most cost-effective approach is to decouple the stable data-preparation stages from the iterative model-training stages: Run export (and potentially preprocessing) once. Store the resulting dataset in Cloud Storage. Use that stored artifact as an input to the training pipeline. During model iteration, only train and calibrate need to run. This avoids repeatedly scanning and exporting the large BigQuery table, which is the primary source of cost.
  5. Sivaram06 👍 1 Answered B
    Adding caching to your pipeline by setting the parameter {"kubeflow.v1.caching": True} is the most efficient and effective approach to reduce model development costs, particularly for steps like data export and preprocessing, which are often time-consuming and costly to repeat during multiple iterations. This will help you avoid unnecessary re-computation and save on resource usage.
  6. b1a8fae 👍 1 Answered C
    I considered B but a search of "kubeflow.v1.caching" on Google only produces 1 result, which is this very question on this very website. Thus, I rule it out as non-existent (please share a resource if there is any that proves it exists) and opt for C.
  7. HaroonRaizada01 👍 1 Answered B
    Use Option B ({"kubeflow.v1.caching": True}) to enable caching in your Vertex AI pipeline. This is the most efficient and cost-effective way to avoid redundant executions of expensive steps like data export and preprocessing.
  8. Foxy2021 👍 1
    I select C: By leveraging a Dataproc cluster, you can maintain compatibility with your existing PySpark jobs, minimize management overhead, and create a scalable proof of concept quickly and efficiently.
  9. BlehMaks 👍 1
    3)i'm not sure but may be it does matter that KubeFlow v2 API and kubeflow.v1.caching have different versions (v1 and v2)
  10. Yan_X 👍 1 Answered C
    C Caching should be enabled for all steps, e.g., export, preprocessing and training.
  11. gscharly 👍 1 Answered A
    see guilhermebutzke

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide