Vertex AI Pipelines Components for Custom Model Training

PMLE By islever Editorial Last updated

Developing end-to-end ML pipelines. Training models.

Short answer

C Use dsl.component to wrap the Python preprocessing functions, dsl.ParallelFor to handle batch iterations, and CustomTrainingJobOp for the custom training task.

Question

You are building a custom image classification model and plan to use Vertex AI Pipelines to implement the end-to-end training. Your dataset consists of images that need to be preprocessed before they can be used to train the model. The preprocessing steps include resizing the images, converting them to grayscale, and extracting features. You have already implemented some Python functions for the preprocessing tasks. Which components should you use in your pipeline?
  1. DataprocSparkBatchOp and CustomTrainingJobOp
  2. DataflowPythonJobOp, WaitGcpResourcesOp, and CustomTrainingJobOp
  3. dsl.ParallelFor, dsl.component, and CustomTrainingJobOp Correct Answer
  4. ImageDatasetImportDataOp, dsl.component, and AutoMLImageTrainingJobRunOp

Key Takeaways

The trap is choosing managed services like Dataflow or Dataproc for simple preprocessing when the prompt specifies using already implemented Python functions, which map directly to dsl.component.

Learn how to integrate existing Python functions into Vertex AI Pipelines using dsl.component and orchestrate them with dsl.ParallelFor for custom model training.

Common mistake: Choosing B (DataflowPythonJobOp) because it sounds like a 'managed' solution for data processing, ignoring that the user already has Python functions that need to be wrapped as components rather than spun up as a separate Dataflow job.

Expert Analysis

The scenario requires implementing an end-to-end training pipeline in Vertex AI Pipelines where existing Python functions must be integrated for preprocessing. The core mechanism involves wrapping these custom scripts into reusable components and orchestrating the workflow.

First, to incorporate the pre-existing Python functions (resizing, grayscale conversion, feature extraction), you must use dsl.component. This decorator wraps standard Python code into a component that can run within the Kubeflow Pipelines SDK environment. While some comments suggest using DataflowPythonJobOp for parallelization, this is generally reserved for large-scale batch processing jobs managed by Google Cloud Dataflow, not simple image manipulation tasks which are better handled as lightweight containerized components or via dsl.ParallelFor for batching.

Second, regarding orchestration, dsl.ParallelFor allows for iterating over datasets or batches of images efficiently within the pipeline graph. However, the question asks for the components to use in the pipeline structure given the Python functions. The most direct translation of "implemented Python functions" into a Vertex AI Pipeline is dsl.component.

Third, for the training phase, since the goal is to build a custom image classification model (not using AutoML), you must use CustomTrainingJobOp. Options suggesting AutoMLImageTrainingJobRunOp (Option D) are incorrect because they contradict the requirement for a custom model. Option A suggests DataprocSparkBatchOp, which is overly heavy for image preprocessing unless specifically dealing with Spark-compatible data at scale, and lacks the explicit integration of the provided Python functions via dsl.component.

Option C correctly identifies dsl.ParallelFor (for handling multiple images/batches), dsl.component (to wrap the Python logic), and CustomTrainingJobOp (for the custom training job). This combination provides the necessary flexibility and control for a custom ML workflow using existing code.

Exam Strategy

When a question mentions 'existing Python functions' in the context of Vertex AI Pipelines, immediately look for dsl.component as the integration method. Avoid selecting options that spin up separate heavy infrastructure (like Dataproc/Dataflow) unless the scale or specific technology requirements explicitly demand it, as dsl.component is the native way to embed custom logic into the pipeline DAG.

Frequently Asked Questions

What is the difference between dsl.component and DataflowPythonJobOp?

dsl.component wraps Python code into a lightweight step within the pipeline DAG, suitable for general computation. DataflowPythonJobOp launches a full Dataflow job, which is heavier and intended for large-scale distributed data processing.

Why not use AutoMLImageTrainingJobRunOp?

The question specifies building a 'custom' image classification model. AutoML is a fully automated service that does not allow for custom training code or specific preprocessing steps defined by the user.

More PMLE FAQ →

Community Votes

B
83%
C
17%

83% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 6 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 6

  1. guilhermebutzke 👍 8 Answered B
    My Answer: B Looking for the options, DataflowPythonJobOp can be used for parallelizing the preprocessing tasks, which is suitable for image resizing, converting to grayscale, and extracting features. dsl.ParallelFor could be useful for parallelizing tasks but might not be the most straightforward option for image preprocessing. Generally DataflowPythonJobOp is followed by WaitGcpResourcesOp. https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/fe7d3e4b8edc137d90ec061789b879b7cc8d3854/notebooks/community/ml_ops/stage3/get_started_with_dataflow_flex_template_component.ipynb
  2. pikachu007 👍 2 Answered B
    A. DataprocSparkBatchOp: While capable of data processing, it's less well-suited for image-specific tasks like resizing and grayscale conversion compared to DataflowPythonJobOp. C. dsl.ParallelFor, dsl.component: While offering flexibility, they require more manual orchestration and potentially less efficient for image preprocessing compared to DataflowPythonJobOp. D. ImageDatasetImportDataOp, AutoMLImageTrainingJobRunOp: These components are designed for AutoML Image training, not directly compatible with custom preprocessing and training tasks.
  3. b1a8fae 👍 2 Answered B
    I go with B. Custom training is surely required. Discarding A because Spark is not mentioned anywhere in the problem description. C involves Kubeflow which seems a bit overkill imo. DataflowPythonJobOp operator lets you create a Vertex AI Pipelines component that prepares data -> seems like the appropriate course of action to me. https://cloud.google.com/vertex-ai/docs/pipelines/dataflow-component#dataflowpythonjobop
  4. Dirtie_Sinkie 👍 2 Answered B
    B is definitely right, no doubt
  5. lance_yu 👍 1 Answered C
    In Vertex AI Pipelines (Kubeflow Pipelines SDK): dsl.component is the standard way to wrap existing Python functions into reusable pipeline components. dsl.ParallelFor can be used to parallelize preprocessing across image batches or data partitions. CustomTrainingJobOp is used to launch the custom model training job on Vertex AI Training. This provides a fully managed, scalable, and reusable pipeline while leveraging your existing Python code.
Show 1 more comment
  1. pinimichele01 👍 1 Answered B
    https://cloud.google.com/vertex-ai/docs/pipelines/dataflow-component#dataflowpythonjobop

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

Building CLV Models with BigQuery ML in Vertex AI Workbench

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide