How to Orchestrate Dataproc Serverless and Vertex AI Training?

You have been tasked with deploying prototype code to production. The feature engineering code is in PySpark and runs on Dataproc Serverless. The model training is executed by using a Vertex AI custom training job. The two steps are not connected, and the model training must currently be run manually after the feature engineering step finishes. You need to create a scalable and maintainable production process that runs end-to-end and tracks the connections between steps. What should you do?

Create a Vertex AI Workbench notebook. Use the notebook to submit the Dataproc Serverless feature engineering job. Use the same notebook to submit the custom model training job. Run the notebook cells sequentially to tie the steps together end-to-end.
Create a Vertex AI Workbench notebook. Initiate an Apache Spark context in the notebook and run the PySpark feature engineering code. Use the same notebook to run the custom model training job in TensorFlow. Run the notebook cells sequentially to tie the steps together end-to-end.
Use the Kubeflow pipelines SDK to write code that specifies two components: Source Reference Answer
Use the Kubeflow pipelines SDK to write code that specifies two components

Community Votes

100%

100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

This question tests the ability to distinguish between prototyping tools (Notebooks) and production orchestration tools (Vertex AI Pipelines), with the common trap being selecting a manual notebook approach for a scalable end-to-end process.

To create a scalable and maintainable production process connecting Dataproc Serverless and Vertex AI training, use Vertex AI Pipelines. The community consensus confirms that the Kubeflow Pipelines SDK is the standard solution for orchestrating these components end-to-end.

Choosing Option A or B (Vertex AI Workbench) is the most common mistake because while notebooks are useful for development, they are not designed for scalable, automated, and maintainable production orchestration.

Community Discussion (7 comments)

Akel123 👍 2 Selected: C

The first is a Dataproc Serverless component that launches the feature engineering job The second is a custom component wrapped in the create_custom_training_job_from_component utility that launches the custom model training job Create a Vertex AI Pipelines job to link and run both components

fitri001 👍 1 Selected: C

CHARLIE2108 👍 1 Selected: C

I went with C

kalle_balle 👍 1 Selected: C

Vote C

36bdc1e 👍 1

C The best option for creating a scalable and maintainable production process that runs end-to-end and tracks the connections between steps, using prototype code to production, feature engineering code in PySpark that runs on Dataproc Serverless, and model training that is executed by using a Vertex AI custom training job, is to use the Kubeflow pipelines SDK to write code that specifies two components. The first is a Dataproc Serverless component that launches the feature engineering job. The second is a custom component wrapped in the create_custom_training_job_from_component utility that launches the custom model training job. This option allows you to leverage the power and simplicity of Kubeflow pipelines to orchestrate and automate your machine learning workflows on Vertex AI. Kubeflow pipelines is a platform that can build, deploy, and manage machine learning pipelines on Kubernetes.

pikachu007 👍 3 Selected: C

By using Kubeflow Pipelines, you establish a structured, scalable, and maintainable production process for end-to-end model development and deployment, ensuring proper orchestration, tracking, and integration with the chosen services.

vale_76_na_xxx 👍 1

I go for C

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

Option C is correct because using the Kubeflow Pipelines SDK allows you to define a Vertex AI Pipeline that orchestrates both the Dataproc Serverless feature engineering job and the Vertex AI custom training job. This approach provides the scalability, maintainability, and automatic lineage tracking required for a production environment, unlike manual execution.

Why the Other Options Are Wrong

Options A and B suggest using Vertex AI Workbench notebooks. While notebooks are excellent for prototyping and experimentation, they are interactive environments that require manual intervention to run cells sequentially. They lack the robust scheduling, retry logic, and metadata tracking features inherent to managed pipeline services like Vertex AI Pipelines.

Community Comment Notes

Community members unanimously voted for Option C, emphasizing that Kubeflow Pipelines establishes a structured process for production. Useful comments specifically noted the use of a Dataproc Serverless component and the create_custom_training_job_from_component utility to link the steps effectively.

Official Reference

Exam Strategy

When you encounter keywords like 'production', 'scalable', or 'end-to-end' in an exam question, immediately eliminate notebook-based solutions (like Vertex AI Workbench) in favor of orchestration tools like Vertex AI Pipelines. Focus on identifying the service that automates workflow execution and tracks metadata.

Related Analysis

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide