Migrating PySpark Workloads to Google Cloud Dataproc

PMLE By islever Editorial Last updated

Developing end-to-end ML pipelines. Choosing appropriate hardware for training.

Short answer

C Create a Standard Dataproc cluster and run a Vertex AI Workbench notebook instance on it to execute the PySpark workload in a managed, scalable environment with minimal setup effort.

Question

You work for a startup that has multiple data science workloads. Your compute infrastructure is currently on-premises, and the data science workloads are native to PySpark. Your team plans to migrate their data science workloads to Google Cloud. You need to build a proof of concept to migrate one data science job to Google Cloud. You want to propose a migration process that requires minimal cost and effort. What should you do first?
  1. Create a n2-standard-4 VM instance and install Java, Scala, and Apache Spark dependencies on it.
  2. Create a Google Kubernetes Engine cluster with a basic node pool configuration, install Java, Scala, and Apache Spark dependencies on it.
  3. Create a Standard (1 master, 3 workers) Dataproc cluster, and run a Vertex AI Workbench notebook instance on it. Correct Answer
  4. Create a Vertex AI Workbench notebook with instance type n2-standard-4.

Key Takeaways

The main trap is confusing Vertex AI Workbench (for interactive dev) with Dataproc (for managed Spark execution); Dataproc is required to properly simulate the migrated Spark environment.

Learn why creating a Dataproc cluster is the preferred method for migrating PySpark workloads to Google Cloud compared to using standalone VMs or GKE clusters for minimal cost and effort.

Common mistake: Choosing Option D (Vertex AI Workbench) because it offers a pre-configured environment for writing PySpark code, but failing to recognize that it does not provide the managed distributed cluster architecture necessary for running the actual migrated workload.

Expert Analysis

The core concept here is the distinction between interactive development and managed cluster execution for Apache Spark workloads. The scenario involves migrating PySpark jobs with a goal of minimal cost and effort for a proof of concept (POC).

Option C is correct because Dataproc is Google Cloud’s managed Hadoop/Spark service. It allows you to run existing PySpark scripts with minimal code changes by providing a pre-configured Spark environment. A Standard cluster (1 master, 3 workers) balances cost and capability for a POC, allowing you to validate the workload in a production-like architecture without managing OS-level dependencies.

Option D is the common trap. While Vertex AI Workbench notebooks come with PySpark libraries installed, they are designed for interactive data exploration and model prototyping on a single node. Running a full PySpark job in a notebook often requires local mode or external cluster connections, which does not represent the actual migration target (a distributed Spark cluster). Furthermore, comments highlight that while D is easier for immediate coding, C represents the actual architectural migration path for Spark workloads, making it the superior choice for a POC aimed at validating the migration process itself.

Exam Strategy

When asked about migrating big data frameworks like Spark or Hadoop, always look for the managed service specific to that framework (Dataproc for Spark/Hadoop, BigQuery for SQL). Avoid options that suggest manual installation on VMs (IaaS) or container orchestration (GKE) unless the question explicitly demands custom container management or extreme scalability beyond what the managed service provides.

Frequently Asked Questions

Can I run PySpark on Vertex AI Workbench without Dataproc?

Yes, you can install PySpark libraries in a Vertex AI Workbench notebook and run them in local mode. However, this does not replicate the distributed cluster environment of your on-premises Spark jobs, making it less suitable for validating a true migration.

Is Dataproc more expensive than a simple VM?

For a POC, Dataproc’s pay-as-you-go pricing and managed nature often result in lower total effort and operational overhead compared to manually configuring and maintaining Java/Scala dependencies on a raw VM.

More PMLE FAQ →

Community Votes

C
53%
D
47%

53% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 24 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 24

  1. guilhermebutzke 👍 6 Answered C
    My answer: C C: This option leverages Google Cloud's Dataproc service, which is designed for running Apache Spark and other big data processing frameworks. By creating a Standard Dataproc cluster, you can easily scale resources as needed for your workload. A. n2-standard-4 VM: This requires manual setup and ongoing maintenance, increasing cost and effort. B. GKE cluster: While offering containerization benefits, it necessitates managing containers and Spark configurations, adding complexity. D. With Vertex AI Workbench, your team can develop, train, and deploy machine learning models using popular frameworks like TensorFlow, PyTorch, and scikit-learn. However, while Vertex AI Workbench supports PySpark, it may not be the optimal choice for migrating existing PySpark workloads, as it's primarily focused on machine learning tasks.
  2. baimus 👍 4 Answered C
    C and D are both valid, as people point out you can technically have Spark preinstalled on D. But this is for a proof of concept for the real design. The concept is not proved by using a notebook, as it's not best practice. Therefore C makes more sense, and is still low effort as it's managed.
  3. TanTran04 👍 3 Answered C
    I'm following option C. Please take a look the concept of 'Dataproc documentation' (ref: https://cloud.google.com/dataproc/docs) With option D: doesn't provide a solution for managing and scaling the Spark environment, which is necessary for running PySpark workloads.
  4. lunalongo 👍 2 Answered C
    C is the right answer because it ensures: Cost-effectiveness: Dataproc is managed and you only pay for the compute time used, which is cost-effective for a POC. A standard cluster is enough for the task. Ease of use: Dataproc simplifies the process of setting up and managing a Spark cluster Minimal effort: a Dataproc cluster + a Vertex AI Workbench instance is a straightforward process through the console or command-line tools, minimizing setup time and effort compared to manually configuring VMs or Kubernetes clusters. *A and B include manual installation steps; D creates a notebook environment but it's not enough to run a PySpark job.
  5. ddogg 👍 2 Answered C
    Agree with BlehMaks https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview Dataproc cluster seems more suitable
Show 19 more comments
  1. Carlose2108 👍 2 Answered C
    I went with C. For Proof Of Concept and requires minimal cost and effort. Furthermore, Vertex AI Workbench notebooks come pre-configured with PySpark.
  2. Carlose2108 👍 2
    You're right but I have a doubt about in a part of Option D "You need to build a proof of concept to migrate one data science job to Google Cloud"
  3. gscharly 👍 2 Answered D
    went with D: https://cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled
  4. Yan_X 👍 2 Answered D
    D Can use Notebook pre-installed libraries and tools, including PySpark.
  5. fitri001 👍 1
    A. Create a n2-standard-4 VM instance: This option requires manually installing Java, Scala, and Spark dependencies, which is time-consuming and prone to errors. It also involves managing the VM instance lifecycle, increasing complexity. B. Create a Google Kubernetes Engine cluster: Setting up and managing a Kubernetes cluster for a single job is overkill for a proof of concept. It adds unnecessary complexity and cost. C. Create a Standard Dataproc cluster: While Dataproc is a managed Spark environment on GCP, setting up a full cluster (master and workers) might be more resource-intensive than needed for a single job, especially for a proof of concept.
  6. pikachu007 👍 1 Answered D
    Minimal setup: Vertex AI Workbench notebooks come pre-configured with PySpark and other data science tools, eliminating the need for manual installation and setup. Cost-effectiveness: Vertex AI Workbench offers managed notebooks with pay-as-you-go pricing, making it a cost-efficient option for proof-of-concept testing. Ease of use: Data scientists can directly run PySpark code in the notebook without managing infrastructure, streamlining the migration process. Scalability: Vertex AI Workbench can easily scale to handle larger workloads or multiple users if the proof-of-concept is successful.
  7. taylz876 👍 1 Answered C
    The workloads are already written in PySpark, and you want a proof of concept with minimal cost and effort. Dataproc, now called Managed Service for Apache Spark (Managed Spark), is Google's managed Hadoop and Spark service. It already has Spark and PySpark installed, so your existing job can run with minimal code changes. You create the cluster in a few clicks and delete it when the POC is done, so you only pay while it runs. The Workbench notebook on top gives you an interactive place to run and test the job.
  8. lance_yu 👍 1 Answered D
    A Vertex AI Workbench instance provides a managed Jupyter notebook environment where you can: Develop and test PySpark code interactively. Install PySpark libraries as needed. Validate the migration quickly without provisioning a full cluster. Pay only for a single VM rather than maintaining a Spark or Kubernetes cluster. For a PoC, the goal is to validate functionality with the least operational overhead, not to build a production-scale Spark environment.
  9. fitri001 👍 1 Answered D
    Vertex AI Workbench notebook: This option provides a pre-configured environment with popular data science libraries like PySpark already installed. It allows you to focus on migrating your PySpark code with minimal changes. n2-standard-4 instance type: This is a general-purpose machine type suitable for various data science tasks. It offers a good balance between cost and performance for initial exploration.
  10. pinimichele01 👍 1 Answered C
    When you want to move your Apache Spark workloads from an on-premises environment to Google Cloud, we recommend using Dataproc to run Apache Spark/Apache Hadoop clusters. https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview
  11. shadz10 👍 1 Answered D
    https://cloud.google.com/vertex-ai-notebooks?hl=en Data Data Lake and Spark in one place Whether you use TensorFlow, PyTorch, or Spark, you can run any engine from Vertex AI Workbench. D is correct
  12. Jason_Cloud_at 👍 1
    Option D doesnt provide Pyspark out of the box, you have to manually install it wherelse in C dataproc is managed spark and hadoop services which supports running pyspark services right away.
  13. pinimichele01 👍 1
    https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview why not c?
  14. pinimichele01 👍 1
    https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview
  15. bigdapper 👍 1 Answered D
    D: minimal cost and effort. Can run PySpark on the notebook.
  16. f084277 👍 1 Answered D
    D. "minimal cost and effort". There's only one answer.
  17. DaleR 👍 1
    D. Just ran a pilot on Workbench
  18. Carlose2108 👍 1 Answered D
    My bad, I mean is Option D.
  19. AK2020 👍 1 Answered C
    C is the answer

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide