What is the Best Way to Migrate PySpark Workloads to Google Cloud?

PMLE By islever Editorial Last updated

Short answer

C Create a Standard (1 master, 3 workers) Dataproc cluster, and run a Vertex AI Workbench notebook instance on it.

Question

You work for a startup that has multiple data science workloads. Your compute infrastructure is currently on-premises, and the data science workloads are native to PySpark. Your team plans to migrate their data science workloads to Google Cloud. You need to build a proof of concept to migrate one data science job to Google Cloud. You want to propose a migration process that requires minimal cost and effort. What should you do first?
  1. Create a n2-standard-4 VM instance and install Java, Scala, and Apache Spark dependencies on it.
  2. Create a Google Kubernetes Engine cluster with a basic node pool configuration, install Java, Scala, and Apache Spark dependencies on it.
  3. Create a Standard (1 master, 3 workers) Dataproc cluster, and run a Vertex AI Workbench notebook instance on it. Correct Answer
  4. Create a Vertex AI Workbench notebook with instance type n2-standard-4.

Key Takeaways

Assesses your ability to distinguish between general-purpose ML environments and purpose-built big data platforms, with the common trap favoring pre-configured notebooks over distributed cluster architectures.

This scenario evaluates the optimal strategy for migrating on-premises Apache Spark jobs to Google Cloud while minimizing initial overhead. The community consensus confirms that leveraging Dataproc ensures a managed, scalable, and cost-effective proof of concept.

Common mistake: Option D is frequently selected because Vertex AI Workbench notebooks include PySpark out-of-the-box, but they fail to provide the dedicated distributed worker nodes necessary for a realistic Spark migration validation.

Expert Analysis

Why the Answer Is Correct

Creating a Dataproc cluster directly addresses the requirement to migrate native PySpark workloads using Google’s recommended architecture. Dataproc is a fully managed service optimized for Spark, eliminating the need for manual dependency installation and ongoing maintenance. Pairing it with a Vertex AI Workbench notebook allows data scientists to interact with the cluster seamlessly during the proof of concept, balancing low initial effort with production-ready scalability.

Why the Other Options Are Wrong

Option A and Option B require significant manual configuration of operating systems, language runtimes, and Spark binaries, which contradicts the minimal effort constraint. While GKE could host Spark containers, it introduces unnecessary orchestration complexity for a straightforward migration POC. Option D provides a convenient notebook interface but lacks the underlying distributed compute topology needed to accurately simulate and validate a real-world Spark workload migration.

Community Comment Notes

Several candidates debated between C and D, noting that Workbench notebooks can technically run PySpark locally or connect to Dataproc. However, commenters like [1] and [8] emphasized that Dataproc remains the industry-standard foundation for Spark migrations due to its managed scaling and cost-effective pay-per-use model. Comment [11] correctly highlighted that while D works for quick testing, C represents the proper architectural pattern for validating a production migration path.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When designing cloud migration proofs of concept, always start with Google’s certified architecture templates rather than building custom infrastructure. Prioritize managed services that natively support your workload’s core technology to reduce operational overhead and accelerate time-to-value.

Community Votes

C
64%
D
36%

64% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 17 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 17

  1. guilhermebutzke 👍 4 Answered C
    My answer: C C: This option leverages Google Cloud's Dataproc service, which is designed for running Apache Spark and other big data processing frameworks. By creating a Standard Dataproc cluster, you can easily scale resources as needed for your workload. A. n2-standard-4 VM: This requires manual setup and ongoing maintenance, increasing cost and effort. B. GKE cluster: While offering containerization benefits, it necessitates managing containers and Spark configurations, adding complexity. D. With Vertex AI Workbench, your team can develop, train, and deploy machine learning models using popular frameworks like TensorFlow, PyTorch, and scikit-learn. However, while Vertex AI Workbench supports PySpark, it may not be the optimal choice for migrating existing PySpark workloads, as it's primarily focused on machine learning tasks.
  2. TanTran04 👍 2 Answered C
    I'm following option C. Please take a look the concept of 'Dataproc documentation' (ref: https://cloud.google.com/dataproc/docs) With option D: doesn't provide a solution for managing and scaling the Spark environment, which is necessary for running PySpark workloads.
  3. ddogg 👍 2 Answered C
    Agree with BlehMaks https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview Dataproc cluster seems more suitable
  4. Carlose2108 👍 2 Answered C
    I went with C. For Proof Of Concept and requires minimal cost and effort. Furthermore, Vertex AI Workbench notebooks come pre-configured with PySpark.
  5. BlehMaks 👍 2 Answered C
    https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview
Show 12 more comments
  1. gscharly 👍 2 Answered D
    went with D: https://cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled
  2. Yan_X 👍 2 Answered D
    D Can use Notebook pre-installed libraries and tools, including PySpark.
  3. lunalongo 👍 1 Answered C
    C is the right answer because it ensures: Cost-effectiveness: Dataproc is managed and you only pay for the compute time used, which is cost-effective for a POC. A standard cluster is enough for the task. Ease of use: Dataproc simplifies the process of setting up and managing a Spark cluster Minimal effort: a Dataproc cluster + a Vertex AI Workbench instance is a straightforward process through the console or command-line tools, minimizing setup time and effort compared to manually configuring VMs or Kubernetes clusters. *A and B include manual installation steps; D creates a notebook environment but it's not enough to run a PySpark job.
  4. pikachu007 👍 1 Answered D
    Minimal setup: Vertex AI Workbench notebooks come pre-configured with PySpark and other data science tools, eliminating the need for manual installation and setup. Cost-effectiveness: Vertex AI Workbench offers managed notebooks with pay-as-you-go pricing, making it a cost-efficient option for proof-of-concept testing. Ease of use: Data scientists can directly run PySpark code in the notebook without managing infrastructure, streamlining the migration process. Scalability: Vertex AI Workbench can easily scale to handle larger workloads or multiple users if the proof-of-concept is successful.
  5. fitri001 👍 1 Answered D
    Vertex AI Workbench notebook: This option provides a pre-configured environment with popular data science libraries like PySpark already installed. It allows you to focus on migrating your PySpark code with minimal changes. n2-standard-4 instance type: This is a general-purpose machine type suitable for various data science tasks. It offers a good balance between cost and performance for initial exploration.
  6. baimus 👍 1 Answered C
    C and D are both valid, as people point out you can technically have Spark preinstalled on D. But this is for a proof of concept for the real design. The concept is not proved by using a notebook, as it's not best practice. Therefore C makes more sense, and is still low effort as it's managed.
  7. pinimichele01 👍 1 Answered C
    When you want to move your Apache Spark workloads from an on-premises environment to Google Cloud, we recommend using Dataproc to run Apache Spark/Apache Hadoop clusters. https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview
  8. shadz10 👍 1 Answered D
    https://cloud.google.com/vertex-ai-notebooks?hl=en Data Data Lake and Spark in one place Whether you use TensorFlow, PyTorch, or Spark, you can run any engine from Vertex AI Workbench. D is correct
  9. f084277 👍 1 Answered D
    D. "minimal cost and effort". There's only one answer.
  10. DaleR 👍 1
    D. Just ran a pilot on Workbench
  11. Carlose2108 👍 1 Answered D
    My bad, I mean is Option D.
  12. AK2020 👍 1 Answered C
    C is the answer

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

How to preprocess BigQuery data for TensorFlow?

S-grade

How to Track Model and Prediction Lineage in Vertex AI?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide