What is the Best Way to Migrate PySpark Workloads to Google Cloud?

You work for a startup that has multiple data science workloads. Your compute infrastructure is currently on-premises, and the data science workloads are native to PySpark. Your team plans to migrate their data science workloads to Google Cloud. You need to build a proof of concept to migrate one data science job to Google Cloud. You want to propose a migration process that requires minimal cost and effort. What should you do first?

Create a n2-standard-4 VM instance and install Java, Scala, and Apache Spark dependencies on it.
Create a Google Kubernetes Engine cluster with a basic node pool configuration, install Java, Scala, and Apache Spark dependencies on it.
Create a Standard (1 master, 3 workers) Dataproc cluster, and run a Vertex AI Workbench notebook instance on it. Source Reference Answer
Create a Vertex AI Workbench notebook with instance type n2-standard-4.

Community Votes

64%

36%

64% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

Assesses your ability to distinguish between general-purpose ML environments and purpose-built big data platforms, with the common trap favoring pre-configured notebooks over distributed cluster architectures.

This scenario evaluates the optimal strategy for migrating on-premises Apache Spark jobs to Google Cloud while minimizing initial overhead. The community consensus confirms that leveraging Dataproc ensures a managed, scalable, and cost-effective proof of concept.

Option D is frequently selected because Vertex AI Workbench notebooks include PySpark out-of-the-box, but they fail to provide the dedicated distributed worker nodes necessary for a realistic Spark migration validation.

Community Discussion (17 comments)

lunalongo 👍 1 Selected: C

C is the right answer because it ensures: Cost-effectiveness: Dataproc is managed and you only pay for the compute time used, which is cost-effective for a POC. A standard cluster is enough for the task. Ease of use: Dataproc simplifies the process of setting up and managing a Spark cluster Minimal effort: a Dataproc cluster + a Vertex AI Workbench instance is a straightforward process through the console or command-line tools, minimizing setup time and effort compared to manually configuring VMs or Kubernetes clusters. *A and B include manual installation steps; D creates a notebook environment but it's not enough to run a PySpark job.

DaleR 👍 1

D. Just ran a pilot on Workbench

f084277 👍 1 Selected: D

D. "minimal cost and effort". There's only one answer.

baimus 👍 1 Selected: C

C and D are both valid, as people point out you can technically have Spark preinstalled on D. But this is for a proof of concept for the real design. The concept is not proved by using a notebook, as it's not best practice. Therefore C makes more sense, and is still low effort as it's managed.

AK2020 👍 1 Selected: C

C is the answer

TanTran04 👍 2 Selected: C

I'm following option C. Please take a look the concept of 'Dataproc documentation' (ref: https://cloud.google.com/dataproc/docs) With option D: doesn't provide a solution for managing and scaling the Spark environment, which is necessary for running PySpark workloads.

fitri001 👍 1 Selected: D

Vertex AI Workbench notebook: This option provides a pre-configured environment with popular data science libraries like PySpark already installed. It allows you to focus on migrating your PySpark code with minimal changes. n2-standard-4 instance type: This is a general-purpose machine type suitable for various data science tasks. It offers a good balance between cost and performance for initial exploration.

gscharly 👍 2 Selected: D

went with D: https://cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled

pinimichele01 👍 1 Selected: C

When you want to move your Apache Spark workloads from an on-premises environment to Google Cloud, we recommend using Dataproc to run Apache Spark/Apache Hadoop clusters. https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview

Yan_X 👍 2 Selected: D

D Can use Notebook pre-installed libraries and tools, including PySpark.

Carlose2108 👍 1 Selected: D

My bad, I mean is Option D.

Carlose2108 👍 2 Selected: C

I went with C. For Proof Of Concept and requires minimal cost and effort. Furthermore, Vertex AI Workbench notebooks come pre-configured with PySpark.

guilhermebutzke 👍 4 Selected: C

My answer: C C: This option leverages Google Cloud's Dataproc service, which is designed for running Apache Spark and other big data processing frameworks. By creating a Standard Dataproc cluster, you can easily scale resources as needed for your workload. A. n2-standard-4 VM: This requires manual setup and ongoing maintenance, increasing cost and effort. B. GKE cluster: While offering containerization benefits, it necessitates managing containers and Spark configurations, adding complexity. D. With Vertex AI Workbench, your team can develop, train, and deploy machine learning models using popular frameworks like TensorFlow, PyTorch, and scikit-learn. However, while Vertex AI Workbench supports PySpark, it may not be the optimal choice for migrating existing PySpark workloads, as it's primarily focused on machine learning tasks.

ddogg 👍 2 Selected: C

Agree with BlehMaks https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview Dataproc cluster seems more suitable

shadz10 👍 1 Selected: D

https://cloud.google.com/vertex-ai-notebooks?hl=en Data Data Lake and Spark in one place Whether you use TensorFlow, PyTorch, or Spark, you can run any engine from Vertex AI Workbench. D is correct

BlehMaks 👍 2 Selected: C

https://cloud.google.com/architecture/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc#overview

pikachu007 👍 1 Selected: D

Minimal setup: Vertex AI Workbench notebooks come pre-configured with PySpark and other data science tools, eliminating the need for manual installation and setup. Cost-effectiveness: Vertex AI Workbench offers managed notebooks with pay-as-you-go pricing, making it a cost-efficient option for proof-of-concept testing. Ease of use: Data scientists can directly run PySpark code in the notebook without managing infrastructure, streamlining the migration process. Scalability: Vertex AI Workbench can easily scale to handle larger workloads or multiple users if the proof-of-concept is successful.

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

Creating a Dataproc cluster directly addresses the requirement to migrate native PySpark workloads using Google’s recommended architecture. Dataproc is a fully managed service optimized for Spark, eliminating the need for manual dependency installation and ongoing maintenance. Pairing it with a Vertex AI Workbench notebook allows data scientists to interact with the cluster seamlessly during the proof of concept, balancing low initial effort with production-ready scalability.

Why the Other Options Are Wrong

Option A and Option B require significant manual configuration of operating systems, language runtimes, and Spark binaries, which contradicts the minimal effort constraint. While GKE could host Spark containers, it introduces unnecessary orchestration complexity for a straightforward migration POC. Option D provides a convenient notebook interface but lacks the underlying distributed compute topology needed to accurately simulate and validate a real-world Spark workload migration.

Community Comment Notes

Several candidates debated between C and D, noting that Workbench notebooks can technically run PySpark locally or connect to Dataproc. However, commenters like [1] and [8] emphasized that Dataproc remains the industry-standard foundation for Spark migrations due to its managed scaling and cost-effective pay-per-use model. Comment [11] correctly highlighted that while D works for quick testing, C represents the proper architectural pattern for validating a production migration path.

Official Reference

Exam Strategy

When designing cloud migration proofs of concept, always start with Google’s certified architecture templates rather than building custom infrastructure. Prioritize managed services that natively support your workload’s core technology to reduce operational overhead and accelerate time-to-value.

Related Analysis

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide