How to Migrate Apache Spark Jobs to Google Cloud

You want to migrate an Apache Spark 3 batch job from on-premises to Google Cloud. You need to minimally change the job so that the job reads from Cloud Storage and writes the result to BigQuery. Your job is optimized for Spark, where each executor has 8 vCPU and 16 GB memory, and you want to be able to choose similar settings. You want to minimize installation and management effort to run your job. What should you do?

Execute the job as part of a deployment in a new Google Kubernetes Engine cluster.
Execute the job from a new Compute Engine VM.
Execute the job in a new Dataproc cluster.
Execute as a Dataproc Serverless job. Source Reference Answer

Community Votes

69%

31%

69% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

This question tests the trade-off between infrastructure control and operational effort, with the trap being the assumption that Dataproc Serverless cannot match specific executor resource requirements.

Dataproc Serverless is the preferred solution for migrating Spark batch jobs when minimizing management overhead is the priority, as it allows resource configuration without the need to manage cluster infrastructure.

Selecting a standard Dataproc cluster (Option C) due to the belief that Serverless does not allow granular configuration of vCPU and memory per executor.

Community Discussion (7 comments)

chicity_de 👍 8 Selected: D

Priority is "minimize installation and management effort" which is done via Dataproc Serverless. Furthermore, with Dataproc serverless you can still specify resource settings for your job, such as the number of vCPUs and memory per executor (https://cloud.google.com/dataproc-serverless/docs/concepts/properties)

gabazzzo 👍 1

I agree that minimizing installation and management means using Dataproc Serverles. Also, Serverles can be configured with up to 16 VPU and up to 29696m of memory in for the premium tier. https://cloud.google.com/dataproc-serverless/docs/concepts/properties#:~:text=Total%20driver%20memory%20per%20driver%20core%2C%20including%20driver%20memory%20overhead%2C%20which%20must%20be%20between%201024m%20and%207424m%20for%20the%20Standard%20compute%20tier%20(24576m%20for%20the%20Premium%20compute%20tier).%20For%20example%2C%20if%20spark.driver.cores%20%3D%204%2C%20then%204096m%20%3C%3D%20spark.driver.memory%20%2B%20spark.driver.memoryOverhead%20%3C%3D%2029696m.

a494e30 👍 1 Selected: C

Needs to be able to configure "similar settings"

plum21 👍 1 Selected: C

It's not possible to specify a machine type using Dataproc Serverless

marlon.andrei 👍 1 Selected: C

I choice "C", just: "where each executor has 8 vCPU and 16 GB memory, and you want to be able to choose similar settings"

Pime13 👍 1 Selected: D

Dataproc Serverless allows you to run Spark jobs without needing to manage the underlying infrastructure. It automatically handles resource provisioning and scaling, which simplifies the process and reduces management overhead

mcdaley 👍 1 Selected: C

Dataproc supports Spark 3, ensuring compatibility with your existing job. It also allows you to customize the cluster configuration, including the number of executors, vCPUs, and memory per executor, to match your on-premises setup (8 vCPU and 16 GB memory)

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

Dataproc Serverless is designed specifically to run Spark jobs without the need to provision or manage clusters, directly addressing the requirement to minimize installation and management effort. It allows users to specify executor resources using standard Spark properties (like spark.executor.cores and spark.executor.memory), enabling the configuration of 8 vCPUs and 16 GB of memory as required.

Why the Other Options Are Wrong

Options A (GKE) and B (Compute Engine) require significant manual setup and ongoing management of the underlying infrastructure, violating the constraint to minimize effort. Option C (Dataproc Cluster) offers full control but requires managing the cluster lifecycle (creation, scaling, deletion), which involves more operational overhead than the serverless alternative.

Community Comment Notes

Community members debated the configurability of Serverless, with some noting it does not allow selecting specific machine types (Comment 4). However, top-rated comments clarified that resource settings like vCPU and memory can indeed be configured via properties (Comment 1), confirming Serverless meets the job's optimization needs while satisfying the management constraints.

Official Reference

https://cloud.google.com/dataproc-serverless/docs/concepts/properties

Exam Strategy

When a question emphasizes minimizing management or installation effort, prioritize serverless or fully managed services. Verify if specific technical constraints (like resource limits) can be met through configuration properties rather than direct infrastructure control.

Related Analysis

← Back to PDE Study Guide