How to Migrate Apache Spark Jobs to Google Cloud

PDE By islever Editorial Last updated

Short answer

D Execute as a Dataproc Serverless job.

Question

You want to migrate an Apache Spark 3 batch job from on-premises to Google Cloud. You need to minimally change the job so that the job reads from Cloud Storage and writes the result to BigQuery. Your job is optimized for Spark, where each executor has 8 vCPU and 16 GB memory, and you want to be able to choose similar settings. You want to minimize installation and management effort to run your job. What should you do?
  1. Execute the job as part of a deployment in a new Google Kubernetes Engine cluster.
  2. Execute the job from a new Compute Engine VM.
  3. Execute the job in a new Dataproc cluster.
  4. Execute as a Dataproc Serverless job. Correct Answer

Key Takeaways

This question tests the trade-off between infrastructure control and operational effort, with the trap being the assumption that Dataproc Serverless cannot match specific executor resource requirements.

Dataproc Serverless is the preferred solution for migrating Spark batch jobs when minimizing management overhead is the priority, as it allows resource configuration without the need to manage cluster infrastructure.

Common mistake: Selecting a standard Dataproc cluster (Option C) due to the belief that Serverless does not allow granular configuration of vCPU and memory per executor.

Expert Analysis

Why the Answer Is Correct

Dataproc Serverless is designed specifically to run Spark jobs without the need to provision or manage clusters, directly addressing the requirement to minimize installation and management effort. It allows users to specify executor resources using standard Spark properties (like spark.executor.cores and spark.executor.memory), enabling the configuration of 8 vCPUs and 16 GB of memory as required.

Why the Other Options Are Wrong

Options A (GKE) and B (Compute Engine) require significant manual setup and ongoing management of the underlying infrastructure, violating the constraint to minimize effort. Option C (Dataproc Cluster) offers full control but requires managing the cluster lifecycle (creation, scaling, deletion), which involves more operational overhead than the serverless alternative.

Community Comment Notes

Community members debated the configurability of Serverless, with some noting it does not allow selecting specific machine types (Comment 4). However, top-rated comments clarified that resource settings like vCPU and memory can indeed be configured via properties (Comment 1), confirming Serverless meets the job's optimization needs while satisfying the management constraints.

Official Reference

https://cloud.google.com/dataproc-serverless/docs/concepts/properties

This link opens the official vendor documentation in a new tab.

Exam Strategy

When a question emphasizes minimizing management or installation effort, prioritize serverless or fully managed services. Verify if specific technical constraints (like resource limits) can be met through configuration properties rather than direct infrastructure control.

Community Votes

D
69%
C
31%

69% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 7 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 7

  1. chicity_de 👍 8 Answered D
    Priority is "minimize installation and management effort" which is done via Dataproc Serverless. Furthermore, with Dataproc serverless you can still specify resource settings for your job, such as the number of vCPUs and memory per executor (https://cloud.google.com/dataproc-serverless/docs/concepts/properties)
  2. gabazzzo 👍 1
    I agree that minimizing installation and management means using Dataproc Serverles. Also, Serverles can be configured with up to 16 VPU and up to 29696m of memory in for the premium tier. https://cloud.google.com/dataproc-serverless/docs/concepts/properties#:~:text=Total%20driver%20memory%20per%20driver%20core%2C%20including%20driver%20memory%20overhead%2C%20which%20must%20be%20between%201024m%20and%207424m%20for%20the%20Standard%20compute%20tier%20(24576m%20for%20the%20Premium%20compute%20tier).%20For%20example%2C%20if%20spark.driver.cores%20%3D%204%2C%20then%204096m%20%3C%3D%20spark.driver.memory%20%2B%20spark.driver.memoryOverhead%20%3C%3D%2029696m.
  3. mcdaley 👍 1 Answered C
    Dataproc supports Spark 3, ensuring compatibility with your existing job. It also allows you to customize the cluster configuration, including the number of executors, vCPUs, and memory per executor, to match your on-premises setup (8 vCPU and 16 GB memory)
  4. Pime13 👍 1 Answered D
    Dataproc Serverless allows you to run Spark jobs without needing to manage the underlying infrastructure. It automatically handles resource provisioning and scaling, which simplifies the process and reduces management overhead
  5. marlon.andrei 👍 1 Answered C
    I choice "C", just: "where each executor has 8 vCPU and 16 GB memory, and you want to be able to choose similar settings"
Show 2 more comments
  1. plum21 👍 1 Answered C
    It's not possible to specify a machine type using Dataproc Serverless
  2. a494e30 👍 1 Answered C
    Needs to be able to configure "similar settings"

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

Triggering Airflow DAGs Reactively in Private Cloud Composer

S-grade

How to resolve Cloud Composer 2 worker pod evictions and task failures?

S-grade

Which Google Cloud database for on-prem PostgreSQL with analytics?

S-grade

← Back to PDE Study Guide