Migrating Hadoop Spark Jobs to Dataproc Serverless and Cloud Storage

PDE By islever Editorial Last updated

Building the pipelines. Selecting storage systems. Designing automation and repeatability.

Short answer

A Migrate data to Cloud Storage, use Dataproc Metastore for metadata, refactor Spark pipelines to interact with Cloud Storage, and run them on Dataproc Serverless.

Question

Your organization stores customer data in an on-premises Apache Hadoop cluster in Apache Parquet format. Data is processed on a daily basis by Apache Spark jobs that run on the cluster. You are migrating the Spark jobs and Parquet data to Google Cloud. BigQuery will be used on future transformation pipelines so you need to ensure that your data is available in BigQuery. You want to use managed services, while minimizing ETL data processing changes and overhead costs. What should you do?
  1. Migrate your data to Cloud Storage and migrate the metadata to Dataproc Metastore (DPMS). Refactor Spark pipelines to write and read data on Cloud Storage, and run them on Dataproc Serverless. Correct Answer
  2. Migrate your data to Cloud Storage and register the bucket as a Dataplex asset. Refactor Spark pipelines to write and read data on Cloud Storage, and run them on Dataproc Serverless.
  3. Migrate your data to BigQuery. Refactor Spark pipelines to write and read data on BigQuery, and run them on Dataproc Serverless.
  4. Migrate your data to BigLake. Refactor Spark pipelines to write and read data on Cloud Storage, and run them on Dataproc on Compute Engine.

Key Takeaways

The trap is thinking 'available in BigQuery' means 'stored in BigQuery'. Using Cloud Storage with BigQuery External Tables allows Spark to keep working on Parquet files while still being accessible to BigQuery.

Learn how to migrate on-premises Hadoop/Spark workloads to Google Cloud by using Cloud Storage for data and Dataproc Serverless for computation to minimize ETL changes.

Common mistake: Candidates often choose C because they associate BigQuery with 'modern analytics'. However, forcing Spark to write to BigQuery requires significant refactoring and increases costs, violating the 'minimize changes' constraint.

Expert Analysis

Managed Spark with External Data The question requires migrating on-premises Hadoop/Spark workloads to Google Cloud while minimizing ETL changes and ensuring future BigQuery accessibility. The correct approach is to keep the raw Parquet data in Cloud Storage (a managed object store) and run the existing Spark jobs on Dataproc Serverless (a managed compute service). This satisfies the "managed services" requirement without the operational overhead of managing Compute Engine instances.

Minimizing Refactoring Option C suggests migrating data directly to BigQuery. While BigQuery is managed, Spark jobs are not natively optimized for direct BigQuery read/write as their primary storage layer without significant code changes (using connectors). More importantly, the prompt emphasizes minimizing ETL processing changes. Moving Parquet files to Cloud Storage allows Spark to continue reading/writing Parquet files exactly as it did in HDFS, requiring only minor path adjustments. This aligns with the goal of minimal refactoring.

BigQuery Integration via External Tables The requirement to ensure data is available in BigQuery does not mean the data must reside inside BigQuery tables. By keeping Parquet files in Cloud Storage, you can create BigQuery external tables that query the data in place. This provides immediate BigQuery compatibility for future pipelines without moving or duplicating the data. Option A explicitly mentions using Dataproc Metastore for metadata, which helps manage the Hive-like schema if needed, but the core value is the combination of Cloud Storage + Dataproc Serverless + External Tables.

Why Other Options Fail

  • Option B adds Dataplex, which is a governance tool, not a necessity for basic migration and minimization of change. It adds complexity.
  • Option D uses Dataproc on Compute Engine, which violates the "minimize overhead costs" and "managed services" preference compared to Serverless. It also suggests BigLake, which is overkill when simple Cloud Storage works.

Operational Consequence Choosing C would force a rewrite of Spark logic to use BigQuery connectors, increasing cost and risk. Choosing A preserves the existing Spark logic, leverages fully managed infrastructure, and enables BigQuery access via external tables, perfectly balancing all constraints.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When a question asks to 'minimize ETL changes' and 'use managed services', look for options that separate storage from compute. Keep raw data in object storage (Cloud Storage) where Spark can read it efficiently, and use a managed compute platform (Dataproc Serverless). If BigQuery accessibility is required, check if 'external tables' or 'connectors' are implied rather than full data migration into BigQuery tables.

Frequently Asked Questions

Why not move data directly to BigQuery (Option C)?

Moving data to BigQuery requires refactoring Spark jobs to use BigQuery connectors, which contradicts the goal of minimizing ETL changes. Additionally, Spark is optimized for file-based storage like Cloud Storage, not necessarily for direct BigQuery table writes as a primary workflow.

Does Dataproc Metastore help with this migration?

Yes, Dataproc Metastore manages Hive metadata. Since the on-prem Hadoop cluster likely used Hive tables, migrating the metadata ensures that the new Dataproc environment understands the schema of the Parquet files in Cloud Storage without needing to redefine everything.

What is the benefit of Dataproc Serverless over Dataproc on Compute Engine?

Dataproc Serverless is fully managed, meaning you don't provision or manage VMs. This reduces operational overhead and costs compared to running Dataproc on Compute Engine, aligning with the 'minimize overhead' requirement.

More PDE FAQ →

Community Votes

A
44%
C
28%
B
22%
D
6%

44% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 22 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 22

  1. raaad 👍 7 Answered A
    - This option involves moving Parquet files to Cloud Storage, which is a common and cost-effective storage solution for big data and is compatible with Spark jobs. - Using Dataproc Metastore to manage metadata allows us to keep Hadoop ecosystem's structural information. - Running Spark jobs on Dataproc Serverless takes advantage of managed Spark services without managing clusters. - Once the data is in Cloud Storage, you can also easily load it into BigQuery for further analysis.
  2. skhaire 👍 6 Answered B
    BigQuery Integration: The requirement is to make data available in BigQuery. Dataplex has built-in integration with BigQuery. It can automatically discover data in Cloud Storage and create external tables in BigQuery, making the data readily queryable. DPMS doesn't have this direct integration with BigQuery.
  3. Ramon98 👍 4 Answered C
    A tricky one, because of "you need to ensure that your data is available in BigQuery". The easiest and most straight forward migration seems answer A to me, and then you can use external tables to make the parquet data directly available in BigQuery. https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet However creating the external tables is an extra step? So therefore maybe C is the answer?
  4. matiijax 👍 4 Answered B
    I think its B and the reason is that egistering the data as a Dataplex asset enables seamless integration with BigQuery later on. Dataplex simplifies data discovery and lineage tracking, making it easier to prepare your data for BigQuery transformations.
  5. hrishi19 👍 4 Answered C
    The question states that the data should be available on BigQuery and only option C meets this requirement.
Show 17 more comments
  1. Anudeep58 👍 3 Answered A
    Option B: Registering the bucket as a Dataplex asset adds an additional layer of data governance and management. While useful, it may not be necessary for your immediate migration needs and can introduce additional complexity. Option C: Migrating data directly to BigQuery would require significant changes to your Spark pipelines since they would need to be refactored to read from and write to BigQuery instead of Parquet files. This approach could introduce higher costs due to BigQuery storage and querying. Option D: Using BigLake and Dataproc on Compute Engine is more complex and requires more management compared to Dataproc Serverless. Additionally, it might not be as cost-effective as leveraging Cloud Storage and Dataproc Serverless.
  2. JyoGCP 👍 3 Answered C
    I think it's C. Dataproc can use BigQuery to read and write data. Dataproc's BigQuery connector is a library that allows Spark and Hadoop applications to process and write data from BigQuery. Here's how Dataproc can be used with BigQuery: Process large datasets: Use Spark to process data stored in BigQuery. Write results: Write the results back to BigQuery or other data storage for further analysis. Read data: The BigQuery connector can read data from BigQuery into a Spark DataFrame. Write data: The connector writes data to BigQuery by buffering all the data into a Cloud Storage temporary table.
  3. e70ea9e 👍 3 Answered A
    Managed Services: Leverages Dataproc Serverless for a fully managed Spark environment, reducing overhead and administrative tasks. Minimal Data Processing Changes: Keeps Spark pipelines largely intact by working with Parquet files on Cloud Storage, minimizing refactoring efforts. BigQuery Integration: Dataproc Serverless can directly access BigQuery, enabling future transformation pipelines without additional data movement. Cost-Effective: Serverless model scales resources only when needed, optimizing costs for intermittent workloads.
  4. 52ed0e5 👍 3 Answered C
    Migrate your data directly to BigQuery. Refactor Spark pipelines to read from and write to BigQuery. Run the Spark jobs on Dataproc Serverless. The best choice for ensuring data availability in BigQuery. It allows seamless integration with BigQuery and minimizes ETL changes.
  5. JyoGCP 👍 3
    As per question.. "BigQuery will be used on future transformation pipelines so you need to ensure that your data is available in BigQuery. You want to use managed services (DATAPROC), while minimizing ETL data processing changes and overhead costs."
  6. saschak94 👍 3
    Why would I select A here? Why not moving the data to BigQuery and running Dataproc Serverless jobs accessing the data in BigQuery?
  7. aoifneofi_ef 👍 2
    Just adding further commentary on why A is correct while why other options are incorrect is explained above. Parquet files have schema engrained in them. Hence Spark pipelines on Hadoop Cluster may not have needed tables at all. Hence the simplest solution would be to move it to Cloud Storage instead of BigQuery and this way there would be minimal changes to the ETL pipelines - just change HDFS file system pointer to GCS file system for read writes and no need for any additional tables
  8. Positron75 👍 2
    How is Dataproc Serverless not a managed service, but running Dataproc on Compute Engine is? D is the first answer to rule out.
  9. carbonscave 👍 1 Answered B
    Why B is the right move The Dataplex Bridge: The hardest requirement in this prompt is ensuring the data is available to both the existing Spark jobs and the future BigQuery pipelines without massive re-engineering. When you register a Cloud Storage bucket as a Dataplex asset, Dataplex automatically crawls the Parquet files, discovers the schema, and publishes the metadata to both Dataproc Metastore (for Spark) and BigQuery (as external tables). The data sits in one place, but both engines can query it instantly. Minimizing ETL Changes: By keeping the data in its native Apache Parquet format and moving it to Cloud Storage, the only change required for your existing Spark code is updating the file paths from hdfs:// to gs://. The fundamental read/write logic remains exactly the same. Minimizing Overhead Cost: Dataproc Serverless completely eliminates the need to provision, manage, patch, or tune a Hadoop cluster. You just submit the Spark job, and Google handles the underlying compute infrastructure.
  10. plum21 👍 1 Answered D
    The requirement: "You want to use managed services" excludes Dataproc Serverless. Dataproc on Compute Engine remains. Next requirement: "BigQuery will be used on future transformation pipelines so you need to ensure that your data is available in BigQuery" -> BigLake Next requirement: "while minimizing ETL data processing changes and overhead costs" -> Refactor Spark pipelines to write and read data on Cloud Storage Notes 1. Dataproc Metastore (DPMS) could be used on Dataproc to read data from BQ but not the other way round.
  11. josech 👍 1 Answered A
    The question says "You want to use managed services, while minimizing ETL data processing changes and overhead costs". Dataproc is a managed service that doesn't need to refactor the data transformation Spark code you already have (you will have to refactor only the wrtie and read code), an it has a Big Query connector for future use. https://cloud.google.com/dataproc/docs/concepts/connectors/bigquery
  12. 380e3c6 👍 1 Answered A
    A is correctbecause it minimizes ETL changes, keeps Parquet data in Cloud Storage (cost-effective and Spark-compatible), and integrates with BigQuery via external tables. C is flawed** since moving directly to BigQuery requires refactoring Spark jobs, increasing complexity and costs. B adds unnecessary governance overhead, and D focuses on infrastructure instead of pipeline efficiency.
  13. 06bbd0e 👍 1 Answered B
    Cloud Storage replaces HDFS. Dataplex (or Knowledge Catalog) replaces the manual metadata work. Dataproc Serverless replaces the cluster management headache.
  14. Moss2011 👍 1 Answered C
    I think the key phrase here is "you need to ensure that your data is available in BigQuery" that's why I thing C it's the best option
  15. LP_PDE 👍 1 Answered A
    Both Spark and BigQuery can directly access data in Cloud Storage.
  16. 22c1725 👍 1 Answered A
    Would Go with A
  17. JamesKarianis 👍 1 Answered A
    A is correct

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

Triggering Airflow DAGs Reactively in Private Cloud Composer

S-grade

Bigtable Row Key Design for Time-Series Data

S-grade

Monitoring Cloud Bigtable for Cluster Scaling Decisions

S-grade

← Back to PDE Study Guide