Cost-Effective GCP Storage for Hadoop ORC Data

PDE By islever Editorial Last updated

Selecting storage systems. Using a data lake.

Short answer

D Copy the ORC files on Cloud Storage, then create external BigQuery tables for the data scientist team.

Question

You created an analytics environment on Google Cloud so that your data scientist team can explore data without impacting the on-premises Apache Hadoop solution. The data in the on-premises Hadoop Distributed File System (HDFS) cluster is in Optimized Row Columnar (ORC) formatted files with multiple columns of Hive partitioning. The data scientist team needs to be able to explore the data in a similar way as they used the on-premises HDFS cluster with SQL on the Hive query engine. You need to choose the most cost-effective storage and processing solution. What should you do?
  1. Import the ORC files to Bigtable tables for the data scientist team.
  2. Import the ORC files to BigQuery tables for the data scientist team.
  3. Copy the ORC files on Cloud Storage, then deploy a Dataproc cluster for the data scientist team.
  4. Copy the ORC files on Cloud Storage, then create external BigQuery tables for the data scientist team. Correct Answer

Key Takeaways

The question tests the trade-off between exact Hive compatibility (Dataproc) and cost-effective serverless SQL exploration (BigQuery) for ad-hoc data science tasks.

BigQuery external tables over Cloud Storage provide the most cost-effective way to explore Hive-partitioned ORC data with SQL, avoiding the compute costs of Dataproc and the ingestion overhead of native BigQuery tables.

Common mistake: Choosing Option C because of the mention of the Hive query engine, overlooking that maintaining a Dataproc cluster for ad-hoc exploration is significantly more expensive than serverless BigQuery.

Expert Analysis

Cost-Effectiveness and Serverless Compute The central constraint is finding the most cost-effective processing solution for ad-hoc data exploration. While a Dataproc cluster (Option C) provides the exact Hive execution engine, it requires provisioning Compute Engine VMs. You pay for these VMs as long as the cluster exists, making it a costly choice for intermittent data science exploration compared to a serverless model. BigQuery (Options B and D) operates on a per-query basis, eliminating idle compute costs and aligning with cost-effective ad-hoc analytics.

Storage and Ingestion Overhead Option B requires importing the ORC files into BigQuery native storage. This incurs both active BigQuery storage costs and the time/compute expense of ingestion. Option D uses Cloud Storage as the landing zone and BigQuery external tables. Cloud Storage offers the lowest-cost object storage, and external tables query the data in place, resulting in zero ingestion overhead.

Fulfilling the 'Similar Way' Requirement The question specifies the team needs to explore data in a similar way using SQL. While Dataproc offers an identical Hive engine, BigQuery provides a standards-compliant SQL interface that fulfills the need for SQL-based exploration. BigQuery external tables natively support reading ORC file formats and Hive partitioning layouts. Therefore, Option D delivers the required SQL exploration capability over partitioned ORC files at the lowest possible cost.

Official Reference

https://cloud.google.com/bigquery/docs/external-data-sources

This link opens the official vendor documentation in a new tab.

Exam Strategy

When a question asks for the most cost-effective solution for ad-hoc or exploratory data processing, prioritize serverless options like BigQuery over managed clusters like Dataproc. Unless strict engine compatibility is explicitly mandated without workaround, serverless pay-per-query models almost always beat persistent VM clusters on cost for intermittent workloads.

Community Votes

D
67%
C
22%
B
11%

67% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 12 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 12

  1. raaad 👍 8 Answered D
    - It leverages the strengths of BigQuery for SQL-based exploration while avoiding additional costs and complexity associated with data transformation or migration. - The data remains in ORC format in Cloud Storage, and BigQuery's external tables feature allows direct querying of this data.
  2. kaisarfarel 👍 7
    I think C is the correct answer, DS want to explore the data in a "similar way as they used the on-premises HDFS cluster with SQL on the Hive query engine". Dataproc can help to create clusters quickly with the Hadoop cluster. CMIIW
  3. Smakyel79 👍 3 Answered D
    This approach leverages BigQuery's powerful analytics capabilities without the overhead of data transformation or maintaining a separate cluster, while also allowing your team to use SQL for data exploration, similar to their experience with the on-premises Hadoop/Hive environment.
  4. 0725f1f 👍 3 Answered C
    it is talking about partition as well
  5. apoio.certificacoes.closer 👍 2
    I think "Similar" is doing a lot of heavy lift on the confusion. If it was equal, I'd say C. Since it similar, it can be GoogleSQL (Bigquery).
Show 7 more comments
  1. Matt_108 👍 2 Answered D
    Option D - leverages BigQuery for SQL-based exploration on direct querying to cloud storage
  2. carbonscave 👍 1 Answered D
    Why D is the right move Cost-Effective Storage: Cloud Storage is the most cost-effective landing zone for massive amounts of raw data on Google Cloud. It acts as the perfect equivalent to an on-premises HDFS layer. Zero Ingestion Overhead: By creating External Tables in BigQuery, you do not have to pay the compute or time cost of ingesting and converting the ORC files into BigQuery's native storage format. BigQuery simply reads the files directly from the Cloud Storage buckets. Native Hive Partitioning Support: BigQuery external tables natively understand and support Hive partitioning directory structures (e.g., gs://my_bucket/table/year=2023/month=10/). When the data scientists run their SQL queries, BigQuery automatically prunes the files it reads based on those partitions, saving massive amounts of money on query costs. Cost-Effective Compute: BigQuery is serverless. The data scientists can write standard SQL (similar to what they did with Hive) and you only pay for the exact bytes scanned during their ad-hoc exploration, rather than paying for a cluster of idle virtual machines.
  3. carbonscave 👍 1
    While Dataproc runs the exact same Hadoop/Hive software they used on-premises, provisioning a Dataproc cluster means you are paying for underlying Compute Engine VMs by the second for as long as the cluster is alive. For ad-hoc data science exploration, maintaining a running Hadoop cluster is significantly more expensive than serverless BigQuery.
  4. SamuelTsch 👍 1 Answered B
    using external tables have always limitations - affecting performance, no preview of the data and no cost estimation. So, why option D is correct?
  5. nadavw 👍 1
    There is a requirement to use a 'hive query engine'', and BQ is using only the hive metastore and his own engine, so 'D' seems a better fit here.
  6. 56d02cd 👍 1 Answered C
    It says that scientists need to "explore the data with SQL on the Hive query engine". That excludes BigQuery.
  7. hanoverquay 👍 1 Answered D
    option d

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

← Back to PDE Study Guide