How to Explore On-Premises ORC Data Cost-Effectively in Google Cloud?
You created an analytics environment on Google Cloud so that your data scientist team can explore data without impacting the on-premises Apache Hadoop solution. The data in the on-premises Hadoop Distributed File System (HDFS) cluster is in Optimized Row Columnar (ORC) formatted files with multiple columns of Hive partitioning. The data scientist team needs to be able to explore the data in a similar way as they used the on-premises HDFS cluster with SQL on the Hive query engine. You need to choose the most cost-effective storage and processing solution. What should you do?
Community Votes
84% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
This tests your understanding of BigQuery external tables with Hive partitioning versus running a Dataproc cluster, with the trap being that Dataproc feels 'similar to Hive' but is not cost-effective for ad-hoc exploration.
The question asks for the most cost-effective storage and processing solution for exploring on-premises Hive-partitioned ORC data on Google Cloud. Community consensus favors copying ORC files to Cloud Storage and creating external BigQuery tables, which allows direct SQL querying without data transformation or cluster management.
Choosing C (copy files to Cloud Storage and deploy a Dataproc cluster) is common because it seems to mirror the Hive experience, but it requires maintaining a running cluster, which is not the most cost-effective solution for exploratory analytics.
Community Discussion (9 comments)
- It leverages the strengths of BigQuery for SQL-based exploration while avoiding additional costs and complexity associated with data transformation or migration. - The data remains in ORC format in Cloud Storage, and BigQuery's external tables feature allows direct querying of this data.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Option D is correct because it leverages BigQuery's serverless architecture to query ORC files directly from Cloud Storage without importing or transforming data. BigQuery external tables support Hive-partitioned ORC files, enabling the data science team to use SQL similarly to Hive. This approach minimizes storage costs (no duplicate data) and eliminates compute costs when the cluster is not running, making it the most cost-effective solution.Why the Other Options Are Wrong
A (Bigtable) is wrong because Bigtable is designed for NoSQL workloads, not SQL-based exploration of ORC files. B (import to BigQuery) is viable but not cost-effective because it duplicates storage and incurs data loading costs, whereas external tables avoid this. C (Dataproc) is functionally similar to Hive but requires a persistent cluster (or paying for cluster startup each time), making it less cost-effective for exploratory workloads compared to serverless BigQuery.Community Comment Notes
Comment [1] correctly notes that BigQuery external tables 'leverage the strengths of BigQuery for SQL-based exploration while avoiding additional costs and complexity.' Comment [2] argues for C due to similarity with Hive, but this misses the cost-effectiveness requirement; comment [7] raises concerns about external table limitations, but BigQuery's support for Hive-partitioned ORC files mitigates the main issue. The 80-vote majority for D reflects the well-established best practice of using BigQuery external tables for ad-hoc SQL analytics on raw data files in Cloud Storage.Official Reference
Exam Strategy
Look for keywords like 'most cost-effective' and 'explore data' in the question. When raw data files already exist, prefer BigQuery external tables over Dataproc clusters or importing into BigQuery, as serverless query engines eliminate idle compute costs and data duplication.