Migrating Proprietary On-Premises Data to BigQuery

PDE By islever Editorial Last updated

Designing data migrations.

Short answer

B Export the data to CSV files, upload the files to Cloud Storage, then load the files into BigQuery.

Question

Your company is planning to migrate a large on-premises data warehouse to BigQuery. The data is currently stored in a proprietary, vendor-specific format. You need to perform a batch migration of this data to BigQuery. What should you do?
  1. Use the bq command-line tool to load the data directly from the on-premises data warehouse.
  2. Export the data to CSV files, upload the files to Cloud Storage, then load the files into BigQuery. Correct Answer
  3. Use the BigQuery Data Transfer Service.
  4. Use Datastream to replicate the data in real time.

Key Takeaways

BigQuery cannot read proprietary vendor-specific formats directly; you must first convert them to open formats like CSV or Parquet before loading.

Learn how to migrate data from a proprietary on-premises data warehouse to BigQuery by exporting to Cloud Storage and using native load jobs, as direct transfer of proprietary formats is unsupported.

Common mistake: Choosing the BigQuery Data Transfer Service (C) because it sounds automated, but DTS only supports specific pre-configured connectors and cannot handle arbitrary proprietary on-premises formats.

Expert Analysis

The scenario involves migrating data from a proprietary on-premises warehouse to BigQuery using a batch process. The critical constraint is that the source data is in a "proprietary, vendor-specific format" which BigQuery cannot natively parse or ingest directly.

Option B outlines the standard, reliable pattern for this situation: extract the proprietary data into an open, supported format (such as CSV, Parquet, or Avro), store it in Cloud Storage, and then use BigQuery's native load jobs to import it. This approach provides maximum control, handles large volumes effectively, and works regardless of the specific proprietary engine.

Option C suggests the BigQuery Data Transfer Service (DTS). While DTS automates transfers for certain cloud sources (like Google Ads or Analytics), it does not support arbitrary on-premises proprietary formats. It requires a connector compatible with the source system, which is not specified here. Therefore, assuming DTS works for any proprietary format is incorrect.

Option A proposes loading directly via the bq tool. Since BigQuery lacks drivers for the proprietary format, direct loading will fail. Option D suggests Datastream for real-time replication, but the requirement is for a "batch migration," making real-time tools unnecessary and potentially more complex than needed.

Exam Strategy

When facing migration questions, always check the compatibility between the source format and the destination service. If the source uses a proprietary or legacy format not explicitly listed as supported by managed services (like DTS), the safe bet is usually to export to a neutral intermediate format (CSV/Parquet) in Cloud Storage and load it natively.

Frequently Asked Questions

Can I use BigQuery Data Transfer Service for on-premises data?

No. The BigQuery Data Transfer Service primarily supports cloud-based sources like Google Analytics, AdSense, and some SaaS platforms. It does not support arbitrary on-premises proprietary databases.

What formats does BigQuery support for batch loads?

BigQuery supports CSV, JSON, Avro, Parquet, ORC, and Apache Beam. You must convert proprietary formats to one of these before loading.

More PDE FAQ →

Community Votes

C
67%
B
33%

67% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 3 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 3

  1. 06bbd0e 👍 2 Answered C
    BigQuery Data Transfer Service (DTS) is the correct answer for batch migration.
  2. Andy_Khan1 👍 1 Answered B
    B (Export to CSV $\rightarrow$ Cloud Storage $\rightarrow$ BigQuery): Correct. Since BigQuery cannot read proprietary vendor formats, the standard migration strategy is to extract the data into a open/standard format (like CSV, Parquet, or Avro), upload those files to Cloud Storage (GCS), and load them into BigQuery using native batch load jobs.
  3. dataschool 👍 1 Answered C
    igQuery Data Transfer Service (DTS) específicamente para automatizar la migración masiva e interactiva desde almacenes de datos (Data Warehouses) heredados y propietarios (como Teradata, Netezza, Redshift, Snowflake o Oracle) hacia BigQuery.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

← Back to PDE Study Guide