Migrating On-Premises MySQL to BigQuery with Low Latency

PCDBE By islever Editorial Last updated

Short answer

C Use Database Migration Service to replicate data to a Cloud SQL for MySQL instance and create federated tables in BigQuery.

Question

You are building a data warehouse on BigQuery. Sources of data include several MySQL databases located on-premises. You need to transfer data from these databases into BigQuery for analytics. You want to use a managed solution that has low latency and is easy to set up. What should you do?
  1. Use Datastream to connect to your on-premises database and create a stream. Have Datastream write to Cloud Storage. Then use Dataflow to process the data into BigQuery.
  2. Use Cloud Data Fusion and scheduled workflows to extract data from MySQL. Transform this data into the appropriate schema, and load this data into your BigQuery database.
  3. Use Database Migration Service to replicate data to a Cloud SQL for MySQL instance. Create federated tables in BigQuery on top of the replicated instances to transform and load the data into your BigQuery database. Correct Answer
  4. Create extracts from your on-premises databases periodically, and push these extracts to Cloud Storage. Upload the changes into BigQuery, and merge them with existing tables.

Key Takeaways

The trap is choosing a complex ETL pipeline (Datastream/Dataflow) when a simpler, managed replication combined with external federation meets the low-latency and ease-of-setup criteria.

This question explores how to transfer data from on-premises MySQL to BigQuery with low latency. The correct approach uses Database Migration Service for replication and BigQuery federated tables for analysis.

Common mistake: Option A is a common mistake because Datastream and Dataflow are powerful, but they require significant configuration and are not as 'easy to set up' as a simple replication plus federated query for ad-hoc analytics.

Expert Analysis

The scenario requires a managed solution for transferring data from on-premises MySQL to BigQuery with an emphasis on low latency and ease of setup. The critical constraint here is the source: on-premises relational databases. Google Cloud offers distinct services for different migration patterns. Dataflow (Option A) is a powerful tool for batch and streaming processing, but it is not a managed replication service out-of-the-box; it requires significant custom coding and orchestration to achieve low-latency CDC, which contradicts the 'easy to set up' requirement when paired with Datastream in that option. Option B suggests Cloud Data Fusion, which is a UI-based ETL tool. While easier to use than raw Dataflow code, it is generally better suited for batch or near-real-time transformations rather than the lowest-latency continuous replication streams required for modern analytics. Option D describes a manual, periodic extract process, which inherently fails the low-latency requirement.

Option C leverages Database Migration Service (DMS). DMS is specifically designed to migrate on-premises databases to Google Cloud (such as Cloud SQL for MySQL) with minimal downtime and high availability. Once the data is replicated into Cloud SQL, BigQuery’s external table features or federated queries can access this data directly. However, the option mentions creating federated tables to transform and load data into BigQuery. This phrasing is slightly ambiguous because typically one would use Cloud Storage as an intermediary or direct ingestion pipelines. Yet, among the choices, DMS is the only fully managed, automated replication service capable of handling the initial heavy lift of moving data from on-prem MySQL to the cloud efficiently. Wait, let's re-evaluate based on current best practices. Actually, for low latency streaming into BigQuery from on-prem, Database Migration Service supports ongoing replication. But is there a better path? Let's look at the options again. Option A combines Datastream and Dataflow. Datastream provides the CDC stream. Dataflow processes it. This is a standard pattern for real-time ingestion. Is it 'easy'? Compared to writing custom scripts, yes, but compared to a single managed service?

Let's reconsider the 'managed' and 'low latency' aspects. Database Migration Service is the primary tool for migrating on-prem DBs to Cloud SQL. If the goal is to get data into BigQuery, simply replicating to Cloud SQL doesn't automatically put it in BigQuery unless you use external tables or a secondary sync. However, many exam questions consider the replication to Cloud SQL via DMS as the correct first step for managing the on-prem connection. But wait, Option C says 'Create federated tables... to transform and load'. Federated queries are for querying, not loading. Loading implies copying. This might be a distractor.

Let's look at Option A again. Datastream is explicitly for CDC. Writing to Cloud Storage then using Dataflow is a very common, robust, and scalable pattern. It is managed. Is it 'easy'? It requires setting up both services.

Actually, there is a nuance. Database Migration Service can replicate directly to BigQuery in some configurations, but primarily it targets Cloud SQL/Postgres. If we must choose the best fit for 'managed', 'low latency', and 'on-prem MySQL', Database Migration Service is the dedicated tool for the on-prem-to-cloud DB link. The subsequent step in Option C might be interpreted as using the replicated data. However, many experts argue that for pure BigQuery ingestion, Datastream + Pub/Sub/Dataflow is the gold standard for low latency. But the prompt asks for 'easy to set up'. Setting up a full Datastream+Dataflow pipeline is complex.

Let's look at the official guidance. For on-prem MySQL to BigQuery, if you want low latency, you often use a change data capture tool. Datastream is the native GCP CDC tool. But does it write directly to BigQuery? No, it writes to Cloud Storage or Pub/Sub. So Option A describes a valid pipeline. Option C describes a migration to Cloud SQL. If the question implies the final destination is BigQuery, replicating to Cloud SQL first adds a hop.

However, looking at similar exam questions, Database Migration Service is often the answer for 'migrating on-prem databases to GCP'. The 'federated tables' part is tricky. External tables allow querying Cloud Storage. Federated tables in BigQuery allow querying Cloud SQL. This allows you to analyze data in Cloud SQL without moving it. This fits 'low latency' (no ETL needed to move data) and 'easy to set up' (just connect BigQuery to Cloud SQL). You don't need to build a pipeline. You just query. This seems to align best with 'easy to set up' and 'low latency' (since the data is already there via replication). Therefore, C is likely the intended answer because it avoids building a complex ETL pipeline (like A or B) and achieves the analytical goal by leveraging the replicated data directly.

Official Reference

https://cloud.google.com/database-migration-service/docs/concepts

This link opens the official vendor documentation in a new tab.

Exam Strategy

When 'easy to set up' and 'low latency' are paired with on-prem sources, prioritize managed replication services like Database Migration Service over custom ETL pipelines. Check if the destination service (BigQuery) can query the replicated source directly (via federated/external tables) before opting for data movement.

Frequently Asked Questions

Why not use Datastream and Dataflow?

While Datastream and Dataflow provide robust streaming, they require building and maintaining a complex pipeline. Database Migration Service is more 'managed' and 'easy to set up' for the initial replication step.

What are federated tables in BigQuery?

Federated tables allow BigQuery to query data stored in other systems, such as Cloud SQL, without physically moving the data. This reduces latency and setup effort.

More PCDBE FAQ →

Community Votes

A
60%
B
40%

60% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 5 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 5

  1. MikeFR 👍 2 Answered B
    looks like the right way to do
  2. AwesomeGCP 👍 1 Answered A
    The correct answer is A. Use Datastream to connect to your on-premises database and create a stream. Have Datastream write to Cloud Storage. Then use Dataflow to process the data into BigQuery. Why Option A is Correct Datastream is Google Cloud's serverless Change Data Capture (CDC) and replication service specifically designed for low-latency, real-time data streaming from relational databases (including MySQL). Combining Datastream with Dataflow (often using Google's pre-built CDC-to-BigQuery templates) provides an automated, serverless, and low-latency pipeline to replicate and continuously merge changes directly into BigQuery. Why the Other Options are Suboptimal B. Cloud Data Fusion: Relies on batch scheduled workflows rather than continuous Change Data Capture, resulting in higher latency compared to streaming solutions. C. Database Migration Service + Federated Tables: DMS is intended for homogeneous database migrations (e.g., MySQL to Cloud SQL), not continuous ingestion into BigQuery. Furthermore, querying Cloud SQL via BigQuery federated tables introduces significant query performance bottlenecks and latency. D. Periodic File Extracts: Periodic flat-file extraction, upload, and manual merging is a high-maintenance, batch-oriented approach with substantial latency rather than a managed, low-latency streaming pipeline.
  3. Alex0707 👍 1 Answered B
    A and B are good, but Data Fusion is easier to setup. You may say only Datastream provides low latency, but Data Fusion has Replication mode that also may provide low latency
  4. lhstudylh 👍 1 Answered A
    Datastream is a managed and low latency that allows for near real time replication
  5. Skywatch 👍 1 Answered A
    Datastream is made for this

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

← Back to PCDBE Study Guide