How to stream 1.5 GB/s to BigQuery with exactly-once semantics?
You are using a Dataflow streaming job to read messages from a message bus that does not support exactly-once delivery. Your job then applies some transformations, and loads the result into BigQuery. You want to ensure that your data is being streamed into BigQuery with exactly-once delivery semantics. You expect your ingestion throughput into BigQuery to be about 1.5 GB per second. What should you do?
Community Votes
57% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
The question tests both the exactly-once delivery capability of the Storage Write API and the throughput limitations of BigQuery locations; the trap is assuming regional tables are faster when they actually have a lower streaming quota.
To achieve exactly-once delivery into BigQuery from a Dataflow streaming job at 1.5 GB/s, use the BigQuery Storage Write API with a multiregional table. The community consensus (B) is that only multiregional tables offer the required 3 GB/s throughput, while regional tables are limited to 300 MB/s.
Option A (Storage Write API + regional table) is the most common wrong answer; it overlooks that regional tables only support 300 MB/s write throughput, which cannot handle the required 1.5 GB/s.
Community Discussion (16 comments)
- BigQuery Storage Write API: This API is designed for high-throughput, low-latency writing of data into BigQuery. It also provides tools to prevent data duplication, which is essential for exactly-once delivery semantics. - Regional Table: Choosing a regional location for the BigQuery table could potentially provide better performance and lower latency, as it would be closer to the Dataflow job if they are in the same region.
- BigQuery Storage Write API: This API is designed for high-throughput, low-latency writing of data into BigQuery. It also provides tools to prevent data duplication, which is essential for exactly-once delivery semantics. - The multiregional table ensures that your data is highly available and can be streamed into BigQuery across multiple regions. It is better suited for high-throughput and low-latency workloads, as it provides distributed write capabilities that can handle large data volumes, such as the 1.5 GB per second you expect to stream.
- Choosing a regional target BigQuery table ensures that data is stored redundantly in a single region, providing high availability and durability.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
The BigQuery Storage Write API is the only option that provides exactly-once delivery semantics, as the older streaming API (tabledata.insertAll) only offers at-least-once. Since the Dataflow job reads from a message bus that lacks exactly-once support, the Storage Write API’s de-duplication capabilities are essential. The throughput requirement of 1.5 GB/s dictates the table location: BigQuery quotas list Storage Write API limits as 3 GB/s for multiregional locations and only 300 MB/s for regional locations. Thus, B (Storage Write API + multiregional table) satisfies both exactly-once and throughput needs.
Why the Other Options Are Wrong
Option A uses the correct API but the wrong location — a regional table’s 300 MB/s ceiling cannot handle 1.5 GB/s. Options C and D use the BigQuery Streaming API, which does not support exactly-once delivery, so they fail the core requirement. Even if D’s multiregional location gives higher throughput, the lack of exactly-once semantics makes it invalid. Therefore, only B meets both constraints.
Community Comment Notes
The majority of comments (56 votes for B) cite the official quota documentation, with comment [3] and [5] pointing directly to the 3 GB/s multiregional vs 300 MB/s regional limits. Comment [2] incorrectly argues that regional tables “could potentially provide better performance,” but that refers to latency, not throughput quotas. Comment [1] summarises the exact reason: “Storage Write API has 3 GB per second throughput in multi-regions; 300 MB per second in regions.” The official docs linked in comments confirm that Storage Write API is recommended for exactly-once and has lower pricing.
Official Reference
Exam Strategy
When a question specifies a numeric throughput requirement, always verify it against BigQuery’s published quotas before choosing a location. Remember that regional tables are not inherently faster for writes; multiregional tables have much higher write throughput limits, so don’t assume that 'regional' means better performance.