How to programmatically stream or batch load masked data to BigQuery?
You are planning to load some of your existing on-premises data into BigQuery on Google Cloud. You want to either stream or batch-load data, depending on your use case. Additionally, you want to mask some sensitive data before loading into BigQuery. You need to do this in a programmatic way while keeping costs to a minimum. What should you do?
Community Votes
100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
This question tests the ability to select a programmatic ETL tool that supports both streaming and batch processing with pre-load data masking, where the common trap is choosing code-free tools like Data Fusion or batch-only services.
Dataflow with Apache Beam is the optimal solution for programmatically loading on-premises data into BigQuery while masking sensitive information. The community consensus highlights its flexibility for both streaming and batch processing and cost-effectiveness compared to code-free alternatives.
Selecting Cloud Data Fusion (Option A) is a frequent error because it is primarily a low-code/no-code platform, failing to meet the specific requirement for a programmatic solution.
Community Discussion (4 comments)
- Programmatic Flexibility: Apache Beam provides extensive control over pipeline design, allowing for customization of data transformations, including integration with Cloud DLP for sensitive data masking. - Streaming and Batch Support: Beam seamlessly supports both streaming and batch data processing modes, enabling flexibility in data loading patterns. - Cost-Effective Processing: Dataflow offers a serverless model, scaling resources as needed, and only charging for resources used, helping optimize costs. - Integration with Cloud DLP: Beam integrates well with Cloud DLP for sensitive data masking, ensuring data privacy before loading into BigQuery.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Option C is correct because Dataflow, utilizing the Apache Beam SDK, is designed for programmatic, serverless data processing pipelines. It natively supports both streaming and batch execution modes within the same codebase, allowing for flexible data loading strategies. Furthermore, it allows for the integration of Cloud DLP API directly within the pipeline logic to de-identify sensitive data before it is written to BigQuery, satisfying the security and cost constraints.Why the Other Options Are Wrong
Option A is incorrect because Cloud Data Fusion is primarily a code-free or low-code visual ETL tool, which contradicts the requirement for a "programmatic way." Option B is incorrect because BigQuery Data Transfer Service is primarily for batch data movement and does not support complex, in-flight transformations like DLP masking before the data reaches BigQuery; it also lacks streaming capabilities. Option D is incorrect because Datastream focuses on change data capture (CDC) for replication and does not offer the programmatic transformation capabilities needed to mask data before loading.Community Comment Notes
Community members strongly support Option C, emphasizing "Programmatic Flexibility" and the ability to customize pipeline options for streaming and batch processing. One user explicitly noted that Option A is incorrect because Data Fusion's main purpose is "code-free," which is a critical distinction to make when answering exam questions. The high vote count for C confirms that understanding the programmatic nature of Apache Beam is essential for this scenario.Official Reference
Exam Strategy
Look for specific keywords like "programmatic" or "code-free" to eliminate options like Data Fusion or Data Transfer Service immediately. Remember that data transformation (like masking) must generally occur in the pipeline (Dataflow) before storage if the requirement specifies "before loading."