How to programmatically stream or batch load masked data to BigQuery?

PDE By islever Editorial Last updated Aug 31, 2026

Short answer

C Create your pipeline with Dataflow through the Apache Beam SDK for Python, customizing separate options within your code for streaming, batch processing, and Cloud DLP. Select BigQuery as your data sink.

Question

You are planning to load some of your existing on-premises data into BigQuery on Google Cloud. You want to either stream or batch-load data, depending on your use case. Additionally, you want to mask some sensitive data before loading into BigQuery. You need to do this in a programmatic way while keeping costs to a minimum. What should you do?

Use Cloud Data Fusion to design your pipeline, use the Cloud DLP plug-in to de-identify data within your pipeline, and then move the data into BigQuery.
Use the BigQuery Data Transfer Service to schedule your migration. After the data is populated in BigQuery, use the connection to the Cloud Data Loss Prevention (Cloud DLP) API to de-identify the necessary data.
Create your pipeline with Dataflow through the Apache Beam SDK for Python, customizing separate options within your code for streaming, batch processing, and Cloud DLP. Select BigQuery as your data sink. Correct Answer
Set up Datastream to replicate your on-premise data on BigQuery.

Key Takeaways

This question tests the ability to select a programmatic ETL tool that supports both streaming and batch processing with pre-load data masking, where the common trap is choosing code-free tools like Data Fusion or batch-only services.

Dataflow with Apache Beam is the optimal solution for programmatically loading on-premises data into BigQuery while masking sensitive information. The community consensus highlights its flexibility for both streaming and batch processing and cost-effectiveness compared to code-free alternatives.

Common mistake: Selecting Cloud Data Fusion (Option A) is a frequent error because it is primarily a low-code/no-code platform, failing to meet the specific requirement for a programmatic solution.

Expert Analysis

Why the Answer Is Correct

Option C is correct because Dataflow, utilizing the Apache Beam SDK, is designed for programmatic, serverless data processing pipelines. It natively supports both streaming and batch execution modes within the same codebase, allowing for flexible data loading strategies. Furthermore, it allows for the integration of Cloud DLP API directly within the pipeline logic to de-identify sensitive data before it is written to BigQuery, satisfying the security and cost constraints.

Why the Other Options Are Wrong

Option A is incorrect because Cloud Data Fusion is primarily a code-free or low-code visual ETL tool, which contradicts the requirement for a "programmatic way." Option B is incorrect because BigQuery Data Transfer Service is primarily for batch data movement and does not support complex, in-flight transformations like DLP masking before the data reaches BigQuery; it also lacks streaming capabilities. Option D is incorrect because Datastream focuses on change data capture (CDC) for replication and does not offer the programmatic transformation capabilities needed to mask data before loading.

Community Comment Notes

Community members strongly support Option C, emphasizing "Programmatic Flexibility" and the ability to customize pipeline options for streaming and batch processing. One user explicitly noted that Option A is incorrect because Data Fusion's main purpose is "code-free," which is a critical distinction to make when answering exam questions. The high vote count for C confirms that understanding the programmatic nature of Apache Beam is essential for this scenario.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

Look for specific keywords like "programmatic" or "code-free" to eliminate options like Data Fusion or Data Transfer Service immediately. Remember that data transformation (like masking) must generally occur in the pipeline (Dataflow) before storage if the requirement specifies "before loading."

Community Votes

100%

100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 4 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 4

raaad 👍 11 Answered C

- Programmatic Flexibility: Apache Beam provides extensive control over pipeline design, allowing for customization of data transformations, including integration with Cloud DLP for sensitive data masking. - Streaming and Batch Support: Beam seamlessly supports both streaming and batch data processing modes, enabling flexibility in data loading patterns. - Cost-Effective Processing: Dataflow offers a serverless model, scaling resources as needed, and only charging for resources used, helping optimize costs. - Integration with Cloud DLP: Beam integrates well with Cloud DLP for sensitive data masking, ensuring data privacy before loading into BigQuery.
tibuenoc 👍 2 Answered C

C is correct. Using Dataflow as Python as programming and BQ as sink. A is incorrect - DataFusion is Code-free as the main propose
scaenruy 👍 1 Answered A

A. Use Cloud Data Fusion to design your pipeline, use the Cloud DLP plug-in to de-identify data within your pipeline, and then move the data into BigQuery.
JyoGCP 👍 1 Answered C

Option C

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

← Back to PDE Study Guide