How to update Dataflow streaming pipeline without data loss?
You created a new version of a Dataflow streaming data ingestion pipeline that reads from Pub/Sub and writes to BigQuery. The previous version of the pipeline that runs in production uses a 5-minute window for processing. You need to deploy the new version of the pipeline without losing any data, creating inconsistencies, or increasing the processing latency by more than 10 minutes. What should you do?
Community Votes
100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
The question tests the safe transition of streaming jobs, where the trap is confusing 'Cancel' (which loses data) with 'Drain' (which processes remaining data).
To deploy a new version of a Dataflow streaming pipeline without data loss, you should drain the old pipeline before starting the new one. This ensures all buffered data is processed and written to BigQuery before the job stops.
Choosing Option D (Cancel) is incorrect because it immediately terminates the job, discarding any data currently in memory or buffers; Option B (Snapshot) is incorrect because draining is the standard method for ensuring data consistency during replacements.
Community Discussion (9 comments)
- Graceful Data Transition: Draining the old pipeline ensures it processes all existing data in its buffers and watermarks before shutting down, preventing data loss or inconsistencies. - Minimal Latency Increase: The latency increase will be limited to the amount of time it takes to drain the old pipeline, typically within the acceptable 10-minute threshold.
- Draining the old pipeline ensures that it finishes processing all in-flight data before stopping, which prevents data loss and inconsistencies. - After draining, you can start the new pipeline, which will begin processing new data from where the old pipeline left off. - This approach maintains a smooth transition between the old and new versions, minimizing latency increases and avoiding data gaps or overlaps. ==> Other options, such as updating, snapshotting, or canceling, might not provide the same level of consistency and could lead to data loss or increased latency beyond the acceptable 10-minute window. Draining is the safest method to ensure a seamless transition.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Draining the pipeline allows it to process all data up to the current watermark and write results to BigQuery before shutting down. This ensures no data is lost and avoids inconsistencies, meeting the requirement to finish within the allowed latency window (10 minutes).Why the Other Options Are Wrong
Updating the old pipeline code (Option A) is not the standard method for replacing a streaming job's logic. Canceling the old pipeline (Option D) causes immediate data loss by dropping in-flight data. While snapshots (Option B) preserve state, the official upgrade guide recommends draining to safely flush data to sinks before replacement.Community Comment Notes
Comments [6] and [7] cite the official Google Cloud documentation, confirming that draining is the preferred action to avoid data loss. Although Comment [2] raises a theoretical concern about partial windows, the overwhelming consensus and official documentation support draining as the correct procedure for this scenario.Official Reference
Exam Strategy
When faced with questions about updating or stopping streaming pipelines, look for the 'Drain' option if the goal is zero data loss. Remember that 'Cancel' is equivalent to pulling the plug, whereas 'Drain' allows the job to finish its current work.