How to update Dataflow streaming pipeline without data loss?

PDE By islever Editorial Last updated Aug 31, 2026

Short answer

C Drain the old pipeline, then start the new pipeline.

Question

You created a new version of a Dataflow streaming data ingestion pipeline that reads from Pub/Sub and writes to BigQuery. The previous version of the pipeline that runs in production uses a 5-minute window for processing. You need to deploy the new version of the pipeline without losing any data, creating inconsistencies, or increasing the processing latency by more than 10 minutes. What should you do?

Update the old pipeline with the new pipeline code.
Snapshot the old pipeline, stop the old pipeline, and then start the new pipeline from the snapshot.
Drain the old pipeline, then start the new pipeline. Correct Answer
Cancel the old pipeline, then start the new pipeline.

Key Takeaways

The question tests the safe transition of streaming jobs, where the trap is confusing 'Cancel' (which loses data) with 'Drain' (which processes remaining data).

To deploy a new version of a Dataflow streaming pipeline without data loss, you should drain the old pipeline before starting the new one. This ensures all buffered data is processed and written to BigQuery before the job stops.

Common mistake: Choosing Option D (Cancel) is incorrect because it immediately terminates the job, discarding any data currently in memory or buffers; Option B (Snapshot) is incorrect because draining is the standard method for ensuring data consistency during replacements.

Expert Analysis

Why the Answer Is Correct

Draining the pipeline allows it to process all data up to the current watermark and write results to BigQuery before shutting down. This ensures no data is lost and avoids inconsistencies, meeting the requirement to finish within the allowed latency window (10 minutes).

Why the Other Options Are Wrong

Updating the old pipeline code (Option A) is not the standard method for replacing a streaming job's logic. Canceling the old pipeline (Option D) causes immediate data loss by dropping in-flight data. While snapshots (Option B) preserve state, the official upgrade guide recommends draining to safely flush data to sinks before replacement.

Community Comment Notes

Comments [6] and [7] cite the official Google Cloud documentation, confirming that draining is the preferred action to avoid data loss. Although Comment [2] raises a theoretical concern about partial windows, the overwhelming consensus and official documentation support draining as the correct procedure for this scenario.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When faced with questions about updating or stopping streaming pipelines, look for the 'Drain' option if the goal is zero data loss. Remember that 'Cancel' is equivalent to pulling the plug, whereas 'Drain' allows the job to finish its current work.

Community Votes

100%

100% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 9 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 9

raaad 👍 8 Answered C

- Graceful Data Transition: Draining the old pipeline ensures it processes all existing data in its buffers and watermarks before shutting down, preventing data loss or inconsistencies. - Minimal Latency Increase: The latency increase will be limited to the amount of time it takes to drain the old pipeline, typically within the acceptable 10-minute threshold.
AlizCert 👍 6

I don't think C is correct, as it will immediately fire the window: "Draining can result in partially filled windows. In that case, if you restart the drained pipeline, the same window might fire a second time, which can cause issues with your data. " https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline#effects Maybe "A" means launching a replacement job? https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#Launching
Ouss_123 👍 2 Answered C

- Draining the old pipeline ensures that it finishes processing all in-flight data before stopping, which prevents data loss and inconsistencies. - After draining, you can start the new pipeline, which will begin processing new data from where the old pipeline left off. - This approach maintains a smooth transition between the old and new versions, minimizing latency increases and avoiding data gaps or overlaps. ==> Other options, such as updating, snapshotting, or canceling, might not provide the same level of consistency and could lead to data loss or increased latency beyond the acceptable 10-minute window. Draining is the safest method to ensure a seamless transition.
d11379b 👍 2 Answered B

I would choose B as mentioned by Alizcert, a simple drain may cause problem Dataflow snapshots save the state of a streaming pipeline, which lets you start a new version of your Dataflow job without losing state. Snapshots are useful for backup and recovery, testing and rolling back updates to streaming pipelines, and other similar scenarios.
scaenruy 👍 2 Answered C

C. Drain the old pipeline, then start the new pipeline.

Show 4 more comments

STEVE_PEGLEG 👍 1 Answered C

There is requirement to avoid data loss. https://cloud.google.com/dataflow/docs/guides/upgrade-guide#stop-and-replace "To avoid data loss, in most cases, draining is the preferred action."
petulda 👍 1

Why not B? https://cloud.google.com/dataflow/docs/guides/upgrade-guide#stop-and-replace
Matt_108 👍 1 Answered C

Option C, draining the old pipeline solves all requests
hanoverquay 👍 1 Answered C

C option

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

← Back to PDE Study Guide