How to capture late data in Dataflow streaming pipelines?

PDE By islever Editorial Last updated Aug 31, 2026

Short answer

A Use watermarks to define the expected data arrival window. Allow late data as it arrives.

Question

You are running a streaming pipeline with Dataflow and are using hopping windows to group the data as the data arrives. You noticed that some data is arriving late but is not being marked as late data, which is resulting in inaccurate aggregations downstream. You need to find a solution that allows you to capture the late data in the appropriate window. What should you do?

Use watermarks to define the expected data arrival window. Allow late data as it arrives. Correct Answer
Change your windowing function to tumbling windows to avoid overlapping window periods.
Change your windowing function to session windows to define your windows based on certain activity.
Expand your hopping window so that the late data has more time to arrive within the grouping.

Key Takeaways

The exam tests the concept that watermarks define the threshold for on-time data, and without explicitly allowing late data, Dataflow ignores it, a common trap for those focusing solely on windowing types.

To handle late data in Dataflow streaming pipelines, you must configure watermarks and explicitly allow late data. The community confirms that defining the expected arrival window via watermarks ensures accurate aggregations by capturing data arriving after the watermark.

Common mistake: Changing the windowing function to tumbling or session windows is a common mistake because it alters data grouping logic rather than addressing the specific mechanism of watermark-based lateness handling.

Expert Analysis

Why the Answer Is Correct

Option A is correct because watermarks are the primary mechanism in Dataflow to track event time progress. By setting a watermark, you define the point in time when the system expects all data for a window to have arrived. Configuring the pipeline to "allow late data" ensures that any records arriving after this watermark are still routed to their correct windows and included in aggregations, rather than being dropped. This directly addresses the issue of inaccurate aggregations due to unhandled late data.

Why the Other Options Are Wrong

Option B and C suggest changing windowing strategies to tumbling or session windows. While these change how windows are created (fixed vs. activity-based), they do not inherently solve the problem of data arriving late relative to the system's watermark; without allowing lateness, late data would still be problematic in these window types. Option D suggests expanding the hopping window size. While this might capture some data as "on-time" by widening the window, it increases processing latency and does not provide a robust solution for data that arrives significantly later than expected.

Community Comment Notes

The community comments strongly support Option A, emphasizing the definition of watermarks as the threshold for expected data arrival. Multiple comments cite official Google Cloud documentation regarding streaming pipelines and watermarks, confirming that allowing late data is the specific configuration required to capture and process records that arrive after the watermark has passed the end of the window.

Official Reference

https://cloud.google.com/dataflow/docs/concepts/streaming-pipelines#watermarks

This link opens the official vendor documentation in a new tab.

Exam Strategy

When encountering questions about late data in streaming contexts, immediately look for options involving watermarks and "allowed lateness" configurations. Remember that changing window types solves grouping problems, not timing problems.

Community Votes

100%

100% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 7 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 7

raaad 👍 7 Answered A

- Watermarks: Watermarks in a streaming pipeline are used to specify the point in time when Dataflow expects all data up to that point to have arrived. - Allow Late Data: configure the pipeline to accept and correctly process data that arrives after the watermark, ensuring it's captured in the appropriate window.
Matt_108 👍 3 Answered A

Option A - https://cloud.google.com/dataflow/docs/concepts/streaming-pipelines#watermarks
Sofiia98 👍 2 Answered A

https://cloud.google.com/dataflow/docs/concepts/streaming-pipelines#watermarks
Pime13 👍 1 Answered A

https://cloud.google.com/dataflow/docs/concepts/streaming-pipelines#watermarks A watermark is a threshold that indicates when Dataflow expects all of the data in a window to have arrived. If the watermark has progressed past the end of the window and new data arrives with a timestamp within the window, the data is considered late data. For more information, see Watermarks and late data in the Apache Beam documentation. Dataflow tracks watermarks because of the following reasons: Data is not guaranteed to arrive in time order or at predictable intervals. Data events are not guaranteed to appear in pipelines in the same order that they were generated.
scaenruy 👍 1 Answered A

A. Use watermarks to define the expected data arrival window. Allow late data as it arrives.

Show 2 more comments

m_a_p_s 👍 1 Answered A

A - https://cloud.google.com/dataflow/docs/concepts/streaming-pipelines#watermarks
JyoGCP 👍 1 Answered A

Option A

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

← Back to PDE Study Guide