How to capture late data in Dataflow streaming pipelines?
You are running a streaming pipeline with Dataflow and are using hopping windows to group the data as the data arrives. You noticed that some data is arriving late but is not being marked as late data, which is resulting in inaccurate aggregations downstream. You need to find a solution that allows you to capture the late data in the appropriate window. What should you do?
Community Votes
100% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
The exam tests the concept that watermarks define the threshold for on-time data, and without explicitly allowing late data, Dataflow ignores it, a common trap for those focusing solely on windowing types.
To handle late data in Dataflow streaming pipelines, you must configure watermarks and explicitly allow late data. The community confirms that defining the expected arrival window via watermarks ensures accurate aggregations by capturing data arriving after the watermark.
Changing the windowing function to tumbling or session windows is a common mistake because it alters data grouping logic rather than addressing the specific mechanism of watermark-based lateness handling.
Community Discussion (7 comments)
- Watermarks: Watermarks in a streaming pipeline are used to specify the point in time when Dataflow expects all data up to that point to have arrived. - Allow Late Data: configure the pipeline to accept and correctly process data that arrives after the watermark, ensuring it's captured in the appropriate window.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Option A is correct because watermarks are the primary mechanism in Dataflow to track event time progress. By setting a watermark, you define the point in time when the system expects all data for a window to have arrived. Configuring the pipeline to "allow late data" ensures that any records arriving after this watermark are still routed to their correct windows and included in aggregations, rather than being dropped. This directly addresses the issue of inaccurate aggregations due to unhandled late data.Why the Other Options Are Wrong
Option B and C suggest changing windowing strategies to tumbling or session windows. While these change how windows are created (fixed vs. activity-based), they do not inherently solve the problem of data arriving late relative to the system's watermark; without allowing lateness, late data would still be problematic in these window types. Option D suggests expanding the hopping window size. While this might capture some data as "on-time" by widening the window, it increases processing latency and does not provide a robust solution for data that arrives significantly later than expected.Community Comment Notes
The community comments strongly support Option A, emphasizing the definition of watermarks as the threshold for expected data arrival. Multiple comments cite official Google Cloud documentation regarding streaming pipelines and watermarks, confirming that allowing late data is the specific configuration required to capture and process records that arrive after the watermark has passed the end of the window.Official Reference
Exam Strategy
When encountering questions about late data in streaming contexts, immediately look for options involving watermarks and "allowed lateness" configurations. Remember that changing window types solves grouping problems, not timing problems.