How to Identify Bottlenecks in a Dataflow Pipeline After Fusion?
You maintain ETL pipelines. You notice that a streaming pipeline running on Dataflow is taking a long time to process incoming data, which causes output delays. You also noticed that the pipeline graph was automatically optimized by Dataflow and merged into one step. You want to identify where the potential bottleneck is occurring. What should you do?
Community Votes
100% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
This question tests your understanding of Dataflow's fusion optimization and how Reshuffle can be used to force a shuffle boundary, enabling per-stage monitoring; the common trap is confusing this with output sinks or logging.
When a Dataflow streaming pipeline is slow and fusion merges steps, inserting Reshuffle operations is the recommended technique to break fusion and monitor individual step metrics. Community consensus strongly supports Reshuffle as the key to identifying bottlenecks.
The most common wrong answer is B (insert output sinks) because it seems to measure throughput, but it alters the pipeline and doesn't reveal internal step performance as effectively as Reshuffle, which is the standard method.
Community Discussion (9 comments)
- The Reshuffle operation is used in Dataflow pipelines to break fusion and redistribute elements, which can sometimes help improve parallelization and identify bottlenecks. - By inserting Reshuffle after each processing step and observing the pipeline's performance in the Dataflow console, you can potentially identify stages that are disproportionately slow or stalled. - This can help in pinpointing the step where the bottleneck might be occurring.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Option A is correct because Dataflow's fusion optimization merges multiple steps into a single stage, hiding individual step metrics. Inserting a Reshuffle after each processing step forces a shuffle boundary, breaking the fusion and allowing the Dataflow console to show execution details for each separate step. This lets you identify which step is disproportionately slow or stalled, directly addressing the need to locate the bottleneck.Why the Other Options Are Wrong
Option B (insert output sinks) is not ideal because adding sinks after every step changes the pipeline semantics, increases cost, and can interfere with streaming performance; it is an indirect and heavy-handed way to measure throughput. Option C (logging debug info in each ParDo) provides logs but not structured performance metrics, making bottleneck identification difficult and slow. Option D (verifying service account permissions) is unrelated to the pipeline slowdown and fusion issue.Community Comment Notes
Comment 2 cites official Dataflow documentation recommending Reshuffle to prevent fusion when the service might guess suboptimal fusion. Comment 4 also references the same documentation, noting that Reshuffle is the option that helps. Comment 6 argues for B, but most comments (including the 90 votes for A) align with the correct answer, recognizing that Reshuffle is the standard diagnostic technique in this scenario.Official Reference
Exam Strategy
On the exam, when you see a Dataflow pipeline where fusion has merged steps and you need to find a bottleneck, look for 'Reshuffle' as the answer—it breaks fusion and enables per-step monitoring. Avoid options that change the pipeline (like adding sinks) or focus on logging/permissions, as they are indirect or irrelevant to the fusion issue.