Why Does Dataflow Autoscaler Not Add Workers? Use Reshuffle?
You are running a Dataflow streaming pipeline, with Streaming Engine and Horizontal Autoscaling enabled. You have set the maximum number of workers to 1000. The input of your pipeline is Pub/Sub messages with notifications from Cloud Storage. One of the pipeline transforms reads CSV files and emits an element for every CSV line. The job performance is low, the pipeline is using only 10 workers, and you notice that the autoscaler is not spinning up additional workers. What should you do to improve performance?
Community Votes
100% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
The exam tests whether you understand that the autoscaler cannot add workers when fused stages limit parallelism; a Reshuffle step forces a shuffle and unlocks parallel execution.
When a Dataflow streaming pipeline uses only 10 workers despite a maximum of 1000 and performance is low, the likely cause is fusion. Community consensus recommends inserting a Reshuffle step to break fused transforms, enabling autoscaling to distribute work across more workers.
Choosing C (increase the maximum number of workers) is a common mistake because the autoscaler isn't adding workers, but the real problem is fusion, not the worker limit. Raising the max workers does nothing if the pipeline's fused stages prevent it from utilizing them.
Community Discussion (7 comments)
- Fusion optimization in Dataflow can lead to steps being "fused" together, which can sometimes hinder parallelization. - Introducing a Reshuffle step can prevent fusion and force the distribution of work across more workers. - This can be an effective way to improve parallelism and potentially trigger the autoscaler to increase the number of workers.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Dataflow's fusion optimization combines adjacent transforms into a single stage, which can centralize processing on one worker. The transform that reads CSV files and emits a line per element is likely fused with its preceding or following transforms, so the pipeline uses only one worker for that work despite autoscaling. Introducing a Reshuffle step inserts a group-by-key boundary that forces data to be shuffled, breaking fusion and enabling the stage to be split across many workers. Comments [1] and [2] correctly explain that fusion limits parallelism and Reshuffle is the fix.Why the Other Options Are Wrong
A (Vertical Autoscaling) increases the resources of existing workers but does not improve parallelism; a fused stage remains a bottleneck on a single worker. C (increase the maximum number of workers) is ineffective because the current workers are under-utilized and the autoscaler won't add to a pipeline that cannot parallelize; the maximum is already 1000. D (Dataflow Prime Right Fitting) accurately sizes worker resources but does not resolve the fusion issue; comment [6] notes that right fitting declares correct resources but won't help if fusion prevents distribution.Community Comment Notes
The community strongly favors B, with 92 votes and clear explanations. Comment [3] links to the official fusion optimization documentation, and comment [5] specifically points to the 'prevent fusion' section. Comment [6] warns that merely increasing the worker count will not help, while comments [1] and [2] describe how fusion limits parallelism and how Reshuffle breaks it. The overall consensus is that Reshuffle is the targeted solution for the under-utilized autoscaler.Official Reference
Exam Strategy
When you see a Dataflow question with 'only a few workers used despite high max workers' and 'low performance', immediately think 'fusion.' Look for an option that mentions Reshuffle or preventing fusion, and avoid picking 'increase max workers' when the autoscaler is already idle. Remember that Reshuffle forces a shuffle boundary, which is the key to enabling parallelization.