Configuring Azure Databricks Cluster Autoscaling and Termination
Short answer
BE Enable autoscaling for Cluster1 to handle workload fluctuations and configure Cluster1 to terminate after 30 minutes of inactivity to minimize idle costs.
Question
You need to configure Cluster1 to meet the following requirements:
Scale up automatically when workloads increase
Scale down automatically when workloads decrease
Minimize costs
Which two actions should you perform? Each correct answer presents part of the solution.
NOTE: Each correct selection is worth one point.
Key Takeaways
The question tests knowledge of cluster lifecycle management, specifically identifying that autoscaling handles dynamic workload changes while auto-termination prevents idle resource waste.
This question addresses how to optimize Azure Databricks cluster costs by configuring automatic scaling and inactivity termination. It establishes that enabling autoscaling and setting an auto-termination timeout are the two required actions to meet the specified requirements.
Common mistake: Learners often select 'Disable Photon acceleration' (Option A) thinking it reduces costs, but this is a distractor as it may increase runtime costs and does not address scaling or idle time directly.
Expert Analysis
Why the Answer Is Correct
To satisfy the requirement to scale up and down automatically, you must enable autoscaling on the cluster (Option B). This allows Azure Databricks to adjust the number of workers based on the workload load. To minimize costs associated with idle resources, you should configure the cluster to terminate after a period of inactivity, such as 30 minutes (Option E). Together, these two settings ensure efficient resource utilization and cost control.
Why the Other Options Are Wrong
Option A (Disable Photon acceleration) is incorrect because disabling a performance accelerator like Photon typically slows down query execution, which can lead to longer runtimes and potentially higher costs, rather than minimizing them. Option C (Apply a compute policy) is about governance and access control, not direct cost or scaling configuration for a specific cluster instance. Option D (Specify a fixed number of workers) contradicts the requirement for automatic scaling, as it prevents the cluster from adjusting its size dynamically.
Community Comment Notes
Community consensus strongly supports options B and E. One user noted that while disabling Photon might reduce DBU costs, it increases runtime, making it a poor choice for overall cost minimization compared to auto-termination. Another user highlighted that autoscaling is the only option that directly addresses the scaling requirements, confirming B and E as the correct pair.
Exam Strategy
When asked to minimize costs and handle variable workloads, always look for 'Autoscaling' to handle the variability and 'Auto-termination' to handle idleness. Avoid options that disable performance features unless explicitly stated as a cost-saving measure for short-lived jobs, as they often have hidden performance penalties.
Frequently Asked Questions
Why is disabling Photon acceleration not the best way to minimize costs?
Does specifying a fixed number of workers help with cost minimization?
Discussion 2 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 2
-
dgerok 👍 1A (Disable Photon) is a distraction: It may reduce DBU cost in some cases. But it can also increase runtime, potentially increasing total cost. Only B (Autoscaling) directly satisfies the scaling requirements, and E (Auto termination) directly reduces idle costs. So, the correct answer is B + E
-
mtc9 👍 1I would pick disable photon for cost minimizing requirement, but autotermination also meets it.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.