Optimize Azure Databricks Cluster Costs for Predictable Workloads
You have an Azure Databricks workspace that contains a cluster named Cluster1. Performance monitoring shows that Cluster1 is consistently overprovisioned for its batch workload: CPU utilization remains below 20 percent, including peak processing periods. The workload is highly predictable and does not spike. The current node type already meets the workload requirements. You need to reduce compute costs without increasing job duration. What should you do?
Community Insight
The scenario tests the understanding of cluster scaling behaviors; the trap is choosing performance enhancements like Photon when the problem is resource over-provisioning.
This question addresses cost optimization in Azure Databricks by identifying the correct configuration change to reduce compute expenses for predictable, low-utilization batch workloads without impacting job duration.
Learners often select 'Enable Photon acceleration' because it improves speed, but it does not inherently reduce the base cost of reserved or provisioned nodes and may increase costs if not optimized correctly.
Community Discussion (1 comments)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
The core issue is that Cluster1 is overprovisioned with CPU utilization consistently below 20%, even during peaks. Since the workload is highly predictable and does not spike, there is no need for dynamic resource adjustment. Disabling autoscaling prevents the cluster from adding unnecessary nodes, and reducing the number of worker nodes directly lowers the compute hours billed, achieving the goal of cost reduction without increasing job duration.Why the Other Options Are Wrong
Option A (Enable Photon acceleration) optimizes query performance but does not address the fundamental issue of having too many idle nodes; it might even increase costs if using on-demand instances without careful management. Option B (Larger node type) would likely increase costs or provide no benefit since the current node type already meets requirements. Option C (Decrease autotermination timeout) affects idle clusters, but the problem describes a cluster actively running a batch workload that is underutilized, not an idle one waiting to terminate.Community Comment Notes
Community consensus strongly supports Option D, noting that the consistent low CPU usage indicates the cluster has more capacity than needed. As user dgerok noted, "The cluster is consistently overprovisioned," making the reduction of worker nodes the logical step to cut costs.Exam Strategy
When asked to reduce costs for predictable workloads, look for options that right-size the infrastructure. Avoid solutions focused solely on speed (like Photon) unless the question explicitly mentions performance bottlenecks as the primary constraint alongside cost.
Frequently Asked Questions
Why not just use Spot Instances?
Spot Instances can save up to 90% but introduce interruption risks. For critical batch workloads where reliability is key, right-sizing reserved/on-demand nodes via fewer workers is safer.
Does disabling autoscaling affect performance?
No, because the workload is predictable and does not spike. Autoscaling is designed for variable loads; removing it for stable loads simply stops paying for unused reserve capacity.