Lakeflow SDP Compute Configuration for DP-750
You have an Azure Databricks workspace. You are creating a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically. You need to configure compute for the pipeline. The solution must minimize operational costs and administrative effort. What should you use?
Community Insight
The exam tests knowledge of Lakeflow SDP execution environments, with the common trap being confusion between job clusters and serverless options for managed pipelines.
This question addresses configuring compute for Azure Databricks Lakeflow Spark Declarative Pipelines (SDP) to minimize operational costs and administrative effort. The correct solution leverages serverless compute capabilities inherent to the platform.
Candidates often select 'a job cluster that uses autoscaling' because they associate automated scaling with cost efficiency, overlooking that serverless compute removes administrative overhead entirely.
Community Discussion (2 comments)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Serverless compute is the optimal choice for Lakeflow Spark Declarative Pipelines (SDP) when the goal is to minimize both operational costs and administrative effort. Serverless infrastructure automatically handles cluster provisioning, configuration, and scaling without any user intervention, aligning perfectly with the requirement to reduce administrative burden. For continuous workloads like streaming pipelines, serverless offers a pay-per-use model that can be more cost-effective than maintaining dedicated clusters.Why the Other Options Are Wrong
A single-node or all-purpose cluster requires manual management and is intended for interactive development, not production pipelines. An existing SQL warehouse is designed for BI and querying, not for executing complex ETL/ELT pipeline logic defined in SDP. While a job cluster with autoscaling reduces idle costs compared to all-purpose clusters, it still requires the user to manage the job definition and monitor the runtime, failing the 'minimize administrative effort' criterion compared to serverless.Community Comment Notes
Community feedback highlights the distinction between serverless and job clusters. One commenter noted that serverless eliminates administrative effort effectively, pointing out that Databricks manages pipeline compute automatically. Another perspective suggested using a job cluster for constant running tasks to save costs, but this overlooks the specific benefits of serverless compute for SDP which abstracts away the cluster management layer entirely.Exam Strategy
When questions mention 'minimize administrative effort' alongside 'scale automatically' for managed services like Lakeflow or Delta Live Tables, prioritize serverless options over managed clusters. Always read the workload type carefully; if it's a declarative pipeline, serverless is often the modern best practice for reducing ops overhead.
Frequently Asked Questions
Why not use a job cluster for Lakeflow SDP?
Job clusters require manual management and setup. Serverless compute automates provisioning and scaling, significantly reducing administrative effort.
Is serverless compute suitable for continuous streaming pipelines?
Yes, serverless compute supports continuous processing modes required by streaming pipelines in Lakeflow SDP while optimizing costs via pay-per-use.