Lakeflow SDP Compute Configuration for DP-750
Short answer
A Use serverless compute for Lakeflow Spark Declarative Pipelines to automatically scale and minimize administrative overhead.
Question
You are creating a Lakeflow Spark Declarative Pipelines (SDP) pipeline that scales automatically.
You need to configure compute for the pipeline. The solution must minimize operational costs and administrative effort.
What should you use?
Key Takeaways
The exam tests knowledge of Lakeflow SDP execution environments, with the common trap being confusion between job clusters and serverless options for managed pipelines.
This question addresses configuring compute for Azure Databricks Lakeflow Spark Declarative Pipelines (SDP) to minimize operational costs and administrative effort. The correct solution leverages serverless compute capabilities inherent to the platform.
Common mistake: Candidates often select 'a job cluster that uses autoscaling' because they associate automated scaling with cost efficiency, overlooking that serverless compute removes administrative overhead entirely.
Expert Analysis
Why the Answer Is Correct
Serverless compute is the optimal choice for Lakeflow Spark Declarative Pipelines (SDP) when the goal is to minimize both operational costs and administrative effort. Serverless infrastructure automatically handles cluster provisioning, configuration, and scaling without any user intervention, aligning perfectly with the requirement to reduce administrative burden. For continuous workloads like streaming pipelines, serverless offers a pay-per-use model that can be more cost-effective than maintaining dedicated clusters.
Why the Other Options Are Wrong
A single-node or all-purpose cluster requires manual management and is intended for interactive development, not production pipelines. An existing SQL warehouse is designed for BI and querying, not for executing complex ETL/ELT pipeline logic defined in SDP. While a job cluster with autoscaling reduces idle costs compared to all-purpose clusters, it still requires the user to manage the job definition and monitor the runtime, failing the 'minimize administrative effort' criterion compared to serverless.
Community Comment Notes
Community feedback highlights the distinction between serverless and job clusters. One commenter noted that serverless eliminates administrative effort effectively, pointing out that Databricks manages pipeline compute automatically. Another perspective suggested using a job cluster for constant running tasks to save costs, but this overlooks the specific benefits of serverless compute for SDP which abstracts away the cluster management layer entirely.
Exam Strategy
When questions mention 'minimize administrative effort' alongside 'scale automatically' for managed services like Lakeflow or Delta Live Tables, prioritize serverless options over managed clusters. Always read the workload type carefully; if it's a declarative pipeline, serverless is often the modern best practice for reducing ops overhead.
Frequently Asked Questions
Why not use a job cluster for Lakeflow SDP?
Is serverless compute suitable for continuous streaming pipelines?
Discussion 3 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 3
-
dgerok 👍 3 Answered ALakeflow Spark Declarative Pipelines (formerly DLT) can run on serverless compute. If serverless is not used, Databricks provisions and manages pipeline compute for you (effectively job-like compute), rather than using an all-purpose cluster. The key phrase in the question is: "scales automatically" + "minimize operational costs and administrative effort" That strongly points to: A. Serverless compute because: automatic scaling no cluster management Databricks manages the infrastructure
-
mtc9 👍 1 Answered EServerless indeed eliminates administrative effort, but it's for ad-hoc operations. Constantly running (lakeflow is usually for streaming), would no way reduce costs, I would use a job cluster for that.
-
Raja419 👍 1 Answered AFor Lakeflow SDP pipelines, remember: - Serverless compute = Minimum management + automatic scaling. - Classic compute = More control over infrastructure, but more administration.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.