Azure Databricks Lakeflow SDP Pipeline Expectations
You have a Lakeflow Spark Declarative Pipelines (SDP) pipeline in Azure Databricks. The pipeline ingests transaction data into a table named Table1. You need to ensure that in the event of an invalid record, the pipeline continues to run. The solution must meet the following requirements: Invalid records must NOT be written to Table1. Invalid records must be preserved for review. Minimize development effort. What should you do?
Community Insight
The question tests the specific behavior of Lakeflow SDP pipeline expectations, specifically that they filter bad records from the target table but save them to a quarantine location for review.
Learn how to handle invalid records in Azure Databricks Lakeflow Spark Declarative Pipelines without halting execution. This page explains why defining pipeline expectations is the correct approach for preserving data while minimizing development effort.
Many learners assume that because the prompt mentions 'quarantine,' option D must be correct. However, implementing advanced logic requires significant manual coding, violating the 'minimize development effort' constraint. Option C is the automated, low-code solution provided by the platform.
Community Discussion (1 comments)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Defining a pipeline expectation (Option C) is the intended mechanism in Lakeflow Spark Declarative Pipelines (SDP) for handling data quality issues. When an expectation fails (e.g., a record violates a schema or value rule), the pipeline automatically prevents that record from being written to the target table (Table1) and routes it to a specified quarantine location. This satisfies all requirements: the pipeline continues running, invalid records are excluded from the main table, they are preserved for review, and it requires minimal code compared to custom logic.Why the Other Options Are Wrong
Option A (CHECK constraint) is incorrect because constraints typically cause the entire transaction or batch to fail if violated, which contradicts the requirement for the pipeline to continue. Option B (WHERE clauses) filters data but does not preserve the invalid records anywhere; they are simply lost. Option D (Advanced logic) would work but violates the 'minimize development effort' requirement by requiring custom ETL code rather than using built-in pipeline features.Community Comment Notes
One commenter argued that expectations do not save invalid records, pointing to Option D. However, this reflects a misunderstanding of the feature's default behavior. In the context of DP-750 and modern lakehouse architectures, 'expectations' are explicitly designed to quarantine bad data automatically. The comment likely confuses simple filtering with the structured quarantine feature of SDP.Exam Strategy
Always look for keywords like 'minimize development effort' when choosing between custom code and platform-native features. If a platform feature exists to solve the problem (like pipeline expectations), it is almost always the correct answer over writing custom logic.
Frequently Asked Questions
Do pipeline expectations stop the pipeline if a record is invalid?
No. In Lakeflow SDP, failed expectations prevent the specific invalid record from being written to the target table and send it to quarantine, allowing the rest of the pipeline to complete successfully.
Why is Option D incorrect for minimizing development effort?
Option D requires writing custom code to identify, filter, and store bad records. Option C uses built-in declarative syntax, which is significantly faster to implement and maintain.