Azure Databricks Lakeflow SDP Pipeline Expectations
Short answer
C Define a pipeline expectation to filter invalid records into quarantine while allowing the pipeline to continue processing valid data.
Question
You need to ensure that in the event of an invalid record, the pipeline continues to run. The solution must meet the following requirements:
Invalid records must NOT be written to Table1.
Invalid records must be preserved for review.
Minimize development effort.
What should you do?
Key Takeaways
The question tests the specific behavior of Lakeflow SDP pipeline expectations, specifically that they filter bad records from the target table but save them to a quarantine location for review.
Learn how to handle invalid records in Azure Databricks Lakeflow Spark Declarative Pipelines without halting execution. This page explains why defining pipeline expectations is the correct approach for preserving data while minimizing development effort.
Common mistake: Many learners assume that because the prompt mentions 'quarantine,' option D must be correct. However, implementing advanced logic requires significant manual coding, violating the 'minimize development effort' constraint. Option C is the automated, low-code solution provided by the platform.
Expert Analysis
Why the Answer Is Correct
Defining a pipeline expectation (Option C) is the intended mechanism in Lakeflow Spark Declarative Pipelines (SDP) for handling data quality issues. When an expectation fails (e.g., a record violates a schema or value rule), the pipeline automatically prevents that record from being written to the target table (Table1) and routes it to a specified quarantine location. This satisfies all requirements: the pipeline continues running, invalid records are excluded from the main table, they are preserved for review, and it requires minimal code compared to custom logic.
Why the Other Options Are Wrong
Option A (CHECK constraint) is incorrect because constraints typically cause the entire transaction or batch to fail if violated, which contradicts the requirement for the pipeline to continue. Option B (WHERE clauses) filters data but does not preserve the invalid records anywhere; they are simply lost. Option D (Advanced logic) would work but violates the 'minimize development effort' requirement by requiring custom ETL code rather than using built-in pipeline features.
Community Comment Notes
One commenter argued that expectations do not save invalid records, pointing to Option D. However, this reflects a misunderstanding of the feature's default behavior. In the context of DP-750 and modern lakehouse architectures, 'expectations' are explicitly designed to quarantine bad data automatically. The comment likely confuses simple filtering with the structured quarantine feature of SDP.
Exam Strategy
Always look for keywords like 'minimize development effort' when choosing between custom code and platform-native features. If a platform feature exists to solve the problem (like pipeline expectations), it is almost always the correct answer over writing custom logic.
Frequently Asked Questions
Do pipeline expectations stop the pipeline if a record is invalid?
Why is Option D incorrect for minimizing development effort?
Discussion 1 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 1
-
51e3df9 👍 1 Answered DDefining pipeline expectations (C), does NOT save invalid records for review.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All DP-750 Questions
Access 139 questions with complete answers and detailed explanations.
View Full DP-750 Practice Test →