How Should You Configure Compute for Databricks Telemetry Ingestion?
This is a case study. Case studies are not timed separately from other exam sections. You can use as much exam time as you would like to complete each case study. However, there might be additional case studies or other exam sections. Manage your time to ensure that you can complete all the exam sections in the time provided. Pay attention to the Exam Progress at the top of the screen so you have sufficient time to complete any exam sections that follow this case study. To answer the case study questions, you will need to reference information that is provided in the case. Case studies and associated questions might contain exhibits or other resources that provide more information about the scenario described in the case. Information provided in an individual question does not apply to the other questions in the case study. A Review Screen will appear at the end of this case study. From the Review Screen, you can review and change your answers before you move to the next exam section. After you leave this case study, you will NOT be able to return to it. To start the case study To display the first question in this case study, select the “Next” button. To the left of the question, a menu provides links to information such as business requirements, the existing environment, and problem statements. Please read through all this information before answering any questions. When you are ready to answer a question, select the “Question” button to return to the question. Overview Company Information Contoso, Inc. is a renewable energy provider that operates solar and wind farms across North America. Existing Environment Azure Environment Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog. Workspace1 contains all-purpose clusters for both development and production workloads. The company's Azure environment contains: In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data Data Environment Contoso ingests the following operational and business data: Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files frequently experience schema drift. Maintenance logs: Maintenance systems generate historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts. Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server. External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts. ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information. Problem Statements The company’s existing analytics environment has several issues: Ingestion Telemetry pipelines fall behind during peak loads. Telemetry ingestion fails when schema drift occurs. Streaming pipelines reprocess events after a pipeline restarts. Compute Production and development workloads run on the same all-purpose clusters. Production and development workloads do NOT support autoscaling or workload isolation. Governance The ERP data is duplicated across systems and development teams. Naming conventions are inconsistent across development teams, regions, and products. Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership. Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required. Pipeline operations Pipelines lack resiliency, alerting, and centralized scheduling. Requirements Planned Changes Contoso plans to implement the following changes: Implement scalable data pipeline orchestration. Create a managed analytics catalog in Unity Catalog. Implement a consistent approach to creating curated datasets. Establish a centralized governance model across ingestion, cleansed, and curated layers. Grant data engineers access to the ERP tables by using minimal development effort. Adopt a compute strategy that isolates production workloads and supports autoscaling. Adopt a slowly changing dimension (SCD) approach to address current data modeling issues. Technical Requirements Contoso identifies the following environment and compute requirements: Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes. Provide fast and consistent performance for business intelligence (BI) workloads. Prevent development activity from affecting production pipelines. Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters. Contoso identifies the following data ingestion and processing requirements: Auto-scale ingestion pipelines to handle bursty workloads. Handle schema drift for the maintenance and telemetry data. Ingest file-based telemetry data by using minimal operational effort. Store all the ingested data in a format that supports incremental processing. Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics. Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server. Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog. Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data. Contoso identifies the following data modeling and optimization requirements: Build curated tables that standardize business logic. Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required. Contoso identifies the following pipeline deployment and operation requirements: Orchestrate multi-step ingestion and transformation workflows. Define a clear execution order and dependencies. Automatically retry failed steps and notify operators. Schedule ingestion and transformation workloads consistently. Governance Requirements Contoso identifies the following governance requirements: Centralize the metadata catalog. Provide isolated development areas that follow standard naming conventions. Establish a consistent structure for organizing raw, cleansed, and curated data. Provide a read-only mechanism to reference the ERP data through a foreign catalog. Business Requirements Contoso identifies the following business requirements: Improve ingestion reliability and reduce operational effort. Standardize data definitions across development teams. You need to configure compute for the ingestion of telemetry data. The solution must meet the data ingestion and processing requirements. What should you do?
Community Insight
This item tests whether you can map the case's compute requirements (autoscaling, isolation, non-interactive execution) onto the right Databricks compute type and feature, and the trap is choosing a bigger all-purpose or shared cluster to fix a throughput problem.
Contoso needs production telemetry ingestion to run on compute that autoscales during bursty event-hub loads while staying isolated from development. The correct move is to enable Photon acceleration on a Databricks job compute cluster so the scheduled, non-interactive pipeline runs on its own scalable compute.
Many candidates pick A and move ingestion to shared compute, assuming shared access mode is the 'modern' way to run pipelines, but shared/interactive compute breaks production isolation and the requirement that ingestion run as scheduled non-interactive jobs rather than on interactive development clusters.
Community Discussion (1 comments)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Choices are judged against the case's technical requirements, which state that production ingestion must run on compute clusters that scale automatically during telemetry spikes, must be isolated from development activity, and must run as scheduled non-interactive pipelines rather than a shared interactive cluster. Enabling Photon acceleration for a job compute cluster satisfies all three: a job cluster is non-interactive and per-run isolated, job clusters support autoscaling so bursty event-hub loads are absorbed, and Photon's vectorized engine accelerates the Spark DataFrame/Structured Streaming work that cleanses raw events before publishingtelemetry.raw_events. Photon also has no effect on the exactly-once or schema-drift semantics, which are handled by the streaming checkpoint and cloudFiles/readStream options, so it is a safe, additive optimization rather than a re-architecture. That makes B the only option that both keeps telemetry ingestion on the correct compute type and improves ingestion throughput.Why the Other Options Are Wrong
Option A moves the ingestion pipelines to shared compute, which is designed for interactive multi-user access rather than for scheduled production pipelines; it undermines the requirement to keep development activity from affecting production and does not deliver per-run autoscaling isolation. Option C increases an all-purpose cluster to a larger fixed node type, but all-purpose clusters are exactly what the case says production pipelines should move away from, and a fixed node type removes the autoscaling the bursty telemetry workload requires. Option D disables autoscaling for a job compute cluster, which directly contradicts the stated requirement to auto-scale ingestion pipelines for bursty workloads; a correctly sized fixed job cluster cannot keep up with 40,000 sensors emitting every few seconds. None of these options addresses both the compute-isolation and elastic-scaling requirements the way B does.Community Comment Notes
Only one participant, dgerok, recorded an answer here, selecting B and observing that when a Databricks question asks how to improve ingestion or transformation performance "with minimal changes, Photon is often the expected answer," which aligns with the case's demand for higher ingestion throughput without re-platforming. Independent scoring confirms the pick: B is the only option consistent with an autoscaling job cluster dedicated to scheduled telemetry ingestion. The low vote count means the result rests on the case requirements and Databricks compute documentation rather than on community consensus.Official Reference
Exam Strategy
When a DP-750 case study lists autoscaling, production isolation and non-interactive execution together, eliminate every option that mentions shared compute, all-purpose clusters or disabling autoscaling; the surviving answer is a job compute cluster, optionally with Photon. Photon is the safe performance pick whenever the question asks how to speed up ingestion or transformation without restructuring the pipeline.
Frequently Asked Questions
Why is moving telemetry ingestion to shared compute wrong here?
Shared compute is interactive and mixes workloads across users, so it cannot guarantee production isolation or per-run autoscaling for scheduled, non-interactive telemetry pipelines.
Does Photon alone satisfy the autoscaling requirement?
No. Photon is a performance accelerator; autoscaling comes from running ingestion on an autoscaling job compute cluster instead of a fixed-node all-purpose cluster.