How to Orchestrate Event-Driven Table-Specific ETL Workflows?
Short answer
D 1. Create an Apache Airflow directed acyclic graph (DAG) in Cloud Composer with sequential tasks by using the Dataproc and BigQuery operators.
Question
Key Takeaways
Tests the choice between monolithic parameterized DAGs versus isolated event-triggered DAGs; candidates often fall into the trap of over-optimizing for manageability at the expense of handling divergent transformation logic.
This scenario evaluates event-driven orchestration using Cloud Composer and Cloud Functions for asynchronous, table-specific data transformations. Community consensus confirms that isolating each table’s pipeline into its own DAG maximizes maintainability while ensuring immediate processing upon data arrival.
Common mistake: Option C is the most frequent incorrect choice, as test-takers assume a single shared DAG simplifies maintenance, but it becomes unwieldy when hundreds of tables require completely different transformation rules and long-running execution times.
Expert Analysis
Why the Answer Is Correct
Selecting option D aligns with Google Cloud best practices for highly variable, long-running workloads. Using a Cloud Storage trigger paired with a Cloud Function to invoke the Cloud Composer API ensures immediate, event-driven execution without polling. Creating a dedicated DAG per table isolates failure domains, simplifies debugging, and accommodates unique transformation logic without bloating a single workflow. This architecture scales efficiently across hundreds of tables while keeping data latency minimal.
Why the Other Options Are Wrong
Options A and B incorrectly include Cloud Storage operators within Airflow; since data arrival is event-driven, Cloud Functions or Pub/Sub are far more efficient than Airflow polling for new objects. Option C proposes a single shared DAG, which contradicts the requirement for distinct per-table transformations. Managing hundreds of unique logic paths in one DAG leads to fragile dependencies, difficult version control, and cascading failure risks that violate maintainability standards.
Community Comment Notes
Multiple users highlighted that the phrase “transformations are different for every table” directly signals the need for isolated DAGs rather than parameterized reuse [1][7]. While some argued that a single DAG with dynamic parameters improves maintainability [2], others correctly noted that hundreds of divergent workflows quickly exceed practical limits [3]. The majority agreed that Cloud Function-to-Composer API triggering is the optimal event-driven pattern for this use case [4][7].
Exam Strategy
Always map keywords like “no fixed schedule,” “different for every table,” and “freshest data” to event-driven architectures (Cloud Functions/Pub/Sub triggering Composer). When workflows have highly divergent logic or long runtimes, prefer isolated DAGs over monolithic parameterized designs to preserve scalability and fault isolation.
Community Votes
57% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 9 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 9
-
cuadradobertolinisebastiancami 👍 8D Transformations are in Dataproc and BigQuery. So you don't need operators for GCS (A and B can be discard) "There is no fixed schedule for when the new data arrives." so you trigger the DAG when a file arrives * "The transformation jobs are different for every table. " so you need a DAG for each table. Then, D is the most suitable answer
-
f74ca0c 👍 3 Answered CA single shared DAG is efficient to manage, and table-specific transformations can be handled using parameters (e.g., passing table names and configurations dynamically). Triggering the DAG using a Cloud Storage object notification and a Cloud Function ensures the workflow starts immediately upon data arrival. Event-driven architecture minimizes delays and provides the freshest data to users. Efficient, maintainable, and event-driven.
-
8ad5266 👍 3 Answered CThis explains why it's not D: maintainable workflow to process hundreds of tables and provide the freshest data to your end users How is creating a DAG for each of the hundreds of tables maintainable?
-
Matt_108 👍 3 Answered DOption D, which gets triggered when the data comes in and accounts for the fact that each table has its own set of transformations
-
raaad 👍 2 Answered D- Option D: Tailored handling and scheduling for each table; triggered by data arrival for more timely and efficient processing.
Show 4 more comments
-
scaenruy 👍 1 Answered DD. 1. Create an Apache Airflow directed acyclic graph (DAG) in Cloud Composer with sequential tasks by using the Dataproc and BigQuery operators. 2. Create a separate DAG for each table that needs to go through the pipeline. 3. Use a Cloud Storage object trigger to launch a Cloud Function that triggers the DAG.
-
choprat1 👍 1 Answered Dmanaging indidivuals DAGs is the best way when they're too different
-
Jordan18 👍 3why not C?
-
JyoGCP 👍 1 Answered DOption D
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.