How to Orchestrate Event-Driven Table-Specific ETL Workflows?
You want to schedule a number of sequential load and transformation jobs. Data files will be added to a Cloud Storage bucket by an upstream process. There is no fixed schedule for when the new data arrives. Next, a Dataproc job is triggered to perform some transformations and write the data to BigQuery. You then need to run additional transformation jobs in BigQuery. The transformation jobs are different for every table. These jobs might take hours to complete. You need to determine the most efficient and maintainable workflow to process hundreds of tables and provide the freshest data to your end users. What should you do?
Community Votes
57% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
Tests the choice between monolithic parameterized DAGs versus isolated event-triggered DAGs; candidates often fall into the trap of over-optimizing for manageability at the expense of handling divergent transformation logic.
This scenario evaluates event-driven orchestration using Cloud Composer and Cloud Functions for asynchronous, table-specific data transformations. Community consensus confirms that isolating each table’s pipeline into its own DAG maximizes maintainability while ensuring immediate processing upon data arrival.
Option C is the most frequent incorrect choice, as test-takers assume a single shared DAG simplifies maintenance, but it becomes unwieldy when hundreds of tables require completely different transformation rules and long-running execution times.
Community Discussion (9 comments)
- Option D: Tailored handling and scheduling for each table; triggered by data arrival for more timely and efficient processing.
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Selecting option D aligns with Google Cloud best practices for highly variable, long-running workloads. Using a Cloud Storage trigger paired with a Cloud Function to invoke the Cloud Composer API ensures immediate, event-driven execution without polling. Creating a dedicated DAG per table isolates failure domains, simplifies debugging, and accommodates unique transformation logic without bloating a single workflow. This architecture scales efficiently across hundreds of tables while keeping data latency minimal.Why the Other Options Are Wrong
Options A and B incorrectly include Cloud Storage operators within Airflow; since data arrival is event-driven, Cloud Functions or Pub/Sub are far more efficient than Airflow polling for new objects. Option C proposes a single shared DAG, which contradicts the requirement for distinct per-table transformations. Managing hundreds of unique logic paths in one DAG leads to fragile dependencies, difficult version control, and cascading failure risks that violate maintainability standards.Community Comment Notes
Multiple users highlighted that the phrase “transformations are different for every table” directly signals the need for isolated DAGs rather than parameterized reuse [1][7]. While some argued that a single DAG with dynamic parameters improves maintainability [2], others correctly noted that hundreds of divergent workflows quickly exceed practical limits [3]. The majority agreed that Cloud Function-to-Composer API triggering is the optimal event-driven pattern for this use case [4][7].Official Reference
Exam Strategy
Always map keywords like “no fixed schedule,” “different for every table,” and “freshest data” to event-driven architectures (Cloud Functions/Pub/Sub triggering Composer). When workflows have highly divergent logic or long runtimes, prefer isolated DAGs over monolithic parameterized designs to preserve scalability and fault isolation.