Orchestrating Dataprep with Variable Dependencies in Cloud Composer

PDE By islever Editorial Last updated

Planning the data pipelines. Designing automation and repeatability.

Short answer

D Export the Dataprep job as a Dataflow template, and incorporate it into a Composer job to handle the variable-execution-time dependency via DAG orchestration.

Question

You used Dataprep to create a recipe on a sample of data in a BigQuery table. You want to reuse this recipe on a daily upload of data with the same schema, after
the load job with variable execution time completes. What should you do?
  1. Create a cron schedule in Dataprep.
  2. Create an App Engine cron job to schedule the execution of the Dataprep job.
  3. Export the recipe as a Dataprep template, and create a job in Cloud Scheduler.
  4. Export the Dataprep job as a Dataflow template, and incorporate it into a Composer job. Correct Answer

Key Takeaways

When a downstream task depends on the completion of an upstream task with unpredictable duration, use an orchestrator like Cloud Composer (Airflow) rather than a time-based scheduler like Cloud Scheduler or native Cron.

Learn how to handle variable execution time dependencies when reusing Dataprep recipes by leveraging Cloud Composer's DAG capabilities instead of fixed cron schedules.

Common mistake: Candidates often choose Option A (Cron schedule in Dataprep) because Dataprep supports scheduling. However, this fails to address the 'variable execution time' constraint, risking data inconsistency if the scheduled time does not align with the load job's completion.

Expert Analysis

The critical constraint in this question is the dependency on a BigQuery load job with variable execution time. The requirement is to trigger the Dataprep recipe only after that upstream load completes. This introduces an event-driven dependency rather than a simple time-based schedule.

Option A (Create a cron schedule in Dataprep) and Option B (App Engine cron job) are incorrect because they rely on fixed time intervals. Since the BigQuery load has "variable execution time," a fixed cron schedule cannot guarantee that the transformation runs after the data is ready; it might run before, during, or long after the load finishes, leading to processing empty or incomplete data.

Option D is the correct approach for handling complex dependencies. While modern Dataprep recipes often execute as Dataflow jobs under the hood, the exam logic focuses on orchestration. Cloud Composer (managed Apache Airflow) is designed specifically to manage Directed Acyclic Graphs (DAGs) where tasks depend on the completion of other tasks. By incorporating the workflow into a Composer job, you can create a DAG that monitors the BigQuery load status and triggers the subsequent Dataprep/Dataflow task only upon success. This ensures the pipeline respects the variable runtime of the source load.

Option C suggests using Cloud Scheduler, which is a time-based scheduler similar to Cron. It lacks the native ability to listen for specific event completions from BigQuery without additional custom code or Pub/Sub integration, making it less suitable than Composer for explicit job-to-job dependency management in this context.

Official Reference

https://cloud.google.com/composer/docs

This link opens the official vendor documentation in a new tab.

Exam Strategy

Always scan for keywords indicating uncertainty or variability in timing, such as 'variable,' 'unknown,' or 'depends on.' When these appear alongside multiple GCP services, prioritize managed orchestration tools like Cloud Composer (Airflow) over simple schedulers like Cloud Scheduler, as they provide the necessary state-tracking and dependency management features.

Frequently Asked Questions

Why can't I just use Cloud Scheduler for this?

Cloud Scheduler is time-based. If the upstream BigQuery load takes longer than expected, a scheduled trigger might run before the data is available, causing errors or missing data.

Does Dataprep always run as a Dataflow job?

Yes, Dataprep recipes are executed using Google Cloud Dataflow. Therefore, exporting as a Dataflow template allows it to be integrated into broader Dataflow-based workflows in Composer.

More PDE FAQ →

Community Votes

D
58%
A
25%
C
17%

58% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 19 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 19

  1. jkhong 👍 12
    I'd pick D because it's the only option which allows variable execution (since we need to execute the dataprep job only after the prior load job). Although D suggests the export of Dataflow templates, this discussion suggests that the export option is no longer available (https://stackoverflow.com/questions/72544839/how-to-get-the-dataflow-template-of-a-dataprep-job), there are already Airflow Operators for Dataprep which we should be using instead - https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/dataprep.html
  2. midgoo 👍 9 Answered D
    Since the load job execution time is unexpected, schedule the Dataprep based on a fixed time window may not work. When the Dataprep job run the first time, we can find the Dataflow job for that in the console. We can use that to create the Template --> With the help of the Composer to determine if the load job is completed, we can then trigger the Dataflow job
  3. jkhong 👍 4
    The question mentions after a load job with variable time, i dont think setting a dataprep cron job can address the issue of variable load times
  4. AWSandeep 👍 4 Answered D
    D. Export the Dataprep job as a Dataflow template, and incorporate it into a Composer job. Reveal Solution
  5. anicloudgirl 👍 4 Answered A
    It's A. You can set it directly in Dataprep a job and it will use Dataflow under the hood.
Show 14 more comments
  1. lucaluca1982 👍 4 Answered C
    I think C it is more straighforward
  2. gaurav0480 👍 3
    The key here is "after the load job with variable execution time completes" which means the execution of this job depends on the completion of another job which has a variable execution time. Hence D
  3. musumusu 👍 3
    Answer C: Use Recipe Template feature of dataprep. Don't need to change the service.
  4. TVH_Data_Engineer 👍 2 Answered A
    Dataprep by Trifacta allows you to schedule the execution of recipes. You can set up a cron schedule directly within Dataprep to automatically run your recipe at specified intervals, such as daily. WHY NOT D ? : This option involves significant additional complexity. Exporting the Dataprep job as a Dataflow template and then incorporating it into a Composer (Apache Airflow) job is a more complicated process and is typically used for more complex orchestration needs that go beyond simple scheduling.
  5. Adswerve 👍 2 Answered A
    A is correct. D is too complicated. A is correct, because you can schedule a job right from Dataprep UI. https://cloud.google.com/blog/products/gcp/scheduling-and-sampling-arrive-for-google-cloud-dataprep Scheduling and sampling arrive for Google Cloud Dataprep Throughout our early releases, users’ most common request has been Flow scheduling. As of Thursday’s release, Flows can be scheduled with minute granularity at any frequency.
  6. anicloudgirl 👍 2
    It's A. You can set it directly in Dataprep a job and it will use Dataflow under the hood. No need to export nor incorporate into a Composer job. Dataprep by trifacta - https://docs.trifacta.com/display/DP/cron+Schedule+Syntax+Reference Dataprep job uses dataflow - https://cloud.google.com/dataprep
  7. John_Pongthorn 👍 2 Answered D
    Dataprep and Dataflow are same famitly
  8. damaldon 👍 2
    It’s D, use composer to schedule tasks
  9. cloudmon 👍 2 Answered D
    It's D
  10. carbonscave 👍 1 Answered D
    Why D is correct Dependency Management: The crucial constraint in the prompt is the variable execution time of the upstream load job. Cloud Composer (managed Apache Airflow) is an orchestration tool built to handle dependencies using Directed Acyclic Graphs (DAGs). It can monitor the unpredictable load job and trigger the transformation only after the load is fully complete. Native Execution: Under the hood, Cloud Dataprep translates its recipes into Apache Beam pipelines and executes them on Cloud Dataflow. You can easily export a Dataprep recipe as a Dataflow template and trigger it programmatically using Composer's DataflowTemplateOperator.
  11. MaxNRG 👍 1 Answered D
    We have external dependency "after the load job with variable execution time completes" which requires DAG -> Airflow (Cloud Composer) The reasons: A scheduler like Cloud Scheduler won't handle the dependency on the BigQuery load completion time Using Composer allows creating a DAG workflow that can: Trigger the BigQuery load Wait for BigQuery load to complete Trigger the Dataprep Dataflow job Dataflow template allows easy reuse of the Dataprep transformation logic Composer coordinates everything based on the dependencies in an automated workflow
  12. god_brainer 👍 1
    This approach ensures the dynamic triggering of the Dataprep job based on the completion of the preceding load job, ensuring data is processed accurately and in sequen
  13. rocky48 👍 1 Answered D
    I'd pick D because it's the only option which allows variable execution
  14. jroig_ 👍 1 Answered C
    Why not C?

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

Triggering Airflow DAGs Reactively in Private Cloud Composer

S-grade

Monitoring Cloud Bigtable for Cluster Scaling Decisions

S-grade

How to resolve Cloud Composer 2 worker pod evictions and task failures?

S-grade

← Back to PDE Study Guide