How to trigger ML pipelines only on new file uploads?

PMLE By islever Editorial Last updated

Short answer

C Create a pipeline in Vertex AI Pipelines. Create a Cloud Function that uses a Cloud Storage trigger and deploys the pipeline.

Question

You are creating an ML pipeline for data processing, model training, and model deployment that uses different Google Cloud services. You have developed code for each individual task, and you expect a high frequency of new files. You now need to create an orchestration layer on top of these tasks. You only want this orchestration pipeline to run if new files are present in your dataset in a Cloud Storage bucket. You also want to minimize the compute node costs. What should you do?
  1. Create a pipeline in Vertex AI Pipelines. Configure the first step to compare the contents of the bucket to the last time the pipeline was run. Use the scheduler API to run the pipeline periodically.
  2. Create a Cloud Function that uses a Cloud Storage trigger and deploys a Cloud Composer directed acyclic graph (DAG).
  3. Create a pipeline in Vertex AI Pipelines. Create a Cloud Function that uses a Cloud Storage trigger and deploys the pipeline. Correct Answer
  4. Deploy a Cloud Composer directed acyclic graph (DAG) with a GCSObjectUpdateSensor class that detects when a new file is added to the Cloud Storage bucket.

Key Takeaways

The question tests the ability to choose a cost-effective

To minimize costs and trigger an ML pipeline only upon new file arrivals

Common mistake: Selecting Option D (Cloud Composer with GCSObjectUpdateSensor) is a common mistake because while it technically fulfills the trigger requirement

Expert Analysis

Why the Answer Is Correct

Option C is correct because it utilizes a serverless architecture that incurs costs only when the event occurs. A Cloud Function triggered by a Cloud Storage event ensures the pipeline runs exactly when a new file is uploaded. Deploying a Vertex AI Pipeline from this function avoids the need for a persistent orchestration environment, significantly reducing compute node costs compared to solutions like Cloud Composer.

Why the Other Options Are Wrong

Option A is incorrect because the Scheduler API runs on a fixed schedule (polling), potentially executing the pipeline even when no new files exist, wasting resources. Option B is incorrect because Cloud Composer runs on GKE clusters that incur costs continuously regardless of pipeline activity. Option D is incorrect for the same cost reason; while GCSObjectUpdateSensor is effective for triggering, the underlying Cloud Composer environment requires expensive, always-on compute nodes.

Community Comment Notes

The community strongly supports Option C, highlighting the cost benefits of serverless components. Comment [3] explicitly notes that Cloud Composer incurs costs whereas Option C is serverless. Comments [1] and [2] reinforce that the event-driven model of Cloud Functions meets the requirement to run only when new files are present. While some users favored Option D for its sensor capabilities, the majority agreed that the cost constraint eliminates Cloud Composer as a viable choice.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When questions emphasize minimizing compute costs or "only run when X happens," look for serverless, event-driven solutions (like Cloud Functions) rather than managed clusters (like Cloud Composer or Dataproc), which have baseline costs regardless of activity.

Community Votes

C
80%
D
20%

80% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 9 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 9

  1. fitri001 👍 5 Answered C
    Option C appears to be the best choice for balancing the requirements of efficient orchestration, cost minimization, and ensuring the pipeline only runs when new files are present. By using a Cloud Function triggered by Cloud Storage events to deploy a Vertex AI Pipeline, you can leverage the event-driven model of Cloud Functions to minimize unnecessary runs and associated costs, while still using the powerful orchestration capabilities of Vertex AI Pipelines.
  2. Yan_X 👍 3 Answered C
    C Cloud Function to be triggered by Cloud storage trigger, and then deploy the Vertex AI pipeline.
  3. juliorevk 👍 2
    Probably C because while D would be good, it specifically says to minimize compute costs which cloud composer does incur whereas C is more serverless.
  4. Foxy2021 👍 1
    My answer is D: While C (Cloud Function + Vertex AI Pipelines) is a viable approach for triggering ML pipelines, D (Cloud Composer DAG with GCSObjectUpdateSensor) is the more appropriate and scalable solution when your orchestration spans multiple Google Cloud services and you want to minimize costs by only triggering the pipeline when new files appear.
  5. tardigradum 👍 1 Answered D
    The key here is "that uses different Google Cloud services". Taking this into account, Cloud Composer is the correct answer (for instance, Vertex AI pipelines is not integrated with classic Dataproc or Cloud Composer DAGs). Moreover, GCSObjectUpdateSensor is more efficient than a Cloud Function.
Show 4 more comments
  1. guilhermebutzke 👍 1 Answered B
    My Answer: B Cloud Function that uses a Cloud Storage trigger (”run if new files are present in your dataset in a Cloud Storage bucket”) and Cloud Composer directed acyclic graph (DAG) (”model deployment that uses different Google Cloud services”, ”orchestration layer on top of these tasks”,)
  2. Kili1 👍 1 Answered D
    "Different Google Cloud services" and GCSObjectUpdateSensor: This sensor class specifically checks for updates to Cloud Storage objects. This ensures the DAG only triggers when there's a new file in the bucket, minimizing unnecessary executions.
  3. JG123 👍 1
    Its C. Vertex pipelines are recommened to run ML pipeline!
  4. CHARLIE2108 👍 1
    Why not D?

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

How to preprocess BigQuery data for TensorFlow?

S-grade

How to Track Model and Prediction Lineage in Vertex AI?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide