How to trigger ML pipelines only on new file uploads?

You are creating an ML pipeline for data processing, model training, and model deployment that uses different Google Cloud services. You have developed code for each individual task, and you expect a high frequency of new files. You now need to create an orchestration layer on top of these tasks. You only want this orchestration pipeline to run if new files are present in your dataset in a Cloud Storage bucket. You also want to minimize the compute node costs. What should you do?

Create a pipeline in Vertex AI Pipelines. Configure the first step to compare the contents of the bucket to the last time the pipeline was run. Use the scheduler API to run the pipeline periodically.
Create a Cloud Function that uses a Cloud Storage trigger and deploys a Cloud Composer directed acyclic graph (DAG).
Create a pipeline in Vertex AI Pipelines. Create a Cloud Function that uses a Cloud Storage trigger and deploys the pipeline. Source Reference Answer
Deploy a Cloud Composer directed acyclic graph (DAG) with a GCSObjectUpdateSensor class that detects when a new file is added to the Cloud Storage bucket.

Community Votes

80%

20%

80% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

The question tests the ability to choose a cost-effective

To minimize costs and trigger an ML pipeline only upon new file arrivals

Selecting Option D (Cloud Composer with GCSObjectUpdateSensor) is a common mistake because while it technically fulfills the trigger requirement

Community Discussion (9 comments)

fitri001 👍 5 Selected: C

Option C appears to be the best choice for balancing the requirements of efficient orchestration, cost minimization, and ensuring the pipeline only runs when new files are present. By using a Cloud Function triggered by Cloud Storage events to deploy a Vertex AI Pipeline, you can leverage the event-driven model of Cloud Functions to minimize unnecessary runs and associated costs, while still using the powerful orchestration capabilities of Vertex AI Pipelines.

juliorevk 👍 2

Probably C because while D would be good, it specifically says to minimize compute costs which cloud composer does incur whereas C is more serverless.

Foxy2021 👍 1

My answer is D: While C (Cloud Function + Vertex AI Pipelines) is a viable approach for triggering ML pipelines, D (Cloud Composer DAG with GCSObjectUpdateSensor) is the more appropriate and scalable solution when your orchestration spans multiple Google Cloud services and you want to minimize costs by only triggering the pipeline when new files appear.

tardigradum 👍 1 Selected: D

The key here is "that uses different Google Cloud services". Taking this into account, Cloud Composer is the correct answer (for instance, Vertex AI pipelines is not integrated with classic Dataproc or Cloud Composer DAGs). Moreover, GCSObjectUpdateSensor is more efficient than a Cloud Function.

Kili1 👍 1 Selected: D

"Different Google Cloud services" and GCSObjectUpdateSensor: This sensor class specifically checks for updates to Cloud Storage objects. This ensures the DAG only triggers when there's a new file in the bucket, minimizing unnecessary executions.

CHARLIE2108 👍 1

Why not D?

Yan_X 👍 3 Selected: C

C Cloud Function to be triggered by Cloud storage trigger, and then deploy the Vertex AI pipeline.

JG123 👍 1

Its C. Vertex pipelines are recommened to run ML pipeline!

guilhermebutzke 👍 1 Selected: B

My Answer: B Cloud Function that uses a Cloud Storage trigger (”run if new files are present in your dataset in a Cloud Storage bucket”) and Cloud Composer directed acyclic graph (DAG) (”model deployment that uses different Google Cloud services”, ”orchestration layer on top of these tasks”,)

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

Option C is correct because it utilizes a serverless architecture that incurs costs only when the event occurs. A Cloud Function triggered by a Cloud Storage event ensures the pipeline runs exactly when a new file is uploaded. Deploying a Vertex AI Pipeline from this function avoids the need for a persistent orchestration environment, significantly reducing compute node costs compared to solutions like Cloud Composer.

Why the Other Options Are Wrong

Option A is incorrect because the Scheduler API runs on a fixed schedule (polling), potentially executing the pipeline even when no new files exist, wasting resources. Option B is incorrect because Cloud Composer runs on GKE clusters that incur costs continuously regardless of pipeline activity. Option D is incorrect for the same cost reason; while GCSObjectUpdateSensor is effective for triggering, the underlying Cloud Composer environment requires expensive, always-on compute nodes.

Community Comment Notes

The community strongly supports Option C, highlighting the cost benefits of serverless components. Comment [3] explicitly notes that Cloud Composer incurs costs whereas Option C is serverless. Comments [1] and [2] reinforce that the event-driven model of Cloud Functions meets the requirement to run only when new files are present. While some users favored Option D for its sensor capabilities, the majority agreed that the cost constraint eliminates Cloud Composer as a viable choice.

Official Reference

Exam Strategy

When questions emphasize minimizing compute costs or "only run when X happens," look for serverless, event-driven solutions (like Cloud Functions) rather than managed clusters (like Cloud Composer or Dataproc), which have baseline costs regardless of activity.

Related Analysis

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide