Triggering Airflow DAGs Reactively in Private Cloud Composer

PDE By islever Editorial Last updated

Short answer

C 1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance.

Question

You are deploying an Apache Airflow directed acyclic graph (DAG) in a Cloud Composer 2 instance. You have incoming files in a Cloud Storage bucket that the DAG processes, one file at a time. The Cloud Composer instance is deployed in a subnetwork with no Internet access. Instead of running the DAG based on a schedule, you want to run the DAG in a reactive way every time a new file is received. What should you do?
    1. Enable Private Google Access in the subnetwork, and set up Cloud Storage notifications to a Pub/Sub topic.
    1. Enable the Cloud Composer API, and set up Cloud Storage notifications to trigger a Cloud Function.
    1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance.
    Correct Answer
    1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance.

Key Takeaways

Tests event-driven architecture constraints in restricted VPCs; the common trap is assuming direct Cloud Function-to-Airflow REST API triggers work in Private IP environments when they officially fail due to egress and DNS limitations.

Reactive DAG execution in a Private IP Cloud Composer environment requires routing Cloud Storage events through Pub/Sub rather than direct service calls, as documented by Google for isolated networks. Community consensus confirms this pattern avoids connectivity blockers while maintaining scalability.

Common mistake: Candidates frequently select C or D, attempting to connect Cloud Functions directly to the Airflow REST API via PSC or VPC Serverless Access. This overlooks Google's explicit documentation stating that direct GCF-to-Airflow triggers are unsupported in Private IP and VPC Service Controls configurations.

Expert Analysis

Why the Answer Is Correct

Option A aligns with Google’s official architecture for reactive DAG execution in Private IP environments. By enabling Private Google Access and routing Cloud Storage object creation events to a Pub/Sub topic, you decouple storage from processing. A Cloud Function subscribed to the Pub/Sub topic can then securely trigger the DAG without requiring direct VPC-level connectivity to the Airflow web server.

Why the Other Options Are Wrong

Options C and D propose triggering the Airflow REST API directly from a Cloud Function. While technically viable in public or standard VPC setups, Google’s documentation explicitly notes this pattern fails in Private IP environments due to unresolvable internal endpoints and blocked egress. Option B incorrectly references the Cloud Composer API instead of leveraging event-driven Pub/Sub messaging, which is the standard scalable pattern for file-based triggers.

Community Comment Notes

Multiple high-voted comments ([2], [3], [7]) correctly cite official Google Cloud documentation confirming that direct GCF-to-Airflow triggers are unsupported in Private environments. Users emphasize that Pub/Sub acts as the necessary reliable bridge, making A the only architecturally sound choice under strict network isolation constraints. The consensus strongly favors A over C/D despite initial intuitive appeals to direct API calling.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

Always map explicit network constraints (like 'no Internet access' or 'Private IP') to Google’s supported integration patterns before selecting direct service-to-service options. When a question restricts outbound connectivity, default to managed message brokers like Pub/Sub as intermediaries rather than forcing direct REST API calls.

Community Votes

C
41%
A
35%
D
24%

41% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 10 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 10

  1. raaad 👍 12 Answered C
    - Enable Airflow REST API: In Cloud Composer, enable the "Airflow web server" option. - Set Up Cloud Storage Notifications: Create a notification for new files, routing to a Cloud Function. - Create PSC Endpoint: Establish a PSC endpoint for Cloud Composer. - Write Cloud Function: Code the function to use the Airflow REST API (via PSC endpoint) to trigger the DAG. ======== Why not Option D - Using the web server URL directly wouldn't work without internet access or a direct path to the web server.
  2. STEVE_PEGLEG 👍 6 Answered A
    This is the guidance how to use method in A: https://cloud.google.com/composer/docs/composer-2/triggering-gcf-pubsub "In this specific example, you create a Cloud Function and deploy two DAGs. The first DAG pulls Pub/Sub messages and triggers the second DAG according to the Pub/Sub message content." For C & D, this guidance says it can't be done when you have Private or VPS Service Controls set up: https://cloud.google.com/composer/docs/composer-2/triggering-with-gcf#check_your_environments_networking_configuration "This solution does not work in Private IP and VPC Service Controls configurations because it is not possible to configure connectivity from Cloud Functions to the Airflow web server in these configurations."
  3. josech 👍 4 Answered A
    C is not correct because "this solution does not work in Private IP and VPC Service Controls configurations because it is not possible to configure connectivity from Cloud Functions to the Airflow web server in these configurations". https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf The correct answer is A using Pub/Sub https://cloud.google.com/composer/docs/composer-2/triggering-gcf-pubsub
  4. d11379b 👍 3 Answered D
    The answer should be D Serverless VPC Access makes it possible for you to connect directly to your Virtual Private Cloud (VPC) network from serverless environments such as Cloud Run, App Engine, or Cloud Functions. Configuring Serverless VPC Access allows your serverless environment to send requests to your VPC network by using internal DNS and internal IP addresses (as defined by RFC 1918 and RFC 6598). The responses to these requests also use your internal network. You can use Serverless VPC Access to access Compute Engine VM instances, Memorystore instances, and any other resources with internal DNS or internal IP address. (Reference: https://cloud.google.com/vpc/docs/serverless-vpc-access) When you use Airflow Rest API to tigger the job, the url is based on the private IP address of Cloud Composer Instance, so you need to use Serverless VPC Access for it.
  5. chrissamharris 👍 3 Answered D
    Why not Option C? C involves creating a Private Service Connect (PSC) endpoint, which, while viable for creating private connections to Google services, adds complexity and might not be required when simpler solutions like VPC Serverless Access (as in Option D) can suffice.
Show 5 more comments
  1. Pime13 👍 2 Answered D
    Why Option D is the Best Choice: Airflow REST API: Enabling the Airflow REST API allows you to programmatically trigger DAG runs, which is essential for a reactive setup. Cloud Storage Notifications: Setting up notifications ensures that your DAG is triggered every time a new file is received in the Cloud Storage bucket. VPC Serverless Access: This allows your Cloud Function to securely access the Cloud Composer web server URL without needing external IP addresses, complying with your subnetwork's no Internet access constraint.
  2. baimus 👍 2 Answered A
    This is A, as steve_pegleg says, there is no way to connect the cloud function to the Airflow instance, without first enabling private access. The pubsub pattern makes sense in this context.
  3. Augustax 👍 1 Answered B
    Option B is the only viable solution because: It uses the Cloud Composer API, which is compatible with Private IP configurations. It leverages VPC Serverless Access to allow Cloud Functions to securely access the Airflow web server within the subnetwork. It avoids the limitations of the Airflow REST API in Private IP environments.
  4. scaenruy 👍 1 Answered C
    C. 1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance. 2. Create a Private Service Connect (PSC) endpoint. 3. Write a Cloud Function that connects to the Cloud Composer cluster through the PSC endpoint.
  5. Matt_108 👍 1 Answered C
    Option C, raaad explained well why

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to establish consistent BigQuery spend?

S-grade

How to resolve Cloud Composer 2 worker pod evictions and task failures?

S-grade

Which Google Cloud database for on-prem PostgreSQL with analytics?

S-grade

How to Improve Data Discoverability Across BigQuery, Pub/Sub, and PostgreSQL?

S-grade

← Back to PDE Study Guide