Automating Legacy ETL Pipelines with Cloud Composer

ADP By islever Editorial Last updated

Design and implement simple data pipelines. Schedule, automate, and monitor basic data processing tasks.

Short answer

D Create a direct acyclic graph (DAG) in Cloud Composer to orchestrate a pipeline trigger daily. Monitor the pipeline's execution using the Apache Airflow web interface and Cloud Monitoring.

Question

You work for a healthcare company. You have a daily ETL pipeline that extracts patient data from a legacy system, transforms it, and loads it into BigQuery for analysis. The pipeline currently runs manually using a shell script. You want to automate this process and add monitoring to ensure pipeline observability and troubleshooting insights. You want one centralized solution, using open-source tooling, without rewriting the ETL code. What should you do?
  1. Create a Cloud Run function that runs the pipeline daily. Monitor the function's execution using Cloud Monitoring.
  2. Configure Cloud Dataflow to implement the ETL pipeline, and use Cloud Scheduler to trigger the Dataflow pipeline daily. Monitor the pipeline's execution using the Dataflow job monitoring interface and Cloud Monitoring.
  3. Use Cloud Scheduler to trigger a Dataproc job to execute the pipeline daily. Monitor the job's progress using the Dataproc job web interface and Cloud Monitoring.
  4. Create a direct acyclic graph (DAG) in Cloud Composer to orchestrate a pipeline trigger daily. Monitor the pipeline's execution using the Apache Airflow web interface and Cloud Monitoring. Correct Answer

Key Takeaways

The main trap is confusing orchestration (Composer/Airflow) with execution engines (Dataflow/Dataproc); only Composer allows wrapping existing shell scripts as-is while providing rich monitoring.

This page explains how to automate legacy shell-based ETL pipelines on Google Cloud using Cloud Composer (Airflow) without rewriting code, ensuring full observability through its native web interface.

Common mistake: Choosing Cloud Dataflow (B) or Dataproc (C) because they are popular data processing tools; however, both require rewriting the existing shell script logic into Beam or Spark, violating the 'no rewrite' constraint.

Expert Analysis

Constraint Analysis: Open-Source Tooling and No Rewrite

The question mandates a centralized solution using open-source tooling that allows the existing ETL to run without rewriting. This eliminates options B and C immediately. Option B (Cloud Dataflow) requires rewriting the logic in Apache Beam (Java/Python), and Option C (Dataproc) involves running Spark/Hadoop jobs, which typically necessitates refactoring shell scripts into PySpark or Scala code. Furthermore, while Cloud Scheduler can trigger tasks, it lacks the robust orchestration features needed for complex pipeline monitoring and dependency management.

Orchestration vs. Execution

Option A suggests using Cloud Run with Cloud Monitoring. While Cloud Run is open-source compatible via Knative, it is primarily an execution environment, not an orchestration platform. It does not provide the built-in DAG visualization, retry policies, or historical logging required for "pipeline observability" in the context of managing daily ETL workflows. It also fails to address the need for a centralized orchestration layer effectively compared to dedicated tools.

Why Composer is Correct

Option D leverages Cloud Composer, which is Google Cloud's managed service for Apache Airflow. Airflow is an open-source platform designed specifically for orchestrating complex data workflows. The key advantage here is compatibility: you can wrap your existing shell script in a BashOperator within an Airflow DAG without modifying the underlying extraction, transformation, or loading logic. This satisfies the "no rewrite" constraint. Additionally, the Airflow web interface provides native visibility into task status, logs, and history, satisfying the requirement for monitoring and troubleshooting insights.

Conclusion

As noted in community discussions, Composer allows dropping existing scripts into a DAG using BashOperator, providing the necessary UI for logs and retries while maintaining open-source standards. Options B and C force a rewrite, and Option A lacks true orchestration capabilities.

Exam Strategy

When faced with questions requiring automation of existing scripts or workflows without code changes, prioritize managed orchestration services like Cloud Composer (Airflow). Look for keywords like 'open-source,' 'DAG,' and 'monitoring' which point directly to Airflow-based solutions rather than compute-heavy execution engines like Dataflow or Dataproc.

Frequently Asked Questions

Can I use Cloud Composer with existing shell scripts?

Yes. You can use the BashOperator in Apache Airflow to execute existing shell scripts directly within a DAG without needing to rewrite the logic.

Why not use Cloud Scheduler alone for this task?

Cloud Scheduler is good for simple time-based triggers but lacks the advanced orchestration features, dependency management, and detailed observability provided by Airflow in Cloud Composer.

More ADP FAQ →

Discussion 1 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 1

  1. Terralogiq 👍 1 Answered D
    Composer (managed Airflow) lets you drop your existing shell script into a DAG using a BashOperator, no rewrite script needed. Airflow's open-source, gives you a UI for logs/retries/history, and pairs with Cloud Monitoring for alerts. B and C both force you to rewrite the ETL (Beam/Spark), and A lacks real orchestration features.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Cleaning Large Datasets in Google Cloud Using BigQuery SQL

S-grade

Forecasting Product Demand with BigQueryML Time Series Models

A-grade

Streaming Data Pipeline Design with Dataflow and Pub/Sub

A-grade

Cloud Storage Bucket Retention Policy for Data Compliance

A-grade

Practice All ADP Questions

Access 65 questions with complete answers and detailed explanations.

View Full ADP Practice Test →

← Back to ADP Study Guide