Orchestrating Multi-Service ETL Pipelines with AWS Step Functions

DEA-C01 By islever Editorial Last updated

Orchestrate data pipelines. Automate data processing by using AWS services.

Short answer

B Use AWS Step Functions to coordinate AWS Glue and Amazon EMR jobs with minimal operational overhead.

Question

A company maintains multiple extract, transform, and load (ETL) workflows that ingest data from the company's operational databases into an Amazon S3 based data lake. The ETL workflows use AWS Glue and Amazon EMR to process data.
The company wants to improve the existing architecture to provide automated orchestration and to require minimal manual effort.
Which solution will meet these requirements with the LEAST operational overhead?
  1. AWS Glue workflows
  2. AWS Step Functions tasks Correct Answer
  3. AWS Lambda functions
  4. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) workflows

Key Takeaways

The main trap is assuming AWS Glue Workflows can orchestrate external services like EMR; they are restricted to Glue-native resources, necessitating Step Functions for cross-service coordination.

This page explains why AWS Step Functions is the optimal choice for orchestrating ETL workflows involving both AWS Glue and Amazon EMR, ensuring minimal operational overhead.

Common mistake: Option A (AWS Glue Workflows) is often chosen incorrectly because users associate 'ETL orchestration' solely with Glue. However, Glue Workflows cannot natively trigger EMR jobs, forcing manual workarounds that increase operational overhead.

Expert Analysis

The core of this question lies in distinguishing between service-specific orchestration and cross-service workflow management. The scenario explicitly states that the ETL workflows utilize both AWS Glue and Amazon EMR. This multi-service requirement is the critical constraint that determines the correct solution.

AWS Glue Workflows (Option A) are a managed orchestration tool, but they are limited to orchestrating AWS Glue jobs, crawlers, and triggers within the AWS Glue ecosystem. They cannot natively trigger or manage Amazon EMR clusters or jobs. Comments highlighting this limitation—such as those noting that "Glue workflows are only for orchestrating AWS Glue" or questioning how Glue can trigger EMR processes—are factually accurate regarding native capabilities. Relying on AWS Glue Workflows would require custom scripting (e.g., using boto3) to invoke EMR, which violates the "minimal manual effort" and "least operational overhead" requirements by introducing code maintenance and error handling logic.

Amazon MWAA (Option D) is a managed Apache Airflow service. While it can orchestrate both Glue and EMR, it requires defining DAGs (Directed Acyclic Graphs), managing Python environments, and configuring operators. Although managed, it still demands more developer effort and operational knowledge than a purpose-built, integrated AWS solution if one existed, but more importantly, Step Functions offers a simpler visual interface for linear/state-based flows without the complexity of Airflow's DAG syntax.

AWS Step Functions (Option B) is a serverless orchestration service designed specifically to coordinate components of distributed applications using state machines. It has native integration with both AWS Glue (via the StartJobRun API action available in its SDK/integrations) and Amazon EMR (via RunJobFlow or TerminateJobFlows). By using Step Functions, you can visually design a workflow that starts a Glue job, waits for completion, then launches an EMR cluster, runs tasks, and cleans up, all with zero infrastructure management. This provides the broadest compatibility across your existing stack while minimizing operational overhead compared to self-managed Airflow or custom Lambda functions.

Getting this wrong leads to either over-engineering (choosing MWAA) or architectural failure (choosing Glue Workflows). The key insight is recognizing that "orchestration" in a heterogeneous environment requires a service-agnostic coordinator like Step Functions.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When faced with questions about orchestrating multiple distinct AWS services (e.g., Glue + EMR, Lambda + SQS, DynamoDB + Step Functions), always look for a service-agnostic orchestrator like AWS Step Functions. Avoid choosing service-specific tools (like Glue Workflows or EventBridge schedules) if the workflow spans beyond that service's native boundaries. Evaluate options based on the breadth of integration capabilities rather than just the primary task type.

Frequently Asked Questions

Can AWS Glue Workflows trigger Amazon EMR jobs?

No, AWS Glue Workflows are designed exclusively for orchestrating AWS Glue resources such as jobs, crawlers, and triggers. They do not have native capabilities to start or manage Amazon EMR clusters.

Why not use Amazon MWAA for this scenario?

While Amazon MWAA can orchestrate both Glue and EMR, it requires writing and maintaining Apache Airflow DAGs. AWS Step Functions offers a lower-code, visual approach with native integrations, resulting in less operational overhead for standard sequential ETL flows.

More DEA-C01 FAQ →

Community Votes

B
59%
A
41%

59% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 23 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 23

  1. valuedate 👍 20 Answered B
    Glue Workflow only orchestrate crawlers and glue jobs
  2. DevoteamAnalytix 👍 9 Answered B
    For me it's B because I did not found a possibility how Glue can trigger/orchestrate EMR processes OOTB. But with StepFunction there is a way: https://aws.amazon.com/blogs/big-data/orchestrate-amazon-emr-serverless-jobs-with-aws-step-functions/
  3. lucas_rfsb 👍 6 Answered A
    Since it seems to me that this pipeline is complex, with multiple workflows, I would go for Glue workflows.
  4. GiorgioGss 👍 5 Answered B
    orchestrating = step function
  5. VerRi 👍 4 Answered B
    There is no way for Glue Workflow to trigger EMR
Show 18 more comments
  1. ottarg 👍 3
    Can you provide an example of Glue initiating an EMR job? Or somewhere in the documents? AFAIK, Glue workflows are only to be used for Glue related things e.g. pull data, transform it, and store it somewhere else (ETL). Executing commands on behalf of other services can be done using boto in glue, but it feels weird using Glue like that when you have step functions which are designed for orchestrating different services.
  2. rralucard_ 👍 3 Answered A
    Option A, AWS Glue Workflows, seems to be the best solution to meet the requirements with the least operational overhead. It offers a seamless integration with the company's existing AWS Glue and Amazon EMR setup, providing a managed and straightforward way to orchestrate their ETL workflows without extensive additional setup or manual intervention.
  3. jasango 👍 3
    Yo me voy por la D) Amazon MWAA porque Glue Workflows solo admite Jobs de Glue y Step Function puede fucionar pero no son workflows de datos. Amazon MWAA son workflows de datos y esta integrado tanto con Glue como EMR: https://aws.amazon.com/blogs/big-data/simplify-aws-glue-job-orchestration-and-monitoring-with-amazon-mwaa/
  4. ttpro1995 👍 3 Answered B
    We have both Glue job and EMR job, so we need Step Functions to connect those. Airflow can do it, but required more dev work.
  5. FunkyFresco 👍 3 Answered B
    EMR in workflows , i dont think so
  6. certplan 👍 2
    Here's an example of how you can use AWS Glue to initiate an EMR (Elastic MapReduce) job: Let's assume you have an AWS Glue job that performs ETL tasks on data stored in Amazon S3. You want to leverage EMR for a specific task within this job, such as running a complex Spark job. 1. Define a Glue Job: Create an AWS Glue job using the AWS Glue console, SDK, or CLI. Define the input and output data sources, as well as the transformations you want to apply. 2. Incorporate EMR Step: Within the Glue job script, include a section where you define an EMR step. An EMR step is a unit of work that performs a specific task on an EMR cluster. Code follows in the next entry...
  7. Rpathak4 👍 2 Answered A
    Why Not the Other Options? B. AWS Step Functions More flexible but requires manual setup of states and transitions for Glue & EMR. Higher operational overhead than Glue Workflows. C. AWS Lambda Lambda is not ideal for long-running ETL workflows. Best suited for lightweight data transformations or event-driven tasks. D. Amazon MWAA (Apache Airflow) More control but requires cluster management and custom DAGs. Higher maintenance than Glue Workflows.
  8. acoshi 👍 2 Answered A
    https://aws.amazon.com/blogs/big-data/orchestrate-an-etl-pipeline-using-aws-glue-workflows-triggers-and-crawlers-with-custom-classifiers/
  9. plutonash 👍 2 Answered B
    it is interesting to choose A for minimum effort but only step functions can trigger the work both on EMR and on GLUE jobs
  10. Palee 👍 2 Answered B
    The company wants to improve the existing architecture so A cannot be the right choice
  11. TonyStark0122 👍 2
    Glue Work flows
  12. V0811 👍 1 Answered A
    AWS Glue Workflows are specifically designed for orchestrating ETL jobs in AWS Glue. They allow you to define and manage complex workflows that include multiple jobs and triggers, all within the AWS Glue environment.Integration: AWS Glue workflows seamlessly integrate with other AWS Glue components, making it easier to manage ETL processes without the need for external orchestration tools.Minimal Operational Overhead: Since AWS Glue is a fully managed service, using Glue workflows will reduce the operational overhead compared to managing separate orchestrators or building custom solutions.While D. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is also a good choice for more complex orchestration, it may involve more management overhead compared to the more straightforward AWS Glue workflows. Thus, AWS Glue workflows provide the least operational overhead given the context of this scenario.
  13. Anonymous learner 👍 1
    Orchestrating different AWS services is a typical use case for Step Functions: https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html
  14. Deno_Dennis 👍 1 Answered B
    The answer is AWS Step Functions because you are orchestrating ETL workflows using BOTH AWS Glue and Amazon EMR to process data. AWS Glue Workflows is only for orchestrating AWS Glue.
  15. Adrifersilva 👍 1 Answered A
    glue workflows is part of the glue ecosystem so its provides seamless integration with minimal changes
  16. Shanmahi 👍 1 Answered A
    Glue workflows are managed services and best for considering least operational overhead.
  17. HunkyBunky 👍 1 Answered B
    B - because AWS Glue can't trigger EMR
  18. Shatheesh 👍 1
    Answer A, Glue workflows

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Amazon Redshift Compound Sort Key Query Optimization

S-grade

Secure Redshift Spectrum Access with VPC Endpoints

S-grade

Least Operational Overhead for Kinesis to Redshift Ingestion

S-grade

Athena Federated Query for Multi-Source Iceberg Pipelines

S-grade

Practice All DEA-C01 Questions

Access 100 questions with complete answers and detailed explanations.

View Full DEA-C01 Practice Test →

← Back to DEA-C01 Study Guide