Migrating ETL to AWS Glue Serverless Orchestration
Short answer
A Migrate all ETL jobs to AWS Glue and use AWS Glue workflows to orchestrate the pipeline, providing a fully managed serverless solution that runs Spark and automates execution.
Question
The company wants to move to a fully managed serverless solution that can orchestrate multiple ETL jobs and automate execution. The new solution must continue to use Spark to process data. The company needs to orchestrate and automate the ETL workflows with minimal manual intervention.
Which solution will meet these requirements?
Key Takeaways
AWS Glue is the native serverless alternative to EMR for Spark jobs, and Glue Workflows provide built-in orchestration, eliminating the need for external tools like Step Functions or EMR.
Learn how to migrate ETL workflows to a fully managed serverless solution using AWS Glue and Glue Workflows for automated Spark job orchestration.
Common mistake: Choosing Step Functions (B) is common because it offers powerful orchestration, but it fails to fully embrace the serverless compute migration by retaining EMR, whereas Glue Workflows offer a more integrated, serverless-native approach for Glue jobs.
Expert Analysis
The core requirement is a fully managed serverless solution that can orchestrate multiple ETL jobs, automate execution, and continue to use Spark. The key constraint is the shift from EMR (managed cluster) to a serverless option while retaining Spark capabilities.
AWS Glue is AWS's fully managed, serverless data integration service. It natively supports running Apache Spark jobs without managing underlying infrastructure (no EMR clusters). AWS Glue Workflows are specifically designed to orchestrate these ETL jobs, handle dependencies, and automate execution with minimal manual intervention. This directly satisfies all constraints: it is serverless, it uses Spark, and it provides orchestration.
Option B suggests using Step Functions and EventBridge to orchestrate both Glue and EMR. While this works, it does not meet the implicit goal of moving away from EMR to a purely serverless architecture as effectively as migrating the compute itself. More importantly, the question asks for a solution that moves to a serverless solution; keeping EMR in Option B contradicts the spirit of "moving to a fully managed serverless solution" by retaining the managed cluster component. Furthermore, Glue Workflows provide a more integrated, native orchestration experience for Glue jobs than manually stitching Step Functions.
Option C (Lambda) is unsuitable for large-scale transformations because Lambda has strict limits on memory, duration (15 minutes max), and is not optimized for heavy Spark-based batch processing.
Option D (Airflow on MWAA) is a managed orchestration tool, but it requires managing the Airflow environment and typically relies on external compute resources (like EMR or EC2) unless integrated deeply with other services. It is not as inherently "serverless-first" for the compute portion as Glue is, and the question emphasizes minimizing manual intervention for the entire pipeline including compute.
Therefore, migrating the Spark jobs to AWS Glue and using Glue Workflows for orchestration is the most aligned with the "fully managed serverless" and "minimal manual intervention" requirements.
Official Reference
- https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl.html
- https://docs.aws.amazon.com/glue/latest/dg/monitor-workflow.html
These links open the official vendor documentation in a new tab.
Exam Strategy
When a question specifies 'fully managed serverless' for data processing, prioritize AWS Glue over EMR or EC2-based solutions. If orchestration is also required, check if the service has native orchestration features (like Glue Workflows) before considering external orchestrators (like Step Functions or Airflow), as native integration often reduces complexity and management overhead.
Frequently Asked Questions
Can AWS Glue run existing Spark jobs?
Why not use Step Functions for orchestration?
Community Votes
50% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 2 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 2
-
Oxcart 👍 1 Answered AA is answer because AWS Glue is fully serverless Glue natively runs Spark jobs Glue Workflows can orchestrate: multiple ETL jobs triggers dependencies Minimal infrastructure management Designed exactly for serverless Spark ETL pipelines
-
AWS82siast 👍 1 Answered Bwhy convert from EMR to Glue just for orchestration? Step Function can handle both workflows and EventBridge to trigger Step Functions.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All DEA-C01 Questions
Access 100 questions with complete answers and detailed explanations.
View Full DEA-C01 Practice Test →