AWS Glue Workflow for Schema-Aware Redshift ETL
A data engineer must build an extract, transform, and load (ETL) pipeline to process and load data from 10 source systems into 10 tables that are in an Amazon Redshift database. All the source systems generate .csv, JSON, or Apache Parquet files every 15 minutes. The source systems all deliver files into one Amazon S3 bucket. The file sizes range from 10 MB to 20 GB. The ETL pipeline must function correctly despite changes to the data schema. Which data pipeline solutions will meet these requirements? (Choose two.)
Community Votes
100% of anonymous learners picked answer BD. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
The question tests the ability to handle dynamic schema changes in ETL pipelines; the trap is choosing static scheduled jobs over workflows with crawler integration.
This page explains how to build a robust AWS Glue workflow that handles schema evolution by integrating crawlers with jobs, ensuring correct data loading into Amazon Redshift.
Many learners select Option A because it is simpler, failing to realize that a standalone Glue job cannot automatically adapt to schema changes without manual intervention.
Community Discussion (9 comments)
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
Options B and D are correct because they utilize AWS Glue Workflows, which can orchestrate dependent tasks like crawlers and jobs. The requirement for the pipeline to "function correctly despite changes to the data schema" necessitates the use of an AWS Glue Crawler to update the Data Catalog before the ETL Job runs. Both options implement this pattern: B uses a scheduled EventBridge trigger, while D uses an S3 event trigger via Lambda to start the workflow. This ensures the schema is refreshed dynamically.Why the Other Options Are Wrong
Option A is incorrect because a standalone Glue Job does not automatically detect schema changes; it relies on the existing catalog definition, which would break if the source schema evolved. Option C is overly complex and introduces unnecessary latency by splitting the trigger logic across two Lambda functions instead of using a native workflow or direct invocation. Option E attempts to use Kinesis Data Firehose, which is not suitable for batch ETL loads requiring schema discovery and transformation logic typical of Glue Jobs, and it complicates the architecture unnecessarily.Community Comment Notes
The community strongly agreed on BD (94% votes). As rralucard_ noted, the key is the "streamlined process" of workflows handling schema changes. Another user, HagarTheHorrible, emphasized that "change od schema is the key," highlighting why static options fail. Some users mistakenly chose A, but comments clarified that A lacks the crawler step required for schema adaptability.Exam Strategy
Always look for keywords like 'schema changes' or 'evolution' in ETL questions. If present, the solution must include a mechanism to update the metadata catalog (like a Glue Crawler) before processing data. Use Workflows to chain these steps together.
Frequently Asked Questions
Why can't I just use a Glue Job triggered by EventBridge?
A standalone Glue Job uses the current Data Catalog definition. If the source schema changes, the job may fail or produce incorrect results because it doesn't auto-update the catalog.
What is the difference between Option B and Option D?
Option B is time-based (every 15 minutes), while Option D is event-based (triggered when files land in S3). Both meet the requirements, but D is more responsive to file arrival.
Related Analysis
Practice All DEA-C01 Questions
Access 100 questions with complete answers and detailed explanations.
View Full DEA-C01 Practice Test →