Optimizing AWS Glue Job Performance with S3 Partitioning and Scaling
Short answer
AB Partition the data in the S3 bucket by year, month, and day, and increase the AWS Glue instance size by scaling up the worker type to reduce job execution time.
Question
A data engineer discovers that dashboard queries are becoming slower over time. The data engineer determines that the root cause of the slowing queries is long-running AWS Glue jobs.
Which actions should the data engineer take to improve the performance of the AWS Glue jobs? (Choose two.)
Key Takeaways
The question tests knowledge of two primary methods for optimizing AWS Glue: reducing data scanned via S3 partitioning and increasing compute capacity via worker scaling.
Learn how to fix slow AWS Glue jobs by partitioning S3 data by date and scaling up worker types. This guide covers DEA-C01 best practices for improving ETL performance.
Common mistake: Candidates often confuse dynamic schema handling (DynamicFrame) with performance tuning. While useful for unstructured data, it does not inherently speed up jobs and may add overhead compared to targeted partitioning or scaling.
Expert Analysis
The scenario describes a data pipeline where AWS Glue jobs are becoming slower as data volume increases in Amazon S3, causing downstream performance issues for QuickSight. To resolve long-running ETL jobs, the engineer must address both data storage efficiency and compute resource allocation.
Option A is correct because partitioning data in S3 by date (year, month, day) allows AWS Glue to perform predicate pushdown. Instead of scanning the entire bucket, the job reads only the relevant partitions, drastically reducing I/O and processing time. This is a fundamental best practice for optimizing Glue performance on growing datasets.
Option B is correct because increasing the instance size (scaling up worker type or DPU count) provides more memory and CPU power to the Glue job. For complex transformations or large datasets that cannot be fully mitigated by partitioning alone, adding resources reduces execution time directly.
Option C is incorrect because DynamicFrames are used for schema flexibility, not performance optimization; they can sometimes introduce overhead compared to optimized Spark DataFrames if not managed carefully.
Option D is incorrect because changing scheduling frequency does not affect the runtime duration or efficiency of the job itself.
Option E is incorrect because IAM permissions control access, not processing speed.
Official Reference
- https://docs.aws.amazon.com/glue/latest/dg/how-it-works.html#how-partitioning-works
- https://docs.aws.amazon.com/glue/latest/dg/add-job.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html
These links open the official vendor documentation in a new tab.
Exam Strategy
When troubleshooting slow ETL jobs, always evaluate data layout first. Partitioning S3 data minimizes scan costs. If partitioning isn't enough or the job is compute-bound, scale the resources (DPU/Worker). Eliminate options related to security (IAM) or scheduling unless explicitly asked about cost or timing rather than runtime duration.
Frequently Asked Questions
Why does partitioning S3 data improve AWS Glue performance?
Can I just increase the number of workers instead of partitioning?
Community Votes
50% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 7 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 7
-
rralucard_ 👍 11A. Partition the data that is in the S3 bucket. Organize the data by year, month, and day. • Partitioning data in Amazon S3 can significantly improve query performance. By organizing the data by year, month, and day, AWS Glue and Amazon QuickSight can scan only the relevant partitions of data, which reduces the amount of data read and processed. This approach is particularly effective for time-series data, where queries often target specific time ranges. B. Increase the AWS Glue instance size by scaling up the worker type. • Scaling up the worker type can provide more computational resources to the AWS Glue jobs, enabling them to process data faster. This can be especially beneficial when dealing with large datasets or complex transformations. It’s important to monitor the performance improvements and cost implications of scaling up.
-
certplan 👍 21. Partition the Data in Amazon S3: - AWS documentation on optimizing Amazon S3 performance: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html - AWS Glue documentation on partitioning data for AWS Glue jobs: https://docs.aws.amazon.com/glue/latest/dg/how-it-works.html#how-partitioning-works - Best practices for partitioning in Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/best-practices-partitioning.html 2. Optimizing AWS Glue Job Settings: - AWS Glue documentation on optimizing job performance: https://docs.aws.amazon.com/glue/latest/dg/best-practices.html - AWS Glue documentation on scaling AWS Glue job resources: https://docs.aws.amazon.com/glue/latest/dg/monitor-profile-glue-job-cloudwatch-metrics.html By referring to these documentation resources, the data engineer can gain insights into best practices and recommendations provided by AWS for optimizing AWS Glue jobs, thereby justifying the suggested actions to address the issue of slowing job performance.
-
tgv 👍 2It looks like there are various worker types in AWS Glue actually. I'll go with AB as well. "With AWS Glue, you only pay for the time your ETL job takes to run. There are no resources to manage, no upfront costs, and you are not charged for startup or shutdown time. You are charged an hourly rate based on the number of Data Processing Units (or DPUs) used to run your ETL job. A single Data Processing Unit (DPU) is also referred to as a worker. AWS Glue comes with three worker types to help you select the configuration that meets your job latency and cost requirements. Workers come in Standard, G.1X, G.2X, and G.025X configurations." https://docs.aws.amazon.com/glue/latest/dg/components-key-concepts.html
-
DevoteamAnalytix 👍 2Here you can find 5 different Worker types: https://docs.aws.amazon.com/glue/latest/dg/add-job.html
-
MLOPS_eng 👍 1How does partitioning data in S3 improve the performance of AWS Glue jobs? Partitioning data s3 improve the query performance, but the question was the action should the DE take to improve the performance of AWS Glue jobs !
Show 2 more comments
-
Deno_Dennis 👍 1The answer is A and B. One way of optimizing Queries is by partitioning data. Increasing the AWS Glue Instance size increases the capacity of resources available for AWS Glue to process jobs.
-
Leo87656789 👍 1I would also go for A, B. But there are no worker types in AWS Glue. You can only increase the DPU.
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All DEA-C01 Questions
Access 100 questions with complete answers and detailed explanations.
View Full DEA-C01 Practice Test →