Optimizing AWS Glue for Millions of Small Files

DEA-C01 By islever Editorial Last updated Oct 12, 2026

Automate data processing by using AWS services.

Short answer

B Use the AWS Glue dynamic frame file-grouping option to ingest raw input files, process them, and load into Redshift tables.

Question

A company receives test results from testing facilities that are located around the world. The company stores the test results in millions of 1 KB JSON files in an Amazon S3 bucket. A data engineer needs to process the files, convert them into Apache Parquet format, and load them into Amazon Redshift tables. The data engineer uses AWS Glue to process the files, AWS Step Functions to orchestrate the processes, and Amazon EventBridge to schedule jobs. The company recently added more testing facilities. The time required to process files is increasing. The data engineer must reduce the data processing time. Which solution will MOST reduce the data processing time?

Use AWS Lambda to group the raw input files into larger files. Write the larger files back to Amazon S3. Use AWS Glue to process the files. Load the files into the Amazon Redshift tables.
Use the AWS Glue dynamic frame file-grouping option to ingest the raw input files. Process the files. Load the files into the Amazon Redshift tables. Correct Answer
Use the Amazon Redshift COPY command to move the raw input files from Amazon S3 directly into the Amazon Redshift tables. Process the files in Amazon Redshift.
Use Amazon EMR instead of AWS Glue to group the raw input files. Process the files in Amazon EMR. Load the files into the Amazon Redshift tables.

Key Takeaways

The core challenge is I/O and task scheduling overhead for small files; the key insight is that AWS Glue's built-in dynamic frame file-grouping feature automatically consolidates these files into optimal sizes for parallel processing.

This question addresses the performance bottleneck caused by processing millions of small JSON files in AWS Glue, where file overhead dominates execution time. It establishes that leveraging native file-grouping capabilities is the most efficient way to reduce processing latency without changing the orchestration architecture.

Common mistake: Many candidates choose Option C (Redshift COPY) because it bypasses Glue entirely, assuming fewer steps mean faster processing. However, this ignores the requirement to use AWS Glue for transformation and fails to address the specific inefficiency of the current Glue setup, often resulting in slower overall pipeline completion due to Redshift's inability to efficiently parse millions of tiny files directly.

Expert Analysis

Why the Answer Is Correct

Option B is the correct solution because AWS Glue's dynamic frame file-grouping option specifically addresses the "small file problem." When ingesting millions of 1 KB files, the overhead of opening, reading, and closing each file, along with the creation of individual Spark tasks per file, drastically increases processing time. By enabling file grouping, Glue combines these small files into larger, more manageable blocks before processing, significantly reducing I/O operations and improving parallelism within the existing Glue job framework.

Why the Other Options Are Wrong

Option A suggests using AWS Lambda to pre-group files. This adds an extra step to the pipeline, increasing complexity and latency, rather than solving the issue at the processing layer. Option C proposes using the Redshift COPY command directly. While Redshift can ingest data, it is not optimized for parsing millions of tiny JSON files efficiently compared to a dedicated ETL tool like Glue, and it contradicts the goal of optimizing the current Glue-based workflow. Option D suggests switching to Amazon EMR, which involves significant architectural change and migration effort, failing to meet the requirement for a quick optimization of the existing setup.

Community Comment Notes

The community consensus strongly favors Option B, with comments highlighting the need to address inefficiencies from large numbers of small files. Users noted that the solution must leverage existing tools like AWS Glue to optimize ingestion rather than adding new services or bypassing the ETL layer entirely.

Official Reference

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-loading.html#loading-dynamicframe-filegrouping

This link opens the official vendor documentation in a new tab.

Exam Strategy

When dealing with performance issues in AWS Glue involving many small files, always look for configuration options related to 'file grouping' or 'splitting' within the Glue DynamicFrame API before considering architectural changes like moving to EMR or Lambda.

Frequently Asked Questions

Why not use Redshift COPY for small files?

Redshift COPY is optimized for large files. Millions of 1 KB files cause high overhead and poor performance during parsing and loading.

Does file grouping increase storage costs?

No, file grouping is a logical operation during ingestion. It does not create additional physical files in S3, so storage costs remain unchanged.

More DEA-C01 FAQ →

Community Votes

100%

100% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 3 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 3

minhhnh 👍 1 Answered B

The key requirement is to reduce processing time for millions of small JSON files stored in Amazon S3. The solution needs to address the inefficiencies caused by the large number of small files while leveraging the existing AWS Glue and Amazon Redshift setup.
matt200 👍 1 Answered B

Option B: Use the AWS Glue dynamic frame file-grouping option to ingest the raw input files. Process the files. Load the files into the Amazon Redshift tables.
aragon_saa 👍 1 Answered B

Answer is B

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Practice All DEA-C01 Questions

Access 100 questions with complete answers and detailed explanations.

View Full DEA-C01 Practice Test →

← Back to DEA-C01 Study Guide