Apache Iceberg Automatic Compaction for Small Files

DEA-C01 By islever Editorial Last updated

Transform and process data. Automate data processing by using AWS services.

Short answer

C Configure Iceberg table properties to enable automatic compaction based on thresholds for file size and the number of files.

Question

A company stores Apache Parquet files in an Amazon S3 data lake. The data lake receives thousands of files from multiple sources every hour. The files range in size from 50 KB to 100 KB.
The company is evaluating the implementation of Apache Iceberg tables for the data lake. The company is using AWS Glue Data Catalog as part of the evaluation. The company needs a solution to optimize query performance in Iceberg. The solution must ensure that Iceberg table performance does not degrade when more files are added over time.
Which solution will meet these requirements?
  1. Use an AWS Glue job to compact the files into a standard size of 512 MB at the end of each day. Run an AWS Glue crawler to update the Data Catalog.
  2. Configure the Data Catalog to automatically compact the files every minute.
  3. Configure Iceberg table properties to enable automatic compaction based on thresholds for file size and the number of files. Correct Answer
  4. Implement a partition strategy in Amazon S3. Run an AWS Glue crawler to update the Data Catalog every 5 minutes.

Key Takeaways

This question tests mitigating the small-files problem in Apache Iceberg, and the main trap is assuming the AWS Glue Data Catalog can execute compaction on a scheduled basis rather than recognizing Iceberg's threshold-based property configuration.

Apache Iceberg handles the small-files problem in Amazon S3 by using table properties to enable automatic compaction based on file size and count thresholds.

Common mistake: Option B is wrong because the AWS Glue Data Catalog is a metadata repository and cannot be configured to execute compaction compute tasks on a schedule.

Expert Analysis

The Small-Files Problem: Ingesting thousands of tiny Parquet files (50–100 KB) every hour creates a severe small-files problem, causing query engines to suffer high metadata overhead and slow read times. Apache Iceberg mitigates this via compaction, which merges small files into larger ones.

Why Option B is a Trap: Option B sounds appealing because AWS Glue offers automated compaction for Iceberg, and a one-minute schedule seems aggressive enough to keep up with continuous ingestions. However, the AWS Glue Data Catalog is strictly a metadata store; it cannot execute compute workloads or be configured to run compaction on a per-minute schedule. Even if scheduling a Glue job, a one-minute interval is impractical and not a native Data Catalog capability.

The Correct Mechanism: Option C correctly leverages Iceberg's native table properties (e.g., setting write.compaction.enable to true and defining thresholds for file size and count). This allows the query engine or writer to automatically trigger compaction actions when the table's metadata indicates the thresholds have been breached. This threshold-based approach dynamically adapts to the data ingestion rate, ensuring performance does not degrade as file counts grow, without relying on arbitrary, unsupported cron schedules.

Why Other Options Fail: Option A uses a daily Glue job and crawler; a daily cadence allows small files to accumulate and degrade performance throughout the day, and crawlers do not update Iceberg metadata correctly. Option D suggests partitioning and a crawler; while partitioning prunes data, it does nothing to merge small files within those partitions.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When evaluating Apache Iceberg features on AWS, remember that the Glue Data Catalog is strictly a metastore and cannot execute compute tasks like compaction. To solve the small-files problem, rely on Iceberg's native table properties to trigger compaction based on file size and count thresholds. This threshold-based approach dynamically adapts to ingestion rates, preventing performance degradation without relying on arbitrary cron schedules or manual crawler updates.

Community Votes

C
75%
B
25%

75% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 4 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 4

  1. Tm75 👍 1 Answered C
    Why C is the right solution You’re dealing with a classic small‑files problem: Thousands of very small Parquet files (50–100 KB) Continuous ingestion (every hour) Query engines that must scan metadata and files efficiently Need to ensure query performance does not degrade over time as files accumulate When evaluating Apache Iceberg, the best practice is to let Iceberg manage file layout over time, not external jobs or crawlers.
  2. 8948d06 👍 1 Answered B
    Answer is B : https://docs.aws.amazon.com/glue/latest/dg/compaction-management.html
  3. AWS82siast 👍 1 Answered C
    Cannot specify schedule for Data Catalog driven Compaction
  4. learner2023 👍 1 Answered C
    C is correct

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Amazon Redshift Compound Sort Key Query Optimization

S-grade

Secure Redshift Spectrum Access with VPC Endpoints

S-grade

Least Operational Overhead for Kinesis to Redshift Ingestion

S-grade

Athena Federated Query for Multi-Source Iceberg Pipelines

S-grade

Practice All DEA-C01 Questions

Access 100 questions with complete answers and detailed explanations.

View Full DEA-C01 Practice Test →

← Back to DEA-C01 Study Guide