Cost-Effective CDC in S3 Data Lake

Why the Answer Is Correct

Option C is the correct answer because open-source data lake formats (such as Apache Iceberg, Delta Lake, or Hudi) are specifically designed to handle ACID transactions, schema evolution, and upserts on data stored in S3. These formats allow you to efficiently merge incoming changes with existing data without needing to move data into a relational database first. This approach is highly scalable and cost-effective for handling both small and tens-of-terabyte files, as it leverages S3's storage efficiency and avoids the overhead of managing additional infrastructure.

Why the Other Options Are Wrong

Option A (Lambda) is not cost-effective for processing tens of terabytes of data; Lambda has timeout limits and high costs for long-running, heavy data transformations. Option B (RDS) and Option D (Aurora Serverless) involve moving data out of S3 into a managed SQL database, which adds significant unnecessary complexity and cost (storage, compute, and DMS licensing/usage) when the goal is to update a data lake, not a transactional database. DMS is typically used for migrating to a database or from a database, not for updating an S3-based data lake using CDC in this context.

Community Comment Notes

The community overwhelmingly supports Option C, recognizing that while AWS services are powerful, open-source table formats are the industry standard for modern data lakes. As noted by user GiorgioGss, implementing CDC-based upserts using formats like Apache Iceberg with AWS Glue is a well-documented best practice. Another user highlighted that while Lambda seems like a simple AWS service choice, the engineering effort and cost for large datasets make open-source APIs far superior. Some users initially considered AWS-centric options but were convinced by the specific mention of "semi-structured" and "transactional" requirements that favor table formats.