AWS Glue Data Catalog for IoT JSON Schema Evolution

DEA-C01 By islever Editorial Last updated

Understand data cataloging systems.

Short answer

A Create an AWS Glue Data Catalog and configure Schema Registry to handle schema evolution, then orchestrate ingestion into Redshift Serverless.

Question

A security company stores IoT data that is in JSON format in an Amazon S3 bucket. The data structure can change when the company upgrades the IoT devices. The company wants to create a data catalog that includes the IoT data. The company's analytics department will use the data catalog to index the data. Which solution will meet these requirements MOST cost-effectively?
  1. Create an AWS Glue Data Catalog. Configure an AWS Glue Schema Registry. Create a new AWS Glue workload to orchestrate the ingestion of the data that the analytics department will use into Amazon Redshift Serverless. Correct Answer
  2. Create an Amazon Redshift provisioned cluster. Create an Amazon Redshift Spectrum database for the analytics department to explore the data that is in Amazon S3. Create Redshift stored procedures to load the data into Amazon Redshift.
  3. Create an Amazon Athena workgroup. Explore the data that is in Amazon S3 by using Apache Spark through Athena. Provide the Athena workgroup schema and tables to the analytics department.
  4. Create an AWS Glue Data Catalog. Configure an AWS Glue Schema Registry. Create AWS Lambda user defined functions (UDFs) by using the Amazon Redshift Data API. Create an AWS Step Functions job to orchestrate the ingestion of the data that the analytics department will use into Amazon Redshift Serverless.

Key Takeaways

The core trap is confusing ad-hoc querying (Athena) with persistent cataloging and ingestion orchestration; Athena queries S3 directly but does not create or maintain a managed metadata catalog in the same way Glue does.

This question tests the creation of a centralized data catalog for evolving semi-structured data using AWS Glue. The correct solution leverages the Glue Data Catalog and Schema Registry to manage schema evolution while orchestrating ingestion into Redshift Serverless.

Common mistake: Many candidates choose Option C (Amazon Athena) because it is cost-effective for querying JSON in S3, but they overlook the requirement to 'create a data catalog' and the need for structured ingestion into a warehouse like Redshift.

Expert Analysis

Why the Answer Is Correct

Option A is the correct answer because it directly addresses all requirements: creating a data catalog, handling schema evolution, and providing a cost-effective analytics destination. AWS Glue Data Catalog is the native service for centralizing metadata across AWS data lakes. Configuring AWS Glue Schema Registry ensures that changes in the IoT JSON structure are tracked and validated automatically. Using an AWS Glue workload (such as Glue Studio jobs) to orchestrate ingestion into Amazon Redshift Serverless provides a serverless, pay-per-use compute model that avoids the fixed costs of provisioned clusters, making it the most cost-effective choice for ongoing analytics.

Why the Other Options Are Wrong

Option B involves provisioning an Amazon Redshift cluster, which incurs significant infrastructure costs regardless of usage, violating the 'most cost-effectively' constraint. It also lacks a robust mechanism for schema evolution compared to Glue Schema Registry. Option C uses Amazon Athena, which allows SQL queries on S3 data without loading it. While Athena handles JSON well, it does not 'create a data catalog' in the sense of a managed metadata repository for ingestion pipelines; it primarily uses its own internal table definitions. Furthermore, Athena is for querying, not for the ingestion orchestration implied by the analytics department's workflow. Option D introduces unnecessary complexity and cost by using Lambda UDFs and Step Functions for simple ingestion when Glue Workloads are purpose-built for this task.

Community Comment Notes

The community split between A and C, with many arguing for C due to its low upfront cost. However, users like chris_spencer correctly identified that "AWS Glue Data Catalog is the best option" for cataloging requirements. Others like VerRi noted that "Athena is not able to create new data catalog," highlighting the distinction between query engines and metadata management services. User rralucard_ emphasized that Glue offers a "serverless approach to manage the evolving data schema," which aligns perfectly with the IoT device upgrade scenario.

Official Reference

https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html

This link opens the official vendor documentation in a new tab.

Exam Strategy

Always distinguish between 'querying' data (Athena) and 'cataloging/managing' data (Glue). When the question asks for a data catalog and mentions schema evolution, Glue is almost always the primary component. Look for 'cost-effective' combined with 'analytics' to guide you toward serverless destinations like Redshift Serverless rather than provisioned clusters.

Frequently Asked Questions

Why isn't Amazon Athena sufficient for creating a data catalog?

Athena is a query engine, not a metadata catalog service. It uses tables defined in its own registry or external catalogs, but it doesn't actively crawl and manage schema evolution for ingestion pipelines like Glue does.

Is Redshift Serverless more cost-effective than Provisioned Redshift?

Yes, for variable workloads. Redshift Serverless scales compute automatically and you only pay for what you use, avoiding the fixed hourly costs of maintaining a provisioned cluster that might be underutilized.

More DEA-C01 FAQ →

Community Votes

A
80%
C
20%

80% of anonymous learners picked answer A. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 7 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 7

  1. rralucard_ 👍 9 Answered A
    Option A, creating an AWS Glue Data Catalog with Glue Schema Registry and orchestrating data ingestion into Amazon Redshift Serverless using AWS Glue, appears to be the most cost-effective and suitable solution. It offers a serverless approach to manage the evolving data schema of the IoT data and efficiently supports data analytics needs without the overhead of managing a provisioned database cluster or complex orchestration setups.
  2. chris_spencer 👍 4 Answered A
    The objective is to create a data catalog that includes the IoT data and AWS Glue Data Catalog is the best option for this requirement. https://docs.aws.amazon.com/glue/latest/dg/catalog-and-crawler.html C is incorrect. While Athena makes it easy to read from S3 using SQL, it does not crawl the data source and create a data catalog.
  3. Christina666 👍 2 Answered C
    Why Option C is the Most Cost-Effective Serverless and Pay-as-you-go: Athena is a serverless query service, meaning you only pay for the queries the analytics department runs. No need to provision and manage always-running clusters. Flexible Schema Handling: Athena works well with semi-structured data like JSON and can handle schema evolution on the fly. This is perfect for the scenario where IoT data structures might change. Spark Integration: Integrating Apache Spark with Athena provides rich capabilities for data processing and transformation. Ease of Use for Analytics: Athena's familiar SQL-like interface and ability to directly query S3 data make it convenient for the analytics department.
  4. khchan123 👍 2 Answered A
    The correct solution is A. Create an AWS Glue Data Catalog. Configure an AWS Glue Schema Registry. Create a new AWS Glue workload to orchestrate the ingestion of the data that the analytics department will use into Amazon Redshift Serverless. Option C (Amazon Athena and Apache Spark) is suitable for ad-hoc querying and exploration but may not be the best choice for the analytics department's ongoing data analysis needs, as Athena is designed for interactive querying rather than complex data transformations.
  5. lucas_rfsb 👍 2 Answered C
    Options A, B, and D involve setting up additional infrastructure (e.g., AWS Glue, Redshift clusters, Lambda functions) which may incur unnecessary costs and complexity for the given requirements. Option C, on the other hand, utilizes a serverless and scalable solution directly querying data in S3, making it the most cost-effective choice.
Show 2 more comments
  1. sdas1 👍 1
    Option C Cost-effectiveness: Amazon Athena allows you to query data directly from Amazon S3 without the need for any infrastructure setup or management. You pay only for the queries you run, making it cost-effective, especially for sporadic or exploratory analysis. Flexibility: Since the data structure can change with IoT device upgrades, using Athena allows for flexibility in querying and analyzing the data regardless of its structure. You don't need to define a fixed schema upfront, enabling you to adapt to changes seamlessly. Apache Spark Support: Athena supports querying data using Apache Spark, which is powerful for processing and analyzing large datasets. This capability ensures that the analytics department can leverage Spark for more advanced analytics if needed. https://www.youtube.com/watch?v=Q93NZJBFSWw
  2. VerRi 👍 1 Answered A
    Athena is not able to create new data catalog

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Amazon Redshift Compound Sort Key Query Optimization

S-grade

Secure Redshift Spectrum Access with VPC Endpoints

S-grade

Athena Federated Query for Multi-Source Iceberg Pipelines

S-grade

Reclaiming Storage in Amazon Redshift Materialized Views

S-grade

Practice All DEA-C01 Questions

Access 100 questions with complete answers and detailed explanations.

View Full DEA-C01 Practice Test →

← Back to DEA-C01 Study Guide