Most Cost-Effective AWS Glue ETL for Small Daily CSV Files?

DEA-C01 By islever Editorial Last updated

Automate data processing by using AWS services. Transform and process data.

Short answer

D Use an AWS Glue Python shell job with pandas, because sub-100 MB daily CSV files are transformed most cheaply on a 1/16 DPU shell job.

Question

A data engineer needs to build an extract, transform, and load (ETL) job. The ETL job will process daily incoming .csv files that users upload to an Amazon S3 bucket. The size of each S3 object is less than 100 MB. Which solution will meet these requirements MOST cost-effectively?
  1. Write a custom Python application. Host the application on an Amazon Elastic Kubernetes Service (Amazon EKS) cluster.
  2. Write a PySpark ETL script. Host the script on an Amazon EMR cluster.
  3. Write an AWS Glue PySpark job. Use Apache Spark to transform the data.
  4. Write an AWS Glue Python shell job. Use pandas to transform the data. Correct Answer

Key Takeaways

The exam tests whether you size ETL compute to the data volume — the trap is assuming Spark/PySpark is always the 'proper' big-data answer when a fraction-of-a-DPU Python shell job is far cheaper for files under 100 MB.

Choosing the cheapest AWS service to run a daily ETL job over sub-100 MB CSV files uploaded to Amazon S3 comes down to matching compute overhead to data size. This page establishes that an AWS Glue Python shell job with pandas (D) beats EKS, EMR, and Glue PySpark on cost for this workload.

Common mistake: Picking option C (AWS Glue PySpark) because Apache Spark sounds more powerful, scalable, and 'production-grade' — but Spark's cluster provisioning and per-DPU-hour billing are unnecessary overhead when each input file is under 100 MB.

Expert Analysis

Why the Answer Is Correct

Option D matches the AWS guidance that Glue Python shell jobs are intended for small-to-medium ETL workloads and can be launched on 1/16 DPU, so the effective per-run cost is a fraction of a full Spark job. At under 100 MB per file, pandas can load, clean, and rewrite each CSV in memory without distributed processing, so no Spark cluster is needed. Because the job runs once daily on modest data, minimizing DPU footprint matters more than horizontal scalability. As atu1789 put it, Python shell jobs are "a good fit for smaller-scale ETL tasks" exactly like these daily CSV uploads. Leo87656789 adds the decisive billing detail: "you can select the option \"1/16 DPU\" in the Job details for a Python Shell Job", which is cheaper per run than any Spark configuration.

Why the Other Options Are Wrong

Option A (custom Python on Amazon EKS) forces you to build, patch, and pay for container infrastructure and worker nodes that sit idle between daily runs — the highest operational and monetary cost of all four. Option B (PySpark on Amazon EMR) means provisioning a cluster (or paying for transient cluster spin-up) for sub-100 MB files, again far more capacity than the job needs. Option C (AWS Glue PySpark) is serverless and legitimate for large datasets, but as khchan123 noted, "C may be overkill for processing small.csv files (less than 100 MB each)" because Spark's per-DPU-hour cost and startup overhead are not justified at this scale. The pricing quotes from halogi and lucas_rfsb compare raw per-DPU rates without factoring in that the shell job consumes only 1/16 of a DPU per run, so the actual invoice is lower for D.

Community Comment Notes

The vote split (D 65 vs C 35) tracks a pricing misunderstanding: halogi and lucas_rfsb cite the official Glue pricing page showing shell jobs at $0.44 per DPU-hour versus PySpark at $0.29 per DPU-hour on flexible execution, concluding Spark is cheaper. That comparison ignores the DPU count actually consumed — Leo87656789's point about the 1/16 DPU setting resolves it in favor of D. cloudata reinforces the fit argument by linking the AWS Glue best-practices whitepaper's note that Python shell jobs suit smaller datasets, and khchan123 echoes the overkill concern for C. pypelyncar fairly calls it "50/50" on approach but still leans pandas, while not one commenter defends the EKS or EMR options.

Official Reference

These links open the official vendor documentation in a new tab.

Exam Strategy

When a DEA-C01 question stresses 'MOST cost-effectively' and gives a small data size (under 100 MB, low file counts, daily batches), look for the smallest managed compute that fits — here the Glue Python shell job — and reject answers that require clusters (EKS, EMR) or distributed Spark. Always check whether a service's minimum billing unit (1/16 DPU vs 1 DPU) changes the real cost, not just the headline per-DPU rate.

Frequently Asked Questions

Why is an AWS Glue Python shell job cheaper than a Glue PySpark job here?

The shell job can run on 1/16 DPU and pandas handles sub-100 MB CSVs in memory, while a PySpark job bills full DPUs plus Spark startup overhead the small files never need.

When would option C, an AWS Glue PySpark job, actually be the better choice?

When files or volumes are large enough that data cannot fit in a single node's memory or the transform must be distributed across a Spark cluster; that scalability is wasted on under-100 MB daily CSVs.

More DEA-C01 FAQ →

Community Votes

D
65%
C
35%

65% of anonymous learners picked answer D. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 13 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 13

  1. halogi 👍 10 Answered C
    AWS Glue Python Shell Job is billed $0.44 per DPU-Hour for each job AWS Glue PySpark is billed $0.29 per DPU-Hour for each job with flexible execution and $0.44 per DPU-Hour for each job with standard execution Source: https://aws.amazon.com/glue/pricing/
  2. atu1789 👍 9 Answered D
    Option D: Write an AWS Glue Python shell job and use pandas to transform the data, is the most cost-effective solution for the described scenario. AWS Glue’s Python shell jobs are a good fit for smaller-scale ETL tasks, especially when dealing with .csv files that are less than 100 MB each. The use of pandas, a powerful and efficient data manipulation library in Python, makes it an ideal tool for processing and transforming these types of files. This approach avoids the overhead and additional costs associated with more complex solutions like Amazon EKS or EMR, which are generally more suited for larger-scale, more complex data processing tasks. Given the requirements – processing daily incoming small-sized .csv files – this solution provides the necessary functionality with minimal resources, aligning well with the goal of cost-effectiveness.
  3. pypelyncar 👍 7 Answered D
    good candidate to be (2 options) for real, either spark and py have similar approaches. I would go with Pandas, although... 50/50.. it could be Spark. I hope not to find this question in the exam
  4. cloudata 👍 6 Answered D
    Python Shell is cheaper and can handle small to medium tasks. https://docs.aws.amazon.com/whitepapers/latest/aws-glue-best-practices-build-performant-data-pipeline/additional-considerations.html
  5. rralucard_ 👍 5 Answered C
    AWS Glue is a fully managed ETL service, which means you don't need to manage infrastructure, and it automatically scales to handle your data processing needs. This reduces operational overhead and cost. PySpark, as a part of AWS Glue, is a powerful and widely-used framework for distributed data processing, and it's well-suited for handling data transformations on a large scale.
Show 8 more comments
  1. [Removed] 👍 5 Answered D
    https://medium.com/@navneetsamarth/reduce-aws-cost-using-glue-python-shell-jobs-70a955d4359f#:~:text=The%20cheapest%20Glue%20Spark%20ETL,1%2F16th%20of%20a%20DPU.&text=This%20can%20result%20in%20massive,just%20a%20better%20design%20overall!
  2. khchan123 👍 4 Answered D
    D. While AWS Glue PySpark jobs are scalable and suitable for large workloads, C may be overkill for processing small .csv files (less than 100 MB each). The overhead of using Apache Spark may not be cost-effective for this specific use case.
  3. Leo87656789 👍 4 Answered D
    Option D: Even though the Python Shell Job is more expensive on a DPU-Hour basis, you can select the option "1/16 DPU" in the Job details for a Python Shell Job, which is definetly cheaper than a Pyspark job.
  4. GiorgioGss 👍 4 Answered D
    D is more cheaper than C. Not so scalable but is cheaper...
  5. VerRi 👍 3 Answered C
    PySpark with Spark(Flexible Execution): $0.29/hr for 1 DPU PySpark with Spark(Standard Execution): $0.44/hr for 1 DPU Python Shell with Pandas: $0.44/hr for 1 DPU
  6. chakka90 👍 3
    D. Because the pyspark is still being the cheap you have to use minimum of 2 DPU. Which would increase the cost anyway so, i feel that d should be correct
  7. LR2023 👍 3 Answered D
    going with D https://docs.aws.amazon.com/whitepapers/latest/aws-glue-best-practices-build-performant-data-pipeline/additional-considerations.html
  8. YUICH 👍 2 Answered D
    It is important not to compare just the “price per DPU hour,” but to consider the total cost by factoring in overhead for job startup, minimum DPU count, execution time, and data volume. For a relatively lightweight workload—such as processing approximately 100 MB of CSV files on a daily basis—option (D), using an AWS Glue Python shell job, is the most cost-effective choice.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Submitted for moderation before publishing. Keep it helpful and respectful.

Amazon Redshift Compound Sort Key Query Optimization

S-grade

Secure Redshift Spectrum Access with VPC Endpoints

S-grade

Athena Federated Query for Multi-Source Iceberg Pipelines

S-grade

Reclaiming Storage in Amazon Redshift Materialized Views

S-grade

Practice All DEA-C01 Questions

Access 100 questions with complete answers and detailed explanations.

View Full DEA-C01 Practice Test →

← Back to DEA-C01 Study Guide