Orchestrating Long-Running Athena Queries Cost-Effectively
Answer
Correct answer: A, B — Use AWS Lambda to invoke Athena queries and AWS Step Functions to orchestrate the workflow with polling states.
A data engineer must orchestrate a series of Amazon Athena queries that will run every day. Each query can run for more than 15 minutes. Which combination of steps will meet these requirements MOST cost-effectively? (Choose two.)
Community Votes
80% of anonymous learners picked answer AB. Votes are pick records left by other test-takers — they are not the verified answer.
Community Insight
The core trap is assuming Lambda's 15-minute timeout prevents it from triggering long-running queries; in reality, Lambda only initiates the asynchronous execution, while Step Functions handles the polling and orchestration.
This question addresses the most cost-effective method to orchestrate a series of Amazon Athena queries that exceed the 15-minute limit. The correct solution leverages AWS Lambda for invocation and Step Functions for state management.
Many candidates choose E (MWAA) because it is a powerful orchestrator, but they overlook the 'MOST cost-effectively' constraint, as MWAA requires provisioning managed infrastructure which is significantly more expensive than serverless Lambda/Step Functions for this scale.
Community Discussion (28 comments)
rralucard_
👍 10
Selected: AB
AWS Lambda can be effectively used to trigger Athena queries. By using the start_query_execution API from the Athena Boto3 client, you can programmatically start Athena queries. Lambda functions are cost-effective as they charge based on the compute time used, and there's no charge when the code is not running. However, Lambda has a maximum execution timeout of 15 minutes, which means it's not suitable for long-running operations but can be used to trigger or start queries. AWS Step Functions can orchestrate multiple AWS services in workflows. By using a Wait state, the workflow can periodically check the status of the Athena query, and proceed to the next step once the query is complete. This approach is more scalable and reliable compared to continuously running a Lambda function, as Step Functions can handle long-running processes better and can maintain the state of each step in the workflow.
GiorgioGss
👍 8
Selected: BE
B - because https://docs.aws.amazon.com/step-functions/latest/dg/sample-athena-query.html E - because https://aws.amazon.com/blogs/big-data/orchestrate-amazon-emr-serverless-spark-jobs-with-amazon-mwaa-and-data-validation-using-amazon-athena/
Evan_Lin
👍 2
Selected: AB
After real-world testing, A is a valid answer. This is because the Lambda only sends the API request to Athena, which runs the query. Even if the Lambda times out, the query result is still stored in the designated S3 bucket.
Udyan
👍 2
Selected: AB
Why? B (Step Functions): Step Functions are ideal for orchestrating long-running workflows, including polling the Athena query status and invoking the next query when ready. A (Lambda): Lambda is used to programmatically trigger Athena queries within Step Functions, despite its 15-minute limitation, because Step Functions can manage the long runtime using Wait states. Why Not C, D, or E? C and D involve Glue, which is better suited for ETL jobs than orchestration, making them less efficient and cost-effective. E (Amazon MWAA) introduces unnecessary cost and complexity for a straightforward workflow.
haby
👍 4
Selected: BC
BC for me A - lambda function will stop at 900s, so it will stop before query finishes(more than 15mins) E - Airflow is way more complex and expensive than step function
altonh
👍 1
Selected: CE
AB - Because of the Lambda timeout CE—is correct. The query will be executed by a glue job, which will be orchestrated by Airflow. The job will be scheduled using AWS Batch.
Eleftheriia
👍 2
Selected: AB
Not E because "You should use Step Functions if you prioritize cost and performance" https://aws.amazon.com/managed-workflows-for-apache-airflow/faqs/ And also the fact that the queries take longer than 15 min can be handled with step functions, therefore AB
truongnguyen86
👍 1
A.Why it's correct: AWS Lambda is a cost-effective, serverless option for invoking Athena queries using the Boto3 API. Lambda charges are based on execution time and memory usage, making it an efficient solution for periodic query execution. B. Why it's correct: Step Functions provide a serverless orchestration option with a pay-per-use pricing model. Adding a Wait state prevents excessive API calls and ensures queries are executed in sequence, making it a cost-effective and scalable solution. Why the other options are less optimal: -- E. Use Amazon Managed Workflows for Apache Airflow (MWAA): MWAA is powerful for complex workflows, but its pricing includes environment uptime costs, which can be higher than Lambda and Step Functions for simple tasks like orchestrating Athena queries. By choosing A and B, you balance cost-effectiveness and simplicity for orchestrating daily Athena queries.
San_Juan
👍 2
Selected: BD
Lambda maximum timeout is 15 minutes. So the query takes more than Lambda could manage. So you cannot use lambda. Use Step-Function (answer B) or glue python (answer D) Airflow is more expensive than Glue/Step-Functions, so E is discarted also.
V0811
👍 2
Selected: AB
It should be AB
alex1991
👍 2
Selected: AB
Since the Athena API supports async/await, users are able to separate the steps into trigger queries and get results after 15 minutes.
pypelyncar
👍 1
Selected: BE
tricky, A is valid. Still, cost effective: B no one doubt on it. then why E? MWAA offers a managed Apache Airflow environment for orchestrating complex workflows. It can handle long-running tasks like Athena queries efficiently. Batch Processing: Leveraging AWS Batch within the Airflow workflow allows for distributed and scalable execution of the Athena queries, improving overall processing efficiency.
valuedate
👍 2
Selected: AB
my opinian
valuedate
👍 2
Selected: AB
I would prefer AB
VerRi
👍 3
Selected: AB
Lambda for kick start Athena Step Functions for orchestration
sdas1
👍 1
Option C and D involve using an AWS Glue Python shell script to run a sleep timer and periodically check whether the current Athena query has finished running. While this approach might seem cost-effective in terms of using AWS Glue, it's not the most efficient way to manage the execution of Athena queries. AWS Glue is primarily designed for ETL (Extract, Transform, Load) tasks rather than orchestrating long-running query execution. Therefore, while both options B, C and D could technically work, they might not be the most cost-effective or efficient solutions for orchestrating long-running Athena queries. Instead, options A and E would likely be more cost-effective and suitable for this scenario.
Christina666
👍 4
Selected: AB
Lambda call Athena query; Step function orchestrate query workflow
arvehisa
👍 4
Selected: AB
A: Lambda is a good option and it only trigger the athena not actually run it. No need 15 min for it. B. it mentioned a series of athena queries and it may means that one query should wait until the former one finished. B is the perfect way to do it. And lambda and step functions are very cost effective.
cd93
👍 2
Selected: CD
Remember, anything involves writing codes are gonna be cheaper than automated/UI guided workflows, so that left ACD, and aws Lambda can't run for more than 15 mins so CD. Guys, go take the associate architect certificate first... this is basic knowledge... stop spamming chatgpt (wrongly) generated answer
certplan
👍 1
2. AWS Glue Documentation: - AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analytics. AWS Glue offers capabilities for running Python shell jobs, which can be used to execute custom scripts for various data processing tasks. The documentation provides details on how to create and manage Python shell jobs, including examples of using scripts to interact with AWS services like Athena. - Reference: [AWS Glue Documentation]https://docs.aws.amazon.com/glue/index.html
certplan
👍 2
To justify the selection of options B and D as the most cost-effective combination for orchestrating Amazon Athena queries, let's refer to the official AWS documentation: 1. AWS Step Functions Documentation: - AWS Step Functions is a fully managed service provided by AWS for coordinating the components of distributed applications and microservices using visual workflows. With Step Functions, you can build workflows that execute a sequence of AWS Lambda functions, API calls, and other AWS services. The documentation provides information on how to create workflows, define states, and configure wait states for checking the status of tasks, which aligns with the requirements of orchestrating Amazon Athena queries. - Reference: [AWS Step Functions Documentation]https://docs.aws.amazon.com/step-functions/index.html
certplan
👍 2
Option E: Using Amazon Managed Workflows for Apache Airflow (Amazon MWAA) to orchestrate Athena queries in AWS Batch. While Amazon MWAA provides managed Apache Airflow environments, which can be used for orchestrating workflows, it might not be the most cost-effective option for orchestrating Athena queries due to: - Complexity: Setting up and managing an Amazon MWAA environment can introduce additional complexity and potentially higher costs compared to other options. - Resource Allocation: Amazon MWAA environments come with a minimum cost, regardless of usage, and managing resources in AWS Batch might not be as cost-efficient for this specific use case compared to simpler solutions like Step Functions or Glue Python shell scripts. Reference: [Amazon Managed Workflows for Apache Airflow Documentation]https://docs.aws.amazon.com/mwaa/index.html
milofficial
👍 2
Selected: AB
I changed my mind
GiorgioGss
👍 3
Guys... stop pasting from GPT... paste some official docs to prove your choice of options.
CalvinL4
👍 1
Lambda is out because it cannot run over 15 min.
BartoszGolebiowski24
👍 2
Selected: AB
We do not need to wait till Athena completes the query, so we will not reach a 15-minute timeout hard limit. So "A" is valid, we will use the step function to orchestrate the process.
TonyStark0122
👍 1
A. Use an AWS Lambda function and the Athena Boto3 client start_query_execution API call to invoke the Athena queries programmatically. B. Create an AWS Step Functions workflow and add two states. Add the first state before the Lambda function. Configure the second state as a Wait state to periodically check whether the Athena query has finished using the Athena Boto3 get_query_execution API call. Configure the workflow to invoke the next query when the current query has finished running.
milofficial
👍 3
Selected: BC
https://docs.aws.amazon.com/step-functions/latest/dg/sample-athena-query.html
Comments & Corrections
No comments yet — spotted an error or have a note? Share it below.
Expert Analysis
Why the Answer Is Correct
The correct combination is A and B. Option A uses an AWS Lambda function to callstart_query_execution. Although Lambda has a 15-minute execution timeout, the Athena query runs asynchronously in the background; Lambda simply triggers the request and can exit immediately or wait briefly, making it cost-effective for initiation. Option B uses AWS Step Functions to orchestrate the workflow. It employs a Wait state with a retry loop using the get_query_execution API to poll for completion, ensuring the next query starts only after the previous one finishes. This serverless approach minimizes costs by charging only for actual compute time used.Why the Other Options Are Wrong
Option C (Glue Python Shell) is less cost-effective than Lambda for simple triggering due to Glue's minimum billing granularity and startup overhead. Option D suggests using a sleep timer in a script, which is inefficient and prone to errors compared to the robust polling mechanisms provided by Step Functions. Option E (Amazon MWAA) is overly complex and expensive for this specific requirement; Managed Workflows for Apache Airflow is designed for large-scale, complex DAGs and incurs higher fixed costs than the pay-per-use model of Lambda and Step Functions.Community Comment Notes
Community consensus strongly supports AB. As user rralucard_ noted, "Lambda functions are cost-effective... no charge when the code is not running." User arvehisa clarified that "Lambda is a good option and it only trigger the athena not actually run it," highlighting the asynchronous nature of the service. Multiple users pointed out that Step Functions provide the necessary orchestration logic via Wait states to handle the long-running queries effectively.Official Reference
Exam Strategy
When asked for the 'most cost-effective' solution, always compare serverless options (Lambda, Step Functions) against managed services (Glue, EMR, MWAA). Remember that Lambda can trigger asynchronous jobs even if its own timeout is shorter than the job duration.
Frequently Asked Questions
Can Lambda trigger Athena queries longer than 15 minutes?
Yes. Lambda calls start_query_execution and exits; the query runs asynchronously in Athena regardless of Lambda's timeout.
Why is MWAA not the best choice here?
MWAA is more expensive and complex. For simple sequential querying, serverless Step Functions are more cost-effective.
Related Analysis
Practice All DEA-C01 Questions
Access 100 questions with complete answers and detailed explanations.
View Full DEA-C01 Practice Test →