Pass records to AWS Step Functions and decouple the sequential steps into tasks and Lambda functions

Why the Answer Is Correct

The problem is that a failure in any step forces the entire record to be reprocessed from the beginning. The fix is to move the orchestration out of the application and into a managed workflow service that maintains per-record state. A web application passes records to AWS Step Functions, and the processing is decoupled into Step Functions tasks implemented by AWS Lambda functions. Step Functions holds the state of the execution between tasks, so when a task fails its built-in retry and error handling apply to that task alone rather than to the whole record, which is exactly the required behaviour. The fact that each step completes within five minutes also fits Step Functions well, since Lambda-backed tasks are suited to short, compute-intensive units of work (D). D is the correct answer.

Why the Other Options Are Wrong

A creates a web application that writes records to Amazon S3, uses S3 Event Notifications to publish to an SNS topic, and uses an EC2 instance to poll SNS and start processing while saving intermediate results to S3 to pass to the next step. Persisting intermediate output in S3 does not provide workflow state or per-step retry; the logic that resumes processing at the correct step and that retries only the failed step would still have to be written, and the current failure behaviour would remain. B converts the application to a container on AWS Fargate and configures the container to invoke itself to pass state from one step to the next. Self-invocation carries no durable workflow state, so after a failure there is no execution state to resume from and the record must be processed again from the start, which is precisely the behaviour the requirement asks to eliminate. C passes records to an Amazon Kinesis data stream and decouples the processing into the stream and Lambda functions. A stream with consumers does not model a multistep sequential workflow, so it provides neither ordered step progression nor per-step resume on failure. D is correct.

Community Comment Notes

Community voted D unanimously. trungtd identified the three properties that make Step Functions plus Lambda the fit: decoupling tasks, error handling and retry logic, and state management, which between them remove the restart-from-beginning behaviour. jamesf cited the Step Functions documentation as the supporting reference, and tgv confirmed D. No alternative received support.