Scalable Real-Time Anomaly Detection on Market Data Streams with Kinesis and Managed Flink Random Cut Forest

Why the Answer Is Correct

The workload is a high-volume real-time stream where thousands of records arrive every second and must be scanned for anomalies, and the requirement is the least operational overhead. Kinesis Data Streams is a managed service for real-time ingestion at that scale, and Amazon Managed Service for Apache Flink is a managed, automatically scaling stream processor, so neither requires the company to run clusters. Crucially, Flink provides the RANDOM_CUT_FOREST function as a built-in for streaming anomaly detection, which detects anomalies directly in the stream without provisioning a separate model endpoint. The vote was 86 for A, the only option that scores meaningfully. Saransundar mapped each part of the requirement to its service, and GiorgioGss cited the Kinesis Data Analytics SQL reference for RANDOM_CUT_FOREST, quoting that it detects anomalies in your data stream.

Why the Other Options Are Wrong

Deploying a SageMaker endpoint for real-time outlier detection with a Lambda function invoked by the data streams (B) works functionally but adds a model to train, host, and monitor plus a function to maintain, when the managed Flink path already includes the detection capability, so it carries more operational overhead. Running Apache Kafka on Amazon EC2 instances with the same SageMaker and Lambda detection (C) compounds that overhead by adding a self-managed Kafka cluster that the company must provision, patch, and operate, which directly contradicts the least-overhead requirement. Sending data to an SQS FIFO queue and having Lambda start a Glue ETL job for batch processing and anomaly detection (D) converts a real-time requirement into a batch one, and a FIFO queue adds ordering constraints and polling latency that make it unsuited to a high-throughput stream.

Community Comment Notes

The community was heavily in favor at 86 votes for A, and the one substantive dissent came from dduenas, who called it a tricky question and argued that RANDOM_CUT_FOREST is a legacy function of Kinesis Data Analytics SQL rather than a Flink function, noting that Flink has a RandomCutForestOperator that is a different thing, and linked the AWS blog on real-time anomaly detection via Random Cut Forest in Managed Flink. The majority position, held by Saransundar and GiorgioGss, treats the RANDOM_CUT_FOREST capability as built in, which is the reading the question intends. The dissent is worth noting because it questions the exact naming of the function rather than the architecture.