Real-Time Feature Engineering with Dataflow and Feature Store

PMLE By islever Editorial Last updated

Serving models. Exploring and preprocessing data for ML.

Short answer

B Create streaming Dataflow pipelines that read transaction data from Pub/Sub. Use a windowing function to aggregate transaction data over time. Store the engineered features in Agent Platform Feature Store.

Question

You work in the credit card fraud detection department of a bank that processes millions of transactions per day. You need to create and serve a real-time fraud detection model. To address the challenge of feature engineering at this scale with limited raw data, you plan to create time-related aggregate features, such as the average transaction amount within the last month or the amount of time since the last transaction. How should you process the data for model training and serving?
  1. Ingest the transaction data into BigQuery. Use the TRANSFORM clause in BigQuery ML to create new features by applying SQL functions to the raw transaction data.
  2. Create streaming Dataflow pipelines that read transaction data from Pub/Sub. Use a windowing function to aggregate transaction data over time. Store the engineered features in Agent Platform Feature Store. Correct Answer
  3. Ingest the transaction data into BigQuery. Use a TRANSFORM_ONLY model in BigQuery ML to create a reusable feature engineering pipeline. Use this pipeline to preprocess the data.
  4. Orchestrate a workflow in Agent Platform Pipelines that exports raw transaction data from BigQuery to Cloud Storage. Perform feature engineering using a Python script in a custom component in the pipeline.

Key Takeaways

The question tests the distinction between batch SQL-based feature engineering and streaming architectures; only Dataflow + Feature Store supports continuous windowing and real-time serving at scale.

Learn how to process real-time transaction data for fraud detection using Dataflow for streaming aggregation and Agent Platform Feature Store for low-latency serving.

Common mistake: Candidates often choose BigQuery ML options (A or C) because they are familiar with SQL-based transformation, but fail to recognize that BigQuery lacks the native low-latency online serving capabilities required for real-time fraud detection APIs.

Expert Analysis

The scenario requires real-time fraud detection for millions of daily transactions, specifically addressing the need for time-windowed aggregate features (e.g., average amount in the last month). This implies a streaming architecture where features must be computed continuously and served with low latency.

Option B is the correct approach. It utilizes Dataflow with Pub/Sub to ingest and process the transaction stream in real-time. By applying windowing functions, it computes the required aggregates continuously. Crucially, it stores these features in the Agent Platform Feature Store, which is designed for low-latency feature serving during online inference, ensuring consistency between training data preparation and real-time prediction.

Option A is incorrect because BigQuery is primarily a batch-oriented warehouse. While it can perform transformations via the TRANSFORM clause, it does not natively support real-time streaming ingestion and low-latency online feature serving required for immediate fraud decisions.

Option C is incorrect for similar reasons; the TRANSFORM_ONLY model in BigQuery ML is a batch processing tool. It cannot provide the sub-second response times needed for real-time API calls.

Option D is less optimal because exporting raw data from BigQuery to Cloud Storage and using custom Python scripts adds significant orchestration overhead and potential latency bottlenecks compared to the managed, scalable streaming pipeline offered by Dataflow combined with the specialized Feature Store service.

Exam Strategy

When you see keywords like 'real-time', 'low latency', and 'continuous aggregation' alongside massive data volumes, immediately rule out batch-only tools like standard BigQuery queries. Look for solutions that integrate a streaming compute engine (like Dataflow) with a dedicated feature serving store (like Feature Store) to ensure the same logic applies to both training and live inference.

Frequently Asked Questions

Why not use BigQuery for real-time fraud detection?

BigQuery is optimized for batch analytics and large-scale historical queries. It does not natively support the sub-millisecond latency required for real-time API-driven fraud decisions without complex external integration.

What is the role of the Agent Platform Feature Store?

It provides a centralized repository for storing and serving features with low latency. It ensures that the features used during model training are identical to those served during real-time prediction, reducing training-serving skew.

More PMLE FAQ →

Community Votes

B
100%

100% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 2 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 2

  1. taylz876 👍 1 Answered B
    Why B is right • Real-time features over time windows: "average transaction amount in the last month" or "time since the last transaction" are aggregates over a time window. A streaming Dataflow pipeline with windowing computes them continuously from the transaction stream at very large scale. • Same features for training and serving: storing them in Feature Store gives low-latency lookups for real-time predictions and one definition for training, which avoids training-serving skew. Why the others are wrong • A and C: BigQuery TRANSFORM or TRANSFORM_ONLY works for batch preprocessing, but it doesn't compute fresh time-window aggregates in real time for serving. • D: Exporting to Cloud Storage and running a Python script is a batch workflow, too slow for real-time fraud detection.
  2. n2183712847 👍 1 Answered B
    Correct answer: B. Streaming Dataflow can compute time-windowed aggregate features from the continuous transaction stream, and Agent Platform Feature Store can serve those features with low latency for both training and real-time prediction; A and C focus on batch transformations in BigQuery ML and do not provide an online feature-serving solution, while D adds unnecessary data movement and pipeline complexity for a real-time use case.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

Building CLV Models with BigQuery ML in Vertex AI Workbench

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide