Preprocessing BigQuery Data for TensorFlow Fraud Detection Models

PMLE By islever Editorial Last updated

Short answer

C Perform preprocessing in BigQuery using SQL and read data directly into TensorFlow with BigQueryClient for cost-effective, efficient training tied to BigQuery batch inference.

Question

You are using Keras and TensorFlow to develop a fraud detection model. Records of customer transactions are stored in a large table in BigQuery. You need to preprocess these records in a cost-effective and efficient way before you use them to train the model. The trained model will be used to perform batch inference in BigQuery. How should you implement the preprocessing workflow?
  1. Implement a preprocessing pipeline by using Apache Spark, and run the pipeline on Dataproc. Save the preprocessed data as CSV files in a Cloud Storage bucket.
  2. Load the data into a pandas DataFrame. Implement the preprocessing steps using pandas transformations, and train the model directly on the DataFrame.
  3. Perform preprocessing in BigQuery by using SQL. Use the BigQueryClient in TensorFlow to read the data directly from BigQuery. Correct Answer
  4. Implement a preprocessing pipeline by using Apache Beam, and run the pipeline on Dataflow. Save the preprocessed data as CSV files in a Cloud Storage bucket.

Key Takeaways

This question tests whether you keep preprocessing in the same serverless warehouse where the data already resides and where inference will run, rather than introducing Spark, Dataflow, or pandas.

For large transaction data already in BigQuery that will be used for batch inference in BigQuery, the most cost-effective and efficient preprocessing approach is to transform the data with SQL in BigQuery and read it directly into TensorFlow using BigQueryClient (C).

Common mistake: Choosing D (Apache Beam on Dataflow) is tempting because it is scalable, but it adds pipeline complexity, runner cost, and CSV materialization in Cloud Storage—unnecessary when the data is already in BigQuery.

Expert Analysis

Data locality drives the answer

The transactions already live in a large BigQuery table, and the trained model will later run batch inference in BigQuery. Keeping the preprocessing inside BigQuery avoids exporting data, standing up clusters, or materializing intermediate files. Option C performs preprocessing with SQL directly in BigQuery and then streams the result into TensorFlow via the BigQueryClient, so the data never leaves the warehouse until training reads it.

Why the alternatives fail the constraints

  • A (Spark on Dataproc) and D (Beam on Dataflow) both require provisioning a separate processing engine and writing the preprocessed output as CSV files in Cloud Storage. That adds cluster or runner cost, operational complexity, and extra storage—contradicting "cost-effective and efficient." One community comment notes that while these are scalable, they "introduce extra complexity for managing" pipelines and data movement.
  • B (pandas DataFrame) forces the large table into a single machine's memory, which will cause resource constraints and performance issues on a large dataset. It also breaks the BigQuery-centric inference path.

The request path for option C

  1. Write SQL transforms (filtering, joins, feature engineering) as a BigQuery query or view.
  2. In TensorFlow, use the BigQueryClient (from tensorflow.python.framework.bigquery) to read rows directly from that query/table in streaming batches.
  3. Train the Keras model on the streamed tensors.
  4. Export the trained model and register it for BigQuery batch inference (e.g., via BigQuery ML or a remote model), so inference reuses the same SQL feature logic.

This path minimizes cost (serverless, per-byte pricing), maximizes efficiency (BigQuery's distributed execution), and keeps preprocessing and inference in one consistent SQL environment. The community comments agree, with one summarizing: "Easiest to preprocess the data on BigQuery," and another explaining that the whole workflow should stay in BigQuery because it is "serverless, fast on large tables and cost-effective."

Official Reference

Exam Strategy

When a question states data is already in BigQuery and inference will also happen in BigQuery, look for the option that keeps the entire workflow inside BigQuery. Eliminate any choice that exports data to Cloud Storage or spins up a separate cluster unless the scenario explicitly demands distributed processing outside the warehouse. Also rule out pandas for large tables because of memory limits.

Community Votes

C
88%
D
12%

88% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 4 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 4

  1. pikachu007 👍 5 Answered C
    A. Spark on Dataproc: While powerful, it incurs additional cluster setup and management costs, potentially less cost-effective for this specific use case. B. pandas DataFrame: Loading large datasets into memory might lead to resource constraints and performance issues, especially for large-scale preprocessing. D. Apache Beam on Dataflow: While scalable, it introduces extra complexity for managing a separate pipeline and storage for preprocessed data.
  2. pinimichele01 👍 2
    Easiest to preprocess the data on BigQuery.
  3. pinimichele01 👍 2 Answered C
    went with C
  4. taylz876 👍 1 Answered C
    Why C is right The transactions are already in a large BigQuery table, and the trained model will run batch inference in BigQuery. So the whole workflow should stay in BigQuery: • Preprocess with SQL in BigQuery: it's serverless, fast on large tables and cost-effective. There is no cluster, no pipeline and no data movement. • Read the data straight into training: TensorFlow's BigQuery reader streams the rows directly into the training pipeline, so you don't need intermediate files. • Consistency for batch inference: since inference also happens in BigQuery, the same SQL can preprocess the data for scoring. Why the others are wrong • A: Spark on Dataproc (now Managed Service for Apache Spark) means running a cluster and exporting data out of BigQuery to CSV files. It's slower and costs more. • B: Loading a large table into a pandas DataFrame needs a lot of memory, and it won't scale to a table this size. • D: Beam on Dataflow works, but it's a heavier pipeline than a SQL statement. It also saves to CSV files, which adds an extra copy of the data and another step.

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide