How to automate weekly batch predictions to BigQuery?

You have recently used TensorFlow to train a classification model on tabular data. You have created a Dataflow pipeline that can transform several terabytes of data into training or prediction datasets consisting of TFRecords. You now need to productionize the model, and you want the predictions to be automatically uploaded to a BigQuery table on a weekly schedule. What should you do?

Import the model into Vertex AI and deploy it to a Vertex AI endpoint. On Vertex AI Pipelines, create a pipeline that uses the DataflowPythonJobOp and the ModelBacthPredictOp components.
Import the model into Vertex AI and deploy it to a Vertex AI endpoint. Create a Dataflow pipeline that reuses the data processing logic sends requests to the endpoint, and then uploads predictions to a BigQuery table.
Import the model into Vertex AI. On Vertex AI Pipelines, create a pipeline that uses the Source Reference Answer
Import the model into BigQuery. Implement the data processing logic in a SQL query. On Vertex AI Pipelines create a pipeline that uses the BigquervQueryJobOp and the BigqueryPredictModelJobOp components.

Community Votes

64%

36%

64% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Community Insight

The exam tests the distinction between online serving and batch processing, where the common trap is unnecessarily deploying a Vertex AI endpoint for a scheduled batch prediction job.

To automate weekly batch predictions for a TensorFlow model, import the model into Vertex AI and orchestrate the workflow using Vertex AI Pipelines with DataflowPythonJobOp and ModelBatchPredictOp. The community consensus confirms that deploying a serving endpoint is unnecessary for batch workloads and that the pipeline components can handle data processing and BigQuery uploads efficiently.

Choosing Option B is a common mistake because users often believe they must deploy an endpoint and manually invoke a Dataflow pipeline to reuse logic, overlooking that DataflowPythonJobOp allows reusing that logic within a managed pipeline without an endpoint.

Community Discussion (16 comments)

BlehMaks 👍 10 Selected: C

The DataflowPythonJobOp operator lets you create a Vertex AI Pipelines component that prepares data by submitting a Python-based Apache Beam job to Dataflow for execution. https://cloud.google.com/vertex-ai/docs/pipelines/dataflow-component#dataflowpythonjobop Using we can specify an output location for Vertex AI to store predictions results https://cloud.google.com/vertex-ai/docs/pipelines/batchprediction-component A - is incorrect since we dont need an endpoint for batch predictions B - creating a new Dataflow pipeline is redundant

lunalongo 👍 1 Selected: C

C is the best option because it uses: 1) Vertex AI Pipelines for orchestrating the flow (managed and scalable). 2) DataflowPythonJobOp for prep and ModelBatchPredictOp for batch predictions on Vertex AI. A deploys the model to a Vertex AI endpoint, inefficient for batch jobs! B uses a single Dataflow pipeline, which needs custom Vertex AI and BQ integration. *D uses BigQuery, a datawarehouse, for model deployment and prediction.

AK2020 👍 1 Selected: B

Uploading predictions directly to BigQuery from the Dataflow pipeline integrates seamlessly with your data storage.

AzureDP900 👍 1

B is right because 1)You've already trained a classification model using TensorFlow, so you need to productionize it by deploying it to a Vertex AI endpoint. 2)To automate the prediction process on a weekly schedule, you can create a Dataflow pipeline that reuses your existing data processing logic. This pipeline will send requests to the deployed model for inference and then upload the predicted results to BigQuery.

Prakzz 👍 1 Selected: B

Only option B talks about loading the data to BigQuery

rcapj 👍 2

B Vertex AI Deployment: Vertex AI provides a managed environment for deploying machine learning models. It simplifies the process and ensures scalability. Dataflow Pipeline Reuse: Reusing the existing Dataflow pipeline for data processing leverages your existing code and avoids redundant logic. Model Endpoint Predictions: Sending requests to the deployed model endpoint allows for efficient prediction generation. BigQuery Upload: Uploading predictions directly to BigQuery from the Dataflow pipeline integrates seamlessly with your data storage.

gscharly 👍 4 Selected: C

No need to deploy to endpoint as we need batch predictions. ModelBatchPredictOp can upload data to BQ. Dataflow pipeline logic can be implemented in DataflowPythonJobOp

fitri001 👍 1 Selected: B

TFRecords is a specific file format designed by TensorFlow for storing data in a way that's efficient for the machine learning framework. Here are some key points about TFRecords:

fitri001 👍 2 Selected: B

Option A: Vertex AI Pipelines' ModelBatchPredictOp is designed for batch prediction within pipelines, not for serving models through an endpoint. Option C: Importing the model directly into BigQuery is not feasible for TensorFlow models. Option D: Vertex AI Pipelines' BigqueryPredictModelJobOp assumes the model is already trained and hosted in BigQuery ML, which isn't the case here.

pinimichele01 👍 2 Selected: C

ModelBatchPredictOp -> upload automatically on BQ No need for endpoint --> C

pinimichele01 👍 1 Selected: C

agree with BlehMaks

pertoise 👍 3

Answer is C. No need for an endpoint here : Simply specify the BigQuery table URI in the ModelBatchPredictOp parameter and you're done automatically uploading to BigQuery

guilhermebutzke 👍 2 Selected: B

My Answer: B The most complete answer, and reuse a created pipeline. Don’t make sense to use DataflowPythonJobOp when you have already created a dataflow pipeline that does the same.

tavva_prudhvi 👍 1 Selected: B

Not A, C as they does not explicitly mention how the predictions will be uploaded to BigQuery.

daidai75 👍 1 Selected: B

The answer is B, optional A and B doesn't mention how to import prediction result to BigQuery.

pikachu007 👍 1 Selected: B

Option A: Vertex AI Pipelines are excellent for orchestrating ML workflows but might not be as efficient as Dataflow for large-scale data processing, especially with existing Dataflow logic. Option C: While Vertex AI Pipelines can handle model loading and prediction, Dataflow is better suited for large-scale data processing and BigQuery integration. Option D: BigQuery ML is primarily for in-database model training and prediction, not ideal for external models or large-scale data processing.

Comments & Corrections

No comments yet — spotted an error or have a note? Share it below.

Expert Analysis

Why the Answer Is Correct

Option C is the correct approach because it leverages Vertex AI Pipelines to orchestrate the entire workflow on a schedule without the overhead of a managed endpoint. By using DataflowPythonJobOp, you can reuse your existing data processing logic, and ModelBatchPredictOp allows you to run batch predictions directly on the imported model resource. This component supports specifying a BigQuery destination table for the output, satisfying the requirement to upload predictions automatically.

Why the Other Options Are Wrong

Option A is incorrect because deploying the model to a Vertex AI endpoint is designed for online, low-latency inference, not weekly batch jobs, and incurs unnecessary costs. Option B is incorrect because it suggests manually orchestrating a Dataflow pipeline to hit a live endpoint, which is a custom implementation that is less robust than the managed ModelBatchPredictOp. Option D is incorrect because importing a TensorFlow model into BigQuery ML and rewriting logic in SQL is inefficient compared to using the existing Dataflow pipeline and Vertex AI infrastructure.

Community Comment Notes

Commenters [1], [2], and [6] highlight that ModelBatchPredictOp can automatically upload results to BigQuery and that there is no need to deploy an endpoint for batch predictions. Some users argued for Option B based on the desire to reuse the existing pipeline code, but the community clarified that DataflowPythonJobOp fulfills this requirement within the Vertex AI Pipeline structure.

Official Reference

Exam Strategy

When you encounter questions requiring batch or scheduled predictions, immediately eliminate options that involve deploying to an endpoint unless online serving is explicitly required. Look for managed pipeline components like ModelBatchPredictOp that handle infrastructure and output integration automatically.

Related Analysis

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide