How to automate weekly batch predictions to BigQuery?

PMLE By islever Editorial Last updated

Short answer

C Import the model into Vertex AI. On Vertex AI Pipelines, create a pipeline that uses the

Question

You have recently used TensorFlow to train a classification model on tabular data. You have created a Dataflow pipeline that can transform several terabytes of data into training or prediction datasets consisting of TFRecords. You now need to productionize the model, and you want the predictions to be automatically uploaded to a BigQuery table on a weekly schedule. What should you do?
  1. Import the model into Vertex AI and deploy it to a Vertex AI endpoint. On Vertex AI Pipelines, create a pipeline that uses the DataflowPythonJobOp and the ModelBacthPredictOp components.
  2. Import the model into Vertex AI and deploy it to a Vertex AI endpoint. Create a Dataflow pipeline that reuses the data processing logic sends requests to the endpoint, and then uploads predictions to a BigQuery table.
  3. Import the model into Vertex AI. On Vertex AI Pipelines, create a pipeline that uses the Correct Answer
  4. Import the model into BigQuery. Implement the data processing logic in a SQL query. On Vertex AI Pipelines create a pipeline that uses the BigquervQueryJobOp and the BigqueryPredictModelJobOp components.

Key Takeaways

The exam tests the distinction between online serving and batch processing, where the common trap is unnecessarily deploying a Vertex AI endpoint for a scheduled batch prediction job.

To automate weekly batch predictions for a TensorFlow model, import the model into Vertex AI and orchestrate the workflow using Vertex AI Pipelines with DataflowPythonJobOp and ModelBatchPredictOp. The community consensus confirms that deploying a serving endpoint is unnecessary for batch workloads and that the pipeline components can handle data processing and BigQuery uploads efficiently.

Common mistake: Choosing Option B is a common mistake because users often believe they must deploy an endpoint and manually invoke a Dataflow pipeline to reuse logic, overlooking that DataflowPythonJobOp allows reusing that logic within a managed pipeline without an endpoint.

Expert Analysis

Why the Answer Is Correct

Option C is the correct approach because it leverages Vertex AI Pipelines to orchestrate the entire workflow on a schedule without the overhead of a managed endpoint. By using DataflowPythonJobOp, you can reuse your existing data processing logic, and ModelBatchPredictOp allows you to run batch predictions directly on the imported model resource. This component supports specifying a BigQuery destination table for the output, satisfying the requirement to upload predictions automatically.

Why the Other Options Are Wrong

Option A is incorrect because deploying the model to a Vertex AI endpoint is designed for online, low-latency inference, not weekly batch jobs, and incurs unnecessary costs. Option B is incorrect because it suggests manually orchestrating a Dataflow pipeline to hit a live endpoint, which is a custom implementation that is less robust than the managed ModelBatchPredictOp. Option D is incorrect because importing a TensorFlow model into BigQuery ML and rewriting logic in SQL is inefficient compared to using the existing Dataflow pipeline and Vertex AI infrastructure.

Community Comment Notes

Commenters [1], [2], and [6] highlight that ModelBatchPredictOp can automatically upload results to BigQuery and that there is no need to deploy an endpoint for batch predictions. Some users argued for Option B based on the desire to reuse the existing pipeline code, but the community clarified that DataflowPythonJobOp fulfills this requirement within the Vertex AI Pipeline structure.

Official Reference

https://cloud.google.com/vertex-ai/docs/pipelines/dataflow-component#dataflowpythonjobop

This link opens the official vendor documentation in a new tab.

Exam Strategy

When you encounter questions requiring batch or scheduled predictions, immediately eliminate options that involve deploying to an endpoint unless online serving is explicitly required. Look for managed pipeline components like ModelBatchPredictOp that handle infrastructure and output integration automatically.

Community Votes

C
64%
B
36%

64% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 16 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 16

  1. BlehMaks 👍 10 Answered C
    The DataflowPythonJobOp operator lets you create a Vertex AI Pipelines component that prepares data by submitting a Python-based Apache Beam job to Dataflow for execution. https://cloud.google.com/vertex-ai/docs/pipelines/dataflow-component#dataflowpythonjobop Using we can specify an output location for Vertex AI to store predictions results https://cloud.google.com/vertex-ai/docs/pipelines/batchprediction-component A - is incorrect since we dont need an endpoint for batch predictions B - creating a new Dataflow pipeline is redundant
  2. gscharly 👍 4 Answered C
    No need to deploy to endpoint as we need batch predictions. ModelBatchPredictOp can upload data to BQ. Dataflow pipeline logic can be implemented in DataflowPythonJobOp
  3. pertoise 👍 3
    Answer is C. No need for an endpoint here : Simply specify the BigQuery table URI in the ModelBatchPredictOp parameter and you're done automatically uploading to BigQuery
  4. rcapj 👍 2
    B Vertex AI Deployment: Vertex AI provides a managed environment for deploying machine learning models. It simplifies the process and ensures scalability. Dataflow Pipeline Reuse: Reusing the existing Dataflow pipeline for data processing leverages your existing code and avoids redundant logic. Model Endpoint Predictions: Sending requests to the deployed model endpoint allows for efficient prediction generation. BigQuery Upload: Uploading predictions directly to BigQuery from the Dataflow pipeline integrates seamlessly with your data storage.
  5. fitri001 👍 2 Answered B
    Option A: Vertex AI Pipelines' ModelBatchPredictOp is designed for batch prediction within pipelines, not for serving models through an endpoint. Option C: Importing the model directly into BigQuery is not feasible for TensorFlow models. Option D: Vertex AI Pipelines' BigqueryPredictModelJobOp assumes the model is already trained and hosted in BigQuery ML, which isn't the case here.
Show 11 more comments
  1. guilhermebutzke 👍 2 Answered B
    My Answer: B The most complete answer, and reuse a created pipeline. Don’t make sense to use DataflowPythonJobOp when you have already created a dataflow pipeline that does the same.
  2. pinimichele01 👍 2 Answered C
    ModelBatchPredictOp -> upload automatically on BQ No need for endpoint --> C
  3. pikachu007 👍 1 Answered B
    Option A: Vertex AI Pipelines are excellent for orchestrating ML workflows but might not be as efficient as Dataflow for large-scale data processing, especially with existing Dataflow logic. Option C: While Vertex AI Pipelines can handle model loading and prediction, Dataflow is better suited for large-scale data processing and BigQuery integration. Option D: BigQuery ML is primarily for in-database model training and prediction, not ideal for external models or large-scale data processing.
  4. lunalongo 👍 1 Answered C
    C is the best option because it uses: 1) Vertex AI Pipelines for orchestrating the flow (managed and scalable). 2) DataflowPythonJobOp for prep and ModelBatchPredictOp for batch predictions on Vertex AI. A deploys the model to a Vertex AI endpoint, inefficient for batch jobs! B uses a single Dataflow pipeline, which needs custom Vertex AI and BQ integration. *D uses BigQuery, a datawarehouse, for model deployment and prediction.
  5. AzureDP900 👍 1
    B is right because 1)You've already trained a classification model using TensorFlow, so you need to productionize it by deploying it to a Vertex AI endpoint. 2)To automate the prediction process on a weekly schedule, you can create a Dataflow pipeline that reuses your existing data processing logic. This pipeline will send requests to the deployed model for inference and then upload the predicted results to BigQuery.
  6. fitri001 👍 1 Answered B
    TFRecords is a specific file format designed by TensorFlow for storing data in a way that's efficient for the machine learning framework. Here are some key points about TFRecords:
  7. AK2020 👍 1 Answered B
    Uploading predictions directly to BigQuery from the Dataflow pipeline integrates seamlessly with your data storage.
  8. tavva_prudhvi 👍 1 Answered B
    Not A, C as they does not explicitly mention how the predictions will be uploaded to BigQuery.
  9. daidai75 👍 1 Answered B
    The answer is B, optional A and B doesn't mention how to import prediction result to BigQuery.
  10. Prakzz 👍 1 Answered B
    Only option B talks about loading the data to BigQuery
  11. pinimichele01 👍 1 Answered C
    agree with BlehMaks

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

How to preprocess BigQuery data for TensorFlow?

S-grade

How to Track Model and Prediction Lineage in Vertex AI?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide