Optimizing ML Pipeline Architecture for Batch Predictions and Periodic Training

PMLE By islever Editorial Last updated

Developing end-to-end ML pipelines. Scaling online model serving.

Short answer

B Implement two separate pipelines in Agent Platform Pipelines: one for monthly training and another for daily batch predictions to minimize costs by decoupling resource-intensive tasks.

Question

You work for a large online retailer. You are productionizing a demand forecasting model designed to predict future sales for inventory management. You need to retrain the model with fresh data every 30 days. The inventory planning team requires daily batch predictions for replenishment orders. You need to design a solution that generates daily predictions while minimizing resource costs. Which pipeline architecture should you implement?
  1. Schedule a pipeline in Agent Platform Pipelines that generates batch predictions daily. Include training as a conditional step that runs if it is day 30 of the cycle.
  2. Implement two pipelines in Agent Platform Pipelines: a continuous training pipeline that runs every 30 days, and a daily batch prediction pipeline that runs the batch prediction job. Correct Answer
  3. Implement a pipeline in Agent Platform Pipelines for training that runs every 30 days to update a versioned model on an Agent Platform endpoint. Configure a Cloud Run function to send the daily batch dataset to the endpoint for predictions.
  4. Implement a pipeline in Agent Platform Pipelines with sequential steps for data preparation, model training, and batch prediction. Schedule the pipeline to run daily.

Key Takeaways

The main trap is combining training and prediction into a single daily pipeline or using costly online serving; decoupling them by schedule is the cost-effective solution.

Learn why separating training and prediction into distinct pipelines in Agent Platform reduces costs while meeting daily prediction and monthly retraining needs for demand forecasting models.

Common mistake: Candidates often choose Option A, thinking a single pipeline with conditional steps is simpler. However, this mixes high-cost training resources with daily low-cost prediction tasks, leading to inefficiency and operational complexity.

Expert Analysis

Separation of Concerns for Cost Efficiency

The core challenge here is balancing two distinct operational requirements: retraining the model monthly and generating daily batch predictions. In Google Cloud's Agent Platform (and ML infrastructure generally), training and inference are resource-intensive processes that should be decoupled to optimize cost and complexity.

Why Option B is Correct Option B proposes two separate pipelines:

  1. A training pipeline running every 30 days. This job consumes significant compute resources to retrain the model on fresh data.
  2. A batch prediction pipeline running daily. This job uses the pre-trained, versioned model from storage or a managed endpoint to generate predictions on new inventory data.

This architecture minimizes costs because the expensive training resources are only allocated once a month. The daily prediction jobs can use lightweight, cost-effective batch processing methods without needing an always-on serving endpoint.

Why Other Options Are Suboptimal

  • Option A: Running a single pipeline daily with a conditional training step introduces unnecessary complexity and risk. If the conditional logic fails, you might miss a retrain or trigger an unwanted one. It also keeps the pipeline definition monolithic, making it harder to manage and monitor.
  • Option C: Configuring a Cloud Run function to send data to an endpoint implies online serving. Online endpoints (even for batch requests) often incur higher costs due to idle capacity or per-request overhead compared to pure batch prediction jobs. Furthermore, managing versioning manually via Cloud Run adds operational overhead.
  • Option D: Running training and prediction daily is wasteful. Retraining every day is unnecessary for a demand forecasting model that updates monthly and incurs high compute costs daily, violating the "minimize resource costs" constraint.

The key insight is that decoupling training and inference schedules allows each process to run at its optimal frequency using the most appropriate (and cost-effective) execution method.

Exam Strategy

When designing ML solutions, always ask: 'Do these tasks need to run on the same schedule?' If training and inference have different frequencies, decouple them into separate pipelines or jobs. This prevents wasting expensive training compute on daily runs and simplifies error handling. Look for answers that explicitly mention 'separate' or 'distinct' workflows for different lifecycle stages.

Frequently Asked Questions

Why not use a single pipeline with conditional training?

A single pipeline complicates orchestration. Conditional logic can fail, and mixing high-cost training with daily prediction tasks makes scaling and monitoring harder. Separate pipelines allow independent optimization.

Is batch prediction cheaper than online serving?

Yes, typically. Batch prediction processes large datasets asynchronously using spot/preemptible instances if allowed, whereas online serving requires always-on resources, which are more expensive for infrequent daily batches.

More PMLE FAQ →

Community Votes

B
100%

100% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 2 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 2

  1. taylz876 👍 1 Answered B
    Why B is right • Training and prediction run on different schedules. Retraining every 30 days and predicting every day are two jobs, so two pipelines keep each one simple. • Minimal cost: batch prediction runs once a day and needs no always-on endpoint, and training runs only every 30 days. Why the others are wrong • A: One daily pipeline with a conditional training step works, but it mixes two schedules in one pipeline and adds complexity. • C: An endpoint with a function sending the daily batch costs money all the time for what is a bulk job. • D: Retraining every day wastes compute, because the model only needs a refresh every 30 days.
  2. RajuMVV 👍 1 Answered B
    The correct answer is B. Implement two pipelines in Agent Platform Pipelines: a continuous training pipeline that runs every 30 days, and a daily batch prediction pipeline that runs the batch prediction job. Decoupled Lifecycle & Schedules Cost Efficiency

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

Monitoring ML Model Performance with Precision and Recall Sampling

S-grade

Optimizing ML Model Recall for Compliance Risk

S-grade

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

Building CLV Models with BigQuery ML in Vertex AI Workbench

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide