Preventing Data Leakage in Time-Series Models via Temporal Splitting

PMLE By islever Editorial Last updated

Exploring and preprocessing data for ML. Building models given the task considering cost, complexity, latency, and scalability.

Short answer

B Split the training and test data based on time rather than a random split to avoid leakage.

Question

You are building a model to predict daily temperatures. You split the data randomly and then transformed the training and test datasets. Temperature data for
model training is uploaded hourly. During testing, your model performed with 97% accuracy; however, after deploying to production, the model's accuracy dropped
to 66%. How can you make your production model more accurate?
  1. Normalize the data for the training, and test datasets as two separate steps.
  2. Split the training and test data based on time rather than a random split to avoid leakage. Correct Answer
  3. Add more data to your test set to ensure that you have a fair distribution and sample for testing.
  4. Apply data transformations before splitting, and cross-validate to make sure that the transformations are applied to both the training and test sets.

Key Takeaways

The main trap is assuming random splitting is sufficient; however, for time-series data with autocorrelation, random splits cause data leakage, necessitating chronological splitting.

This question addresses data leakage in time-series forecasting where random splitting inflates test accuracy. The correct approach is to use a temporal split to ensure the model is evaluated on unseen future data.

Common mistake: Option D is a common distractor because proper data transformation is important, but it does not fix the leakage problem inherent in random splitting of time-series data.

Expert Analysis

The scenario describes a classic case of data leakage in time-series forecasting. The key indicators are the high test accuracy (97%) versus significantly lower production accuracy (66%) and the nature of the data (hourly temperature uploads). Temperature data exhibits strong temporal autocorrelation, meaning adjacent data points are highly similar.

When using a random split, the training and test sets are mixed chronologically. This allows the model to learn patterns from 'future' data points that are temporally close to 'past' points in the same hour or day. Essentially, the model sees near-duplicates of its training data in the test set, leading to inflated performance metrics that do not generalize to unseen future data.

To resolve this, you must respect the temporal order of the data. By splitting the data based on time (e.g., using all data up to date T for training and data after T for testing), you ensure that the model is evaluated on truly unseen future data. This simulates the real-world deployment scenario where predictions are made on new incoming data. Options A, C, and D fail to address the fundamental issue of temporal leakage caused by random splitting. Option D suggests applying transformations before splitting, but if the split itself is random, leakage persists regardless of preprocessing steps.

Exam Strategy

Always identify the data type before selecting a validation strategy. For time-series or sequential data, look for keywords like 'hourly', 'daily', or 'temporal'. If present, reject random splits and choose temporal/chronological splitting to prevent data leakage, which artificially inflates test scores.

Frequently Asked Questions

Why does random splitting cause data leakage in time-series?

Random mixing allows the model to see data points from the future relative to the training point during testing. Since time-series data often has autocorrelation (nearby points are similar), the model memorizes these patterns instead of learning generalizable trends.

Does normalizing data before or after splitting matter?

Yes, normalizing before splitting can introduce leakage if statistics (mean/std) are calculated on the entire dataset including the test set. However, the primary issue here is the split method itself, not just the normalization timing.

More PMLE FAQ →

Community Votes

B
86%
D
14%

86% of anonymous learners picked answer B. Votes are pick records left by other test-takers — they are not the verified answer.

Discussion 20 contributions

Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.

From public community threads 20

  1. maartenalexander 👍 38
    B. If you do time series prediction, you can't borrow information from the future to predict the future. If you do, you are artificially increasing your accuracy.
  2. giaZ 👍 3 Answered B
    If you do random split in a time series, your risk that training data will contain information about the target (definition of leakage), but similar data won't be available when the model is used for prediction. Leakage causes the model to look accurate until you start making actual predictions with it.
  3. Danny2021 👍 3
    B. D doesn't improve anything at all. Split and Transform is no different than Transform and Split if the transform logic is the same.
  4. David_ml 👍 3 Answered B
    You don't split data randomly for time series prediction.
  5. mmona19 👍 2 Answered B
    B should be the answer. D is incorrect as normalize before split is going to do data leak https://community.rapidminer.com/discussion/32592/normalising-data-before-data-split-or-after
Show 15 more comments
  1. Mohamed_Mossad 👍 2 Answered B
    train accuracy 97% , production accuracy 66% ---> time series data ---> random split ---> cause leakage , answer is B
  2. PhilipKoku 👍 2 Answered B
    B) Time split to avoid leaking data.
  3. SergioRubiano 👍 2 Answered D
    D is correct. cross-validate
  4. xiaoF 👍 2
    agree B as well
  5. JobQ 👍 2
    I think is B
  6. taylz876 👍 1 Answered B
    B - The model scored 97% in testing but only 66% in production. That big gap is the classic sign of data leakage. Temperature data is a time series, uploaded hourly, and temperatures at neighboring hours are very similar. A random split puts hours from the same period into both the training and test sets, so the test set contains near-copies of the training data. The model looks great on the test set and then fails on truly new, future data. Splitting by time (train on older data, test on newer) mimics what happens in production and fixes the leakage.
  7. OpenKnowledge 👍 1 Answered B
    The core principle is to use historical data (the training set) to build a model and then evaluate its performance on future, unseen data (the test set). So, the dataset must be split into training and testing sets based on time (temporal split/out-of-time split/chronological split). This approach simulates real-world scenarios where models must make predictions on data that occurred after they were trained, making it especially useful for time series forecasting.
  8. desertlotus1211 👍 1 Answered B
    D is incorrect: Applying transformations before splitting is important, but it does not resolve the issue of time leakage. Even if transformations are done correctly, the random split will still lead to inflated test accuracy and poor production performance. This option focuses on correct data processing, but it does not address the leakage caused by random splitting in time series data.
  9. Sum_Sum 👍 1 Answered B
    they did not explicitly say forecasting, but splitting by time is the number one rule you learn
  10. baimus 👍 1
    B I mean. Sorry I wrote that comment very early and there is no delete key!
  11. jsalvasoler 👍 1 Answered B
    temporal split is a must in time series forecasting evaluation
  12. fragkris 👍 1 Answered B
    Definetely B
  13. Jijiji 👍 1
    seems like D
  14. M25 👍 1 Answered B
    Went with B
  15. baimus 👍 1 Answered D
    It's D

Reader comments & corrections

No reader corrections yet — spotted an error or have a note? Share it below.

Log in to comment, report an error, or add a note about this question.

How to Minimize Development Time for Weekly TensorFlow Retraining in Vertex AI?

S-grade

How to Handle Categorical Features in BigQuery ML for Fast Deployment?

S-grade

How to preprocess BigQuery data for TensorFlow?

S-grade

How to Track Model and Prediction Lineage in Vertex AI?

S-grade

Practice All PMLE Questions

Access 65 questions with complete answers and detailed explanations.

View Full PMLE Practice Test →

← Back to PMLE Study Guide