Why a 99% Training AUC Signals Data Leakage, Not Overfitting
Short answer
C Address data leakage by removing features highly correlated with the target value, the classic target-leakage symptom behind an instant 99% training AUC ROC.
Question
You haven't explored using any sophisticated algorithms or spent any time on hyperparameter tuning.
What should your next step be to identify and fix the problem?
Key Takeaways
The exam tests whether you recognize that near-perfect training performance with almost no effort is the fingerprint of target leakage in time-series data, and that the fix is inspecting feature-to-target correlations.
A 99% training AUC ROC achieved within a few experiments, without tuning or complex algorithms, indicates data leakage rather than overfitting. The correct fix is to identify and remove features whose correlation with the target reveals label leakage.
Common mistake: Choosing A or D by labeling the result overfitting. Overfitting requires substantial model capacity or tuning pressure, both explicitly absent here, so simplifying the algorithm or tuning hyperparameters cannot explain or fix a 99% score reached in a few experiments. Option B is a close second error: nested cross-validation rebalances performance estimates but does not locate or remove the leak.
Expert Analysis
The trap is treating this as an overfitting question. Options A and D both diagnose the 99% training AUC ROC as overfitting. That reading is tempting because an inflated training score looks like classic high variance. But the question supplies two decisive constraints: the score came "after just a few experiments" and "without sophisticated algorithms or any hyperparameter tuning." A model cannot overfit aggressively when you have barely fit it — there has been no capacity or tuning pressure to drive a generalization gap. So A (swap to a simpler algorithm) and D (tune hyperparameters downward) attack a cause that the scenario explicitly rules out.
Why leakage is the real diagnosis. The data is time-series, the single most leakage-prone setting in supervised learning, because target-derived or future information routinely rides into the features or across the train/test split. When leakage is present, even a trivial model scores near-perfectly on the leaked target — which is exactly the signature described.
Why C, not B. Both B and C claim to address leakage, but only C matches Google's own doctrine. The Vertex AI AutoML Tables documentation instructs users analyzing training results to inspect each feature's correlation with the target and, if that correlation is high, verify it is expected and "not an indication of target leakage." Option C restates that check verbatim as the remediation. Option B, nested cross-validation, is a model-selection and hyperparameter-selection protocol; it produces a less optimistic performance estimate, but it neither identifies which feature is leaking nor removes the leak — the model still trains on contaminated data. As one critique in the discussion notes, it does not directly address the problem.
Verdict: C. Remove the target-correlated features that are leaking label information.
Official Reference
- https://cloud.google.com/automl-tables/docs/train#analyze
- https://cloud.google.com/vertex-ai/docs/training/automl-tables
- https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/
These links open the official vendor documentation in a new tab.
Exam Strategy
In PMLE diagnostic questions, use effort-versus-performance to pick the failure mode: high performance after minimal effort means leakage, while high training performance after extensive tuning means overfitting. When two options name the same root cause, choose the one that performs the concrete detection or remediation step rather than a re-estimation technique like nested cross-validation, and prefer remedies that mirror the vendor's documented troubleshooting guidance.
Frequently Asked Questions
Why isn't 99% training AUC treated as overfitting in this question?
How does nested cross-validation relate to data leakage?
How do I spot target leakage in time-series classification features?
Community Votes
64% of anonymous learners picked answer C. Votes are pick records left by other test-takers — they are not the verified answer.
Discussion 34 contributions
Compiled from public community threads and reader submissions, lightly edited for clarity. Handles and vote tallies are shown as recorded, and comments reflect the views of individual contributors.
From public community threads 30 of 34
-
Paul_Dirac 👍 28Ans: B (Ref: https://towardsdatascience.com/time-series-nested-cross-validation-76adba623eb9) (C) High correlation doesn't mean leakage. The question may suggest target leakage and the defining point of this leakage is the availability of data after the target is available.(https://www.kaggle.com/dansbecker/data-leakage)
-
John_Pongthorn 👍 8 Answered CC: this is correct choice 1000000000% This is data leakage issue on training data https://cloud.google.com/automl-tables/docs/train#analyze The question is from this content. If a column's Correlation with Target value is high, make sure that is expected, and not an indication of target leakage. Let 's explain on my owner way, sometime the feature used on training data use value to calculate something from target value unintentionally, it result in high correlation with each other. for instance , you predict stock price by using moving average, MACD , RSI despite the fact that 3 features have been calculated from price (target).
-
Jarek7 👍 6This ref doesn't explain WHY we should use NCV in this case - it just explains HOW to use NCV when dealing with time series. Cross-validation, including nested cross-validation, is a powerful tool for model evaluation and hyperparameter tuning, but it does NOT DIRECTLY ADDRESS data leakage. Data leakage refers to a situation where information from the test dataset leaks into the training dataset, causing the model to have an unrealistically high performance. Nested cross-validation can indeed help provide a more accurate estimation of the model's performance on unseen data, but IT DOESN'T SOLVE the underlying issue of data leakage if it's already present.
-
sensev 👍 5D is incorrect since they mentioned that the initial model without sophisticated algorithm (e.g. model architecture) and without parameter tuning already achieves 99% accuracy. This suggests data leakage, and especially so since they mentioned it is time-series data, which suggest incorrect data split for train and evaluation.
-
pico 👍 3Option B: Nested cross-validation is primarily used to estimate model performance accurately and select the best model hyperparameters. While it's a good practice, it doesn't directly address the overfitting issue. It helps prevent over-optimistic model performance estimates but doesn't necessarily fix the overfitting problem. Option C: Removing features highly correlated with the target value can be a valid step in feature selection or preprocessing. However, it doesn't directly address the overfitting issue or explain why the model is performing exceptionally well on the training data. It's a separate step from mitigating overfitting. Option D: This option is incorrect. Tuning hyperparameters should aim to improve model performance on the validation set, not reduce it. In summary, the most appropriate next step is Option A:
Show 25 more comments
-
Sivaram06 👍 3 Answered BGemini Explanation: Nested Cross-validation: Nested cross-validation is a robust technique to detect and mitigate data leakage. It involves two loops of cross-validation: Inner loop: Tunes hyperparameters and performs model selection. Outer loop: Evaluates the model's performance on unseen data, giving you a more realistic estimate of how well your model generalizes. Why not C : C. Address data leakage by removing features highly correlated with the target value: While highly correlated features can sometimes be a sign of leakage, they might also be genuinely informative features. Removing them without proper analysis might hurt your model's performance.
-
David_ml 👍 3 Answered BQuite tricky but through elimination, correct answer is B. Model overfitting doesn't apply here as we can't tell if a model is overfitting by just looking at training data results.
-
Paul_Dirac 👍 3No. We won't be able to know whether the model is overfitting just by looking at the training set alone.
-
tavva_prudhvi 👍 2 Answered BOption C is a good step to avoid overfitting, but it's not necessarily the best approach to address data leakage. Data leakage occurs when information from the validation or test data leaks into the training data, leading to overly optimistic performance metrics. In time-series data, it's important to avoid using future information to predict past events. Removing features highly correlated with the target value may help to reduce overfitting, but it does not necessarily address data leakage. Therefore, applying nested cross-validation during model training is a better approach to address data leakage in this scenario.
-
Foxy2021 👍 2Select answer: C. --reason--- While B (nested cross-validation) helps improve the evaluation process and prevents over-optimistic performance estimates, it doesn't tackle the root cause of data leakage. Data leakage is often caused by features that are too closely tied to the target—in this case, the unusually high AUC suggests that the model is gaining unfair information.
-
AnnaR 👍 2B: correct. considering c, but why should we remove a feature of highly predictive nature?? for me, this does not explain the problem of overfitting... a highly predictive feature is also useful for good performance evaluated on the test set. --> Decide for B!
-
Pau1234 👍 2 Answered BAs per the PMLE cert book the answer is B. Since the model is performing well with training data, it is a case of data leakage. Cross‐validation is one of the strategies to overcome data leakage.
-
black_scissors 👍 2I agree. Besides, when a CV is done randomly (not split by the time point) it can make things worse.
-
Celia20210714 👍 2ANS: D (AUC ROC) value of 99% for training data after just a few experiments >> overfitting
-
hiromi 👍 2 Answered BB I agree with Paul_Dirac
-
taylz876 👍 1 Answered BA 99% AUC after a few quick experiments, with no tuning and no sophisticated algorithm, is a red flag for data leakage. Time series data is especially prone to it, because future information can slip into the features or the split. B is the only option that gives you a way to check for and expose it. Nested cross-validation scores the model on several held-out slices of data that are never used for tuning, so the over-optimistic score drops to a realistic one. For time series, the folds must follow time order. Why the others are wrong • A: Using a simpler algorithm treats the problem as overfitting. You haven't used a complex model, and your notes say changing the model won't fix leakage. • C: Removing features just because they are highly correlated with the target can throw away legitimate predictors. It only makes sense after you've shown which features leak. • D: Tuning hyperparameters to lower the AUC is backwards, since you never tune a model to degrade a metric. Tuning also doesn't fix leakage.
-
mayankblitzster 👍 1 Answered CC. Address data leakage by removing features highly correlated with the target value.Why? Achieving 99% AUC ROC on training data with minimal effort is a red flag. The most likely cause is data leakage, where the model has access to information it shouldn't during training. Features that are highly correlated with the target often leak label information, leading to unrealistically high performance. ❌ Why not the others? A. Use a less complex algorithm: Overfitting might not be the issue here—your model is likely learning from leaked data, not overfitting due to complexity. B. Nested cross-validation: This helps with model selection and tuning, but it doesn't prevent leakage if the data itself is flawed. D. Tuning hyperparameters to reduce AUC ROC: You don’t tune to reduce performance; you tune to generalize better. High training AUC isn't the goal—generalization is.
-
b7ad1d9 👍 1 Answered BStrangely high ROC => overly low bias/high variance =>data leakage. The time series nature of the data also points to data leakage as data leakage is more common in time series datasets. Now only B reference a method to reduce data leakage, i.e. NEsted Cross Validation which runs validation on multiple slices of data with different hyperparams (if I understood it correctly). Option C talks about removing highly correlated feature, but there isnt enough info that THAT is what is leading to data leakage. If the correlation is due to leakage , the correct fix is to re-engineer or remove the leaky component, not just remove every highly correlated feature.
-
b1a8fae 👍 1 Answered BI initially went with B- however after reading this: https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/ I think C is right. Quoted from the link: "Nested cross-validation is an approach to model hyperparameter optimization and model selection that attempts to overcome the problem of overfitting the training dataset.". Overfitting is exactly our problem here. Correlated features in the dataset may be a sign of data leakage, but they are not necessarily.
-
Fer660 👍 1 Answered AFrankly, none of the answers is fully acceptable. Certainly not C -- if we remove correlated features we are removing information from the data. Certainly not D -- if we tune to reduce AUROC, we are working against our own interest. Issues with B are described below by others. I thought A could be reasonable, except that the statement already tells us that we are not using a particularly sophisticated model. This question should be flagged for review by a public community thread.
-
atlas_lyon 👍 1 Answered BB: If splits are done chronologically(as it is always advised), Nested CV should work C: High correlation with target means we have to check if this is strong explanatory power or data leakage. dropping the features won't help us distinguish in those cases but may help reveal independence contribution of remaining features
-
pico 👍 1 Answered AOption A: This option is a reasonable choice. Switching to a less complex algorithm can help reduce overfitting, and using k-fold cross-validation can provide a better estimate of how well the model will generalize to unseen data. It's essential to ensure that the high performance isn't solely due to overfitting.
-
Mohamed_Mossad 👍 1 Answered B"You haven't explored using any sophisticated algorithms or spent any time on hyperparameter tuning" so we have a base /default hyperparameters estimator so overfitting is quite not possible , so it is a leakage problem , by inspection C is wrong , so it will be B
-
Sum_Sum 👍 1 Answered BI think its B. GPT4 makes a good argument about C: While this is a valid approach to handling data leakage, it might not be sufficient if the leakage is due to reasons other than high correlation, such as temporal leakage in time-series data.
-
desertlotus1211 👍 1The book mentions cross-validation.. NOT 'nested cross-validation' (page 34), however this answer is better than C. you want to remove value that are NOT correlated versus correlated as in answer C. ;)
-
Jarek7 👍 1 Answered Chttps://towardsdatascience.com/avoiding-data-leakage-in-timeseries-101-25ea13fcb15f Directly says: "Dive straight into the MVP, cross-validate later!" MVP stands for Minimum Viable Product
-
girgu 👍 1 Answered CNested cross validation will not work for time series data. Time series data require the expanding widow training data set. Seems most likely the issue is high correlation in columns.
-
black_scissors 👍 1 Answered CThere can be a feature causing data leakage which might have been overlooked. In addition, when cross-validation is done randomly, the leakage can be even bigger.
-
Liting 👍 1 Answered BAgree with Paul_Dirac. Also it is recommended to use nested-cross-validation to avoid data leakage in time series data.
-
382fca6 👍 1 Answered Cdata leakege and NCV wont solve it
Reader comments & corrections
No reader corrections yet — spotted an error or have a note? Share it below.
Related Analysis
Practice All PMLE Questions
Access 65 questions with complete answers and detailed explanations.
View Full PMLE Practice Test →