Data Splitting, Validation Strategy & Data Leakage

Mastering dataset partitioning, generalization metrics, stratified sampling for imbalanced data, chronological time-series splits, and avoiding data leakage.

Why Do We Split the Dataset?

One of the biggest mistakes beginners make is evaluating a model on the same data it was trained on.

Exam Analogy
Imagine you are preparing for an exam. If your teacher gives you the exact same questions during the final exam that you already memorized, you will score 100%. But does that mean you actually understand the subject? No. The real test is solving new questions that you have never seen before.

Machine Learning works exactly the same way. We train on one set of data and evaluate on completely unseen data. This tells us how well the model can generalize.

What is Generalization?

Generalization is the ability of a machine learning model to make correct predictions on new, unseen data.

A good model should not only remember training examples but should also perform well on future data.

Example: Cats and Dogs Classifier

Training Images: Cat, Dog, Cat, Dog...

Testing Images: New Cat Image, New Dog Image

If the model predicts correctly on these unseen images, it has good generalization.

Standard Data Split

The dataset is usually divided into three parts:

Entire Dataset → Training (60–80%) & Remaining → Validation (10–20%) & Test (10–20%)

Dataset	Percentage	Purpose
Training Set	60–80%	Learn model parameters
Validation Set	10–20%	Tune hyperparameters and compare models
Test Set	10–20%	Final unbiased evaluation

These percentages are guidelines, not strict rules.

Training Set

The training set is the data from which the model learns. During training, the algorithm adjusts its internal parameters (weights) to minimize prediction errors.

House Price Example:

Area: 1200 sq ft, Bedrooms: 2 → 50 Lakh

Area: 1500 sq ft, Bedrooms: 3 → 72 Lakh

Area: 1800 sq ft, Bedrooms: 3 → 82 Lakh

The model learns relationships like: Larger area → Higher price; More bedrooms → Higher price.

Validation Set

The validation set helps us choose the best model and tune hyperparameters. The model does not learn from validation data. Instead, validation answers:

Which algorithm performs better?
Which learning rate should I use?
What should be the regularization strength?
How many trees should Random Forest have?

Model	Validation Accuracy
Logistic Regression	91%
Decision Tree	88%
Random Forest	95% (Selected)

Test Set

The test set is used only once. Its purpose is to estimate how the final model will perform in the real world. Think of it as the final university examination. Once you have looked at the test results and changed your model accordingly, it is no longer an unbiased test.

The Test Set Is Sacred
The test set must remain untouched during training and tuning to maintain an unbiased evaluation.

Why Should the Test Set Be Used Only Once?

Suppose you repeatedly check test accuracy:

Train → Test → Improve Model → Test Again → Improve Again → Test Again

Eventually, your model starts indirectly learning the test set. Now the test accuracy is no longer trustworthy.

Training data is used many times.
Validation data is used many times.
Test data is used only once at the very end.

Complete ML Pipeline

Dataset → Split Data → Training (Train Model) → Validation (Hyperparameter Tuning) → Test (Final Evaluation)

Why Not Train on the Entire Dataset?

If we train and test on the same data:

Training Accuracy = 100% | Testing Accuracy = 100% (Memorized)
Real-World Accuracy = 65% (Fails on new data)

The model simply memorized the training data instead of learning true patterns.

Choosing the Right Split

There is no universal rule. Typical choices are:

Dataset Size	Common Split
Small Dataset	80 : 20
Medium Dataset	70 : 15 : 15
Large Dataset	80 : 10 : 10

For extremely large datasets, even 1–5% test data may be sufficient.

Stratified Sampling

Sometimes datasets are imbalanced.

Disease Detection Imbalanced Example:

Healthy: 9,500 | Disease: 500 (Only 5% positive)

A pure random split might accidentally produce a test set with zero disease cases, rendering evaluation meaningless.

What is Stratified Split?

A stratified split preserves the exact class distribution in every partitioned dataset.

Original Dataset: 90% Cats | 10% Dogs

Training Set: 90% Cats | 10% Dogs

Validation Set: 90% Cats | 10% Dogs

Test Set: 90% Cats | 10% Dogs

Why is Stratified Sampling Important?

Fair evaluation
Stable metrics
Better representation of minority classes
More reliable model comparison

Use stratified sampling whenever dealing with classification tasks with imbalanced data.

Data Splitting for Time Series

Time-series data is different. Future values should never influence predictions about the past.

Wrong Split (Random Shuffle)

Jan → Apr → Feb → May → Mar

This leaks future information into past predictions!

Correct Split (Chronological)

Train (Jan, Feb, Mar) → Val (Apr) → Test (May)

Always train on the past and evaluate on the future.

Why Time-Series Cannot Be Randomly Shuffled

Suppose today's stock price depends on yesterday. If tomorrow's price appears in training, the model indirectly learns future information, producing unrealistically high accuracy.

What is Data Leakage?

Data leakage occurs when information that would not be available during real prediction is accidentally used while training.

Consequences of Data Leakage:

Validation accuracy becomes unrealistically high.
Test accuracy also looks impressive.
Production performance becomes poor.

Data leakage is one of the most common causes of ML failures.

Simple Example of Data Leakage

Suppose we want to predict whether a student will pass:

Features: Study Hours, Attendance, Assignment Marks

Leaked Feature: Final Exam Score

The final exam score is generated after the prediction should have been made. During real prediction, the final exam score is unknown. This is data leakage.

Common Causes of Data Leakage

1. Using Future Information

Predicting tomorrow's sales using tomorrow's revenue — impossible in real life.

2. Target Leakage

A feature is directly derived from the target (e.g. Loan Status, Approved Date in loan approval prediction).

3. Preprocessing Before Splitting

Wrong: Normalize Entire Dataset → Split Data (Leaks test set statistics!)

Correct: Split Data → Normalize Training Set → Apply Same Transformation to Val & Test

If entire dataset mean = 120 and training mean = 105, normalizing with 120 indirectly gains future test sample info.

4. Duplicate Samples

Duplicate images or rows appearing in both Training and Test sets artificially inflates test accuracy.

The Golden Rule
Always remember: Split first, preprocess later.
Raw Dataset → Split → Training → Fit Scaler → Transform Training → Use Same Scaler → Validation → Test

Machine Learning Pipelines

Modern ML libraries (like Scikit-Learn Pipelines) help prevent leakage:

Split → Training → Scaler → Feature Engineering → Model → Prediction

Everything learned during preprocessing comes only from training data.

A Simple Question to Detect Leakage
"Would this information be available when making a real prediction?"
If the answer is No, then it is almost certainly data leakage.

Best Practices & Common Mistakes

Best Practices

Split data before preprocessing.
Never fit preprocessing on validation or test data.
Use stratified splits for imbalanced classification datasets.
Preserve chronological order for time-series data.
Keep the test set untouched until the final evaluation.
Remove duplicate records across splits.
Use ML pipelines to avoid accidental leakage.

Common Mistakes

Training on test data: Gives unrealistic accuracy.
Normalizing before split: Leaks future statistics.
Random split for time-series: Uses future information.
Duplicate images in train/test: Artificially increases accuracy.
Tuning on test set: Makes test set biased.

Key Takeaways

The purpose of data splitting is to estimate real-world performance.
Training data is used for learning.
Validation data is used for model selection and hyperparameter tuning.
The test set should be used only once.
Stratified sampling preserves class distribution.
Time-series datasets should always be split chronologically.
Data leakage occurs when future or unavailable information is used during training.
The golden rule is: Split first, preprocess later.

Interview Questions and Answers

Prev: Intro to ML & Workflow Next: Underfitting, Overfitting & Bias-Variance