Linear Regression & Regularization
Comprehensive guide to linear regression, Ordinary Least Squares (OLS), MSE cost function, Normal Equation, OLS assumptions, Ridge (L2) vs Lasso (L1) regularization, and lambda hyperparameters.
What is Linear Regression?
Linear Regression is one of the simplest and most widely used supervised machine learning algorithms. It is used when the target variable is continuous.
The goal is to find the relationship between one or more input features and a continuous output.
- Predicting house prices
- Predicting salary
- Predicting temperature
- Predicting sales
- Predicting demand
Why is it Called "Linear"?
It is called linear because it assumes that the relationship between the input features and the output is approximately a straight line (or a linear combination of features).
Example
| Area (sq.ft) | Price (Lakhs) |
|---|---|
| 1000 | 50 |
| 1200 | 60 |
| 1500 | 75 |
| 1800 | 90 |
A Linear Regression model learns a line that best fits these points. Once trained, if someone asks: Area = 1600 sq.ft, the model predicts an approximate house price.
Mathematical Equation
For one feature:
- ŷ: Predicted value
- w₀: Bias (Intercept)
- w₁: Weight (Slope)
- x: Input feature
Multiple Linear Regression
Real-world problems usually involve multiple features. The equation becomes:
Predict house price using Area, Bedrooms, Bathrooms, and Age of House:
Understanding Weights
Weights indicate how much each feature influences the prediction.
- Every extra square foot increases the price by 0.04 units.
- Every additional bedroom increases the price by 5 units.
What is the Bias (Intercept)?
The bias is the predicted output when all feature values are zero (ŷ = w₀ when x₁ = x₂ = ... = xₙ = 0). Although it may not always have a practical meaning, it allows the regression line to shift up or down to fit the data better.
Goal of Linear Regression & Residuals
The objective is to find the best values of the weights so that the predicted values are as close as possible to the actual values.
Residual (Prediction Error)
The difference between the actual value and the predicted value is called the Residual:
Actual Price = 100 | Predicted Price = 95 | Residual = 5
Cost Function: Mean Squared Error (MSE)
For Linear Regression, the most common cost function is Mean Squared Error (MSE):
Why Do We Square the Error?
- Prevents positive and negative errors from canceling each other.
- Penalizes large errors more heavily.
- The function is smooth and differentiable, making optimization easier.
After squaring: 2² + (-2)² = 4 + 4 = 8 (Correctly reflects prediction mistakes).
Why Large Errors Are Penalized More
Model A Errors: 1, 2, 2
Squared Error: 1 + 4 + 4 = 9
Model B Errors: 1, 1, 5
Squared Error: 1 + 1 + 25 = 27
Model B receives a much larger penalty because it made one very large mistake (error of 5).
Ordinary Least Squares (OLS) & Normal Equation
Ordinary Least Squares (OLS) is the method used to find the best regression line by minimizing the sum of squared residuals: Minimize Σ (y − ŷ)².
The Normal Equation
One way to compute the optimal weights directly without iterative optimization:
- No learning rate required.
- No iterative optimization.
- Produces exact least-squares solution.
- Matrix inversion (XᵀX)⁻¹ is computationally expensive (O(n³)).
- Not suitable for datasets with very large feature counts.
- Numerically unstable if features are highly correlated.
Assumptions of Linear Regression (OLS Assumptions)
The relationship between the features and the target should be approximately linear.
Prediction errors should be independent of one another (especially critical in time-series data).
The variance of the residuals should remain approximately constant across all values of the input (even spread vs funnel shape).
Input features should not be highly correlated with each other (e.g. House Area in sq ft vs House Area in m²).
Multicollinearity causes unstable coefficients and erratic weight shifts.
Residual Analysis
Residual analysis helps verify whether OLS assumptions hold. A good residual plot should look like purely random noise without curved or funnel patterns.
Regularization: Ridge (L2) & Lasso (L1)
When features are high-dimensional or correlated, learned weights can become excessively large, causing overfitting. Regularization adds a penalty for large weights.
Ridge Regression (L2 Regularization)
Adds squared values of the weights to the loss function:
Shrinks all coefficients toward zero, but rarely makes coefficients exactly zero. Keeps all features in the model.
Lasso Regression (L1 Regularization)
Adds absolute values of the weights to the loss function:
Shrinks coefficients and can set weights exactly to zero, performing automatic feature selection.
Ridge vs Lasso Comparison
| Ridge (L2) | Lasso (L1) |
|---|---|
| Penalizes squared weights (w²) | Penalizes absolute weights (|w|) |
| Shrinks coefficients close to 0 | Sets coefficients exactly to 0 |
| Keeps all features | Performs feature selection |
| Good for correlated features | Good for sparse models |
Lambda (λ) Hyperparameter
Lambda controls the strength of regularization. Larger λ → Higher Bias, Lower Variance. Smaller λ → Lower Bias, Higher Variance.
Key Takeaways
- Linear Regression predicts continuous values.
- OLS finds the best-fit line by minimizing the sum of squared residuals.
- MSE is the most common cost function because it penalizes large errors.
- The Normal Equation provides a direct solution but is inefficient for very large datasets.
- Linear Regression assumes linearity, independent errors, homoscedasticity, and no multicollinearity.
- Ridge Regression (L2) shrinks coefficients but keeps all features.
- Lasso Regression (L1) performs automatic feature selection by setting some coefficients exactly to zero.
- Lambda controls the strength of regularization and helps balance bias and variance.