Robust Regression for Machine Learning in Python

In machine learning, regression analysis is a crucial tool for predicting continuous numerical outcomes based on input variables. Traditional regression techniques assume that the data follows a normal distribution and lacks outliers. However, real-world datasets often deviate from these assumptions, resulting in unreliable predictions. To combat this challenge, robust regression methods have been developed to offer more accurate and dependable results, even in the presence of outliers. This article delves into robust regression and explores how to implement these techniques using Python, one of the most popular programming languages for machine learning.

What is Robust Regression?

Robust regression is a variation of traditional regression analysis that is less sensitive to outliers in the data. Outliers are data points that deviate significantly from the majority of the data points, and they can have a substantial impact on the regression model's performance. Traditional regression methods, such as ordinary least squares (OLS), treat all data points equally, regardless of their distance from the central cluster. This makes them highly influenced by outliers, resulting in biased parameter estimates and poor predictive performance.

Robust regression techniques, on the other hand, aim to down-weight the impact of outliers by assigning lower weights to these data points during the model fitting process. By giving less weight to outliers, robust regression models can provide more accurate parameter estimates and better predictions.

Types of Robust Regression Methods

Several robust regression methods have been developed over the years. Let's discuss a few commonly used ones:

Huber Regression

Huber regression is a robust regression method that combines the advantages of both least squares regression and absolute deviation regression. It minimizes the sum of squared residuals for data points close to the regression line while minimizing the absolute residuals for data points that deviate significantly from the line. This way, it strikes a balance between the two and provides robust parameter estimates.

Theil-Sen Regression

Theil-Sen regression is a non-parametric robust regression method that estimates the slope of the regression line by considering all possible pairs of points. It calculates the median of the slopes of the lines connecting each pair of points and provides a robust estimate of the overall slope. The Theil-Sen method is computationally efficient and provides robust estimates even when up to 29% of the data points are outliers.

RANSAC (RANdom SAmple Consensus)

RANSAC is an iterative robust regression method that randomly selects a subset of data points, fits a regression model to these points, and then calculates the number of inliers (data points that are consistent with the model) and outliers (data points that deviate from the model). It repeats this process for a certain number of iterations, selecting the model with the highest number of inliers as the final robust regression model.

Implementing Robust Regression in Python

Python offers numerous libraries that provide reliable regression methods. A well-known library for this purpose is statsmodels, renowned for its extensive statistical modeling capabilities, including the implementation of robust regression. Let's explore an example using synthetic data to demonstrate robust regression techniques.

Example: Huber Regression vs Ordinary Least Squares

import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from sklearn.linear_model import LinearRegression, HuberRegressor
from sklearn.preprocessing import StandardScaler

# Generate synthetic data with outliers
np.random.seed(42)
n_samples = 100
X = np.random.randn(n_samples, 1)
y = 2 * X.ravel() + np.random.randn(n_samples) * 0.5

# Add outliers
n_outliers = 10
X_outliers = np.random.uniform(low=-3, high=3, size=(n_outliers, 1))
y_outliers = 10 * np.random.randn(n_outliers)

# Combine data
X_combined = np.vstack([X, X_outliers])
y_combined = np.hstack([y, y_outliers])

print(f"Dataset size: {len(X_combined)} samples with {n_outliers} outliers")

Dataset size: 110 samples with 10 outliers

Comparing OLS and Huber Regression

# Fit OLS regression
ols_model = LinearRegression()
ols_model.fit(X_combined, y_combined)

# Fit Huber regression
huber_model = HuberRegressor(epsilon=1.35)
huber_model.fit(X_combined, y_combined)

print(f"OLS coefficient: {ols_model.coef_[0]:.3f}")
print(f"Huber coefficient: {huber_model.coef_[0]:.3f}")
print(f"True coefficient: 2.000")

OLS coefficient: 1.423
Huber coefficient: 1.987
True coefficient: 2.000

Using Statsmodels for Robust Regression

# Using statsmodels for more detailed analysis
X_sm = sm.add_constant(X_combined)

# Fit robust regression model
robust_model = sm.RLM(y_combined, X_sm, M=sm.robust.norms.HuberT())
robust_results = robust_model.fit()

print("Robust Regression Summary:")
print(f"Intercept: {robust_results.params[0]:.3f}")
print(f"Coefficient: {robust_results.params[1]:.3f}")
print(f"Converged: {robust_results.converged}")

Robust Regression Summary:
Intercept: 0.042
Coefficient: 1.981
Converged: True

Benefits of Robust Regression

Robust regression techniques offer several advantages over traditional regression methods when dealing with data containing outliers:

Key advantages include:

• Increased robustness: Robust regression methods handle outliers and influential observations better, providing more reliable parameter estimates and improved predictive performance.
• Better model interpretation: By down-weighting outliers, robust regression provides parameter estimates that better represent the majority of the data.
• Versatility: These techniques can be applied to various regression problems, from simple linear to complex nonlinear relationships.
• Easy implementation: Modern Python libraries make it straightforward to implement robust regression in existing workflows.

Conclusion

Robust regression is a valuable technique for improving machine learning model reliability when data contains outliers or violates traditional regression assumptions. By down-weighting extreme observations, robust regression provides more accurate parameter estimates and better predictive performance. Python libraries like statsmodels and scikit-learn make implementing these techniques straightforward and accessible for practitioners.