Understanding Geometric Interpretation of Regression

Regression analysis is one of the most fundamental statistical methods for examining relationships between variables. The geometric interpretation of regression provides visual insights into how variables relate to each other in dimensional space, making complex relationships easier to understand and interpret.

What is Regression Analysis?

Regression analysis models the relationship between independent variables (predictors) and a dependent variable (response). The goal is to find the line or curve that best represents this relationship, allowing us to predict the dependent variable's value based on the independent variables.

There are two main types:

• Simple Linear Regression One independent variable
• Multiple Linear Regression Two or more independent variables

Simple Linear Regression: The Line of Best Fit

In simple linear regression, the relationship between variables x and y is represented as a straight line in two-dimensional space. This regression line minimizes the sum of squared residuals (differences between actual and predicted values).

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Generate sample data
np.random.seed(42)
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]).reshape(-1, 1)
y = 2 * x.ravel() + 1 + np.random.normal(0, 1, 10)

# Fit linear regression
model = LinearRegression()
model.fit(x, y)

# Make predictions
y_pred = model.predict(x)

# Plot the results
plt.figure(figsize=(8, 6))
plt.scatter(x, y, color='blue', label='Data points')
plt.plot(x, y_pred, color='red', linewidth=2, label='Regression line')
plt.xlabel('Independent Variable (x)')
plt.ylabel('Dependent Variable (y)')
plt.title('Simple Linear Regression')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

print(f"Slope: {model.coef_[0]:.2f}")
print(f"Intercept: {model.intercept_:.2f}")

The slope indicates how much y changes for each unit increase in x. The intercept represents the value of y when x equals zero.

Multiple Linear Regression: The Regression Plane

With two independent variables, the relationship forms a regression plane in three-dimensional space ?

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.linear_model import LinearRegression

# Generate 3D data
np.random.seed(42)
x1 = np.random.uniform(0, 10, 50)
x2 = np.random.uniform(0, 10, 50)
y = 2 * x1 + 3 * x2 + 5 + np.random.normal(0, 2, 50)

# Prepare data for regression
X = np.column_stack((x1, x2))

# Fit multiple linear regression
model = LinearRegression()
model.fit(X, y)

# Create a mesh for the plane
x1_mesh, x2_mesh = np.meshgrid(np.linspace(0, 10, 20), np.linspace(0, 10, 20))
X_mesh = np.column_stack((x1_mesh.ravel(), x2_mesh.ravel()))
y_mesh = model.predict(X_mesh).reshape(x1_mesh.shape)

# 3D visualization
fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')

# Plot data points
ax.scatter(x1, x2, y, color='blue', alpha=0.6, label='Data points')

# Plot regression plane
ax.plot_surface(x1_mesh, x2_mesh, y_mesh, alpha=0.3, color='red')

ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('Y')
ax.set_title('Multiple Linear Regression Plane')
plt.show()

print(f"Coefficients: {model.coef_}")
print(f"Intercept: {model.intercept_:.2f}")

Residual Analysis

Residual plots help validate regression assumptions. In a good model, residuals should be randomly scattered around zero with no clear pattern ?

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Generate sample data
np.random.seed(42)
x = np.linspace(1, 10, 50).reshape(-1, 1)
y = 2 * x.ravel() + 1 + np.random.normal(0, 1, 50)

# Fit model and calculate residuals
model = LinearRegression()
model.fit(x, y)
y_pred = model.predict(x)
residuals = y - y_pred

# Create residual plot
plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)
plt.scatter(x, y, color='blue', alpha=0.6, label='Data points')
plt.plot(x, y_pred, color='red', linewidth=2, label='Regression line')
plt.xlabel('X')
plt.ylabel('Y')
plt.title('Regression Line')
plt.legend()
plt.grid(True, alpha=0.3)

plt.subplot(1, 2, 2)
plt.scatter(y_pred, residuals, color='green', alpha=0.6)
plt.axhline(y=0, color='red', linestyle='--')
plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
plt.title('Residual Plot')
plt.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

print(f"Mean of residuals: {np.mean(residuals):.4f}")
print(f"Standard deviation of residuals: {np.std(residuals):.4f}")

Coefficient of Determination (R-squared)

R-squared measures how well the model explains the variance in the dependent variable. Values range from 0 to 1, where 1 indicates perfect fit ?

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

# Generate data with different noise levels
np.random.seed(42)
x = np.linspace(1, 10, 30).reshape(-1, 1)

# High correlation (low noise)
y_high = 2 * x.ravel() + 1 + np.random.normal(0, 0.5, 30)
model_high = LinearRegression().fit(x, y_high)
r2_high = r2_score(y_high, model_high.predict(x))

# Low correlation (high noise)
y_low = 2 * x.ravel() + 1 + np.random.normal(0, 3, 30)
model_low = LinearRegression().fit(x, y_low)
r2_low = r2_score(y_low, model_low.predict(x))

print("High Correlation Data:")
print(f"R-squared: {r2_high:.3f}")
print(f"This model explains {r2_high*100:.1f}% of the variance")

print("\nLow Correlation Data:")
print(f"R-squared: {r2_low:.3f}")
print(f"This model explains {r2_low*100:.1f}% of the variance")

High Correlation Data:
R-squared: 0.963
This model explains 96.3% of the variance

Low Correlation Data:
R-squared: 0.407
This model explains 40.7% of the variance

Key Mathematical Formulas

The essential formulas for understanding regression geometry:

• Slope: slope = ?(xy) - n?(x)?(y) / ?(x²) - n(?(x))²
• Intercept: intercept = ? - slope × x?
• R-squared: R² = 1 - (SS_res / SS_tot)

Practical Applications

Conclusion

The geometric interpretation of regression transforms abstract statistical relationships into visual, intuitive concepts. Understanding regression as lines in 2D space or planes in 3D space helps identify patterns, validate assumptions, and communicate results effectively across various fields and applications.