Scikit Learn - Ridge Regression

Ridge regression or Tikhonov regularization is the regularization technique that performs L2 regularization. It modifies the loss function by adding the penalty (shrinkage quantity) equivalent to the square of the magnitude of coefficients.

$$\displaystyle\sum\limits_{j=1}^m\left(Y_{i}-W_{0}-\displaystyle\sum\limits_{i=1}^nW_{i}X_{ji} \right)^{2}+\alpha\displaystyle\sum\limits_{i=1}^nW_i^2=loss_{-}function+\alpha\displaystyle\sum\limits_{i=1}^nW_i^2$$

  • sklearn.linear_model.Ridge is the module used to solve a regression model where loss function is the linear least squares function and regularization is L2.

Parameters

Following table consists the parameters used by Ridge module −

Sr.No Parameter & Description
1

alpha − {float, array-like}, shape(n_targets)

Alpha is the tuning parameter that decides how much we want to penalize the model.
2

fit_intercept − Boolean

This parameter specifies that a constant (bias or intercept) should be added to the decision function. No intercept will be used in calculation, if it will set to false.
3

tol − float, optional, default=1e-4

It represents the precision of the solution.
4

normalize − Boolean, optional, default = False

If this parameter is set to True, the regressor X will be normalized before regression. The normalization will be done by subtracting the mean and dividing it by L2 norm. If fit_intercept = False, this parameter will be ignored.
5

copy_X − Boolean, optional, default = True

By default, it is true which means X will be copied. But if it is set to false, X may be overwritten.
6

max_iter − int, optional

As name suggest, it represents the maximum number of iterations taken for conjugate gradient solvers.
7

solver − str, {‘auto’, ‘svd’, ‘cholesky’, ‘lsqr’, ‘sparse_cg’, ‘sag’, ‘saga’}’

This parameter represents which solver to use in the computational routines. Following are the properties of options under this parameter

  • auto − It let choose the solver automatically based on the type of data.

  • svd − In order to calculate the Ridge coefficients, this parameter uses a Singular Value Decomposition of X.

  • cholesky − This parameter uses the standard scipy.linalg.solve() function to get a closed-form solution.

  • lsqr − It is the fastest and uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr.

  • sag − It uses iterative process and a Stochastic Average Gradient descent.

  • saga − It also uses iterative process and an improved Stochastic Average Gradient descent.
8

random_state − int, RandomState instance or None, optional, default = none

This parameter represents the seed of the pseudo random number generated which is used while shuffling the data. Following are the options −

  • int − In this case, random_state is the seed used by random number generator.

  • RandomState instance − In this case, random_state is the random number generator.

  • None − In this case, the random number generator is the RandonState instance used by np.random.

Attributes

Followings table consist the attributes used by Ridge module −

Sr.No Attributes & Description
1

coef_ − array, shape(n_features,) or (n_target, n_features)

This attribute provides the weight vectors.
2

Intercept_ − float | array, shape = (n_targets)

It represents the independent term in decision function.
3

n_iter_ − array or None, shape (n_targets)

Available for only ‘sag’ and ‘lsqr’ solver, returns the actual number of iterations for each target.

Implementation Example

Following Python script provides a simple example of implementing Ridge Regression. We are using 15 samples and 10 features. The value of alpha is 0.5 in our case. There are two methods namely fit() and score() used to fit this model and calculate the score respectively.

from sklearn.linear_model import Ridge
import numpy as np
n_samples, n_features = 15, 10
rng = np.random.RandomState(0)
y = rng.randn(n_samples)
X = rng.randn(n_samples, n_features)
rdg = Ridge(alpha = 0.5)
rdg.fit(X, y)
rdg.score(X,y)

Output

0.76294987

The output shows that the above Ridge Regression model gave the score of around 76 percent. For more accuracy, we can increase the number of samples and features.

Example

For the above example, we can get the weight vector with the help of following python script −

rdg.coef_

Output

array([ 0.32720254, -0.34503436, -0.2913278 , 0.2693125 , -0.22832508,
   -0.8635094 , -0.17079403, -0.36288055, -0.17241081, -0.43136046])

Example

Similarly, we can get the value of intercept with the help of following python script −

rdg.intercept_

Output

0.527486
Advertisements