Limitations of fixed basis function

Introduction

Fixed basis functions are functions that help us to extend linear models in Machine Learning, by taking linear combinations of nonlinear functions. Since Linear models depend on the linear combination of parameters, they suffer a significant limitation.

The radial function thus helps model such a group of models by utilizing non-linearity in the data while keeping the parameters linear.

Different linear combinations of the fixed basis functions are used within the linear regression by creating complex functions.

In this article let us look into the fixed basis function and its limitations

Fixed Basis function

A linear regression model can be represented with the below equation,

$$\mathrm{y(x\:,\:y)\:=\:w^{T}\:x}$$

The above model represents a linear relationship between the parameters and input features.

It cannot accept nonlinear relationships. To make the above model behave like a non-linear model by using basis functions or a combination of basis functions in place of the input variables. The overall model still retails its linear nature as depicted below.

$$\mathrm{y(x\:,\:y)\:=\:w_{o}\theta_{0}(x)\:+\:w_{1}\theta_{1}(x)\:+\:w_{2}\theta_{2}(x)\:......\:w_{N-1}\theta_{N-1}(x)}$$

$$=\mathrm{\displaystyle\sum\limits_{n=0}^{N-1}w_{n}\theta_{n}(x)}$$

$$=\mathrm{w^{T}\theta(x)}$$

Therefore, non-linear relationships of x can be used by using nonlinear fixed basis functions.

Limitations of Fixed Basis function

Fixed basis functions generally transform the inputs into higher dimensions. Some of their limitations are.

• The existing overlap between two classes cannot be removed by fixed basis functions, however, they can increase the overlap

• Sometimes fixed basis functions are not suitable for a task because it is difficult to tell whether these functions are best fit for the model.

• These functions may lead to overfitting since we may want to use fixed functions and we may try a lot of them.

Example

Code Implementation of Linear Fixed basis function

## fixed basis

import numpy as np

def gb(x_arr, mue, gam=1):
    return np.exp(-gam * np.linalg.norm(mue-x_arr)**2)

x_arr = np.array([-0.5, -1, -0.5, -0.3, -0.1, 0, 0.4, 0.3, 0.6, 0.7, 1])
t_arr = np.array([-3.9, -3.6, -2.0, 0.8, 0.5, -1.9, -1.6, 0.7, 2.2, 2.8, 6.6])
N = 5
l = 0.0001

# phi matrix
phi = np.ones((t_arr.shape[0], N))
for n in range(N-1):
    mue = n/N
    phi[:, n+1] = np.vectorize(gb)(x_arr, mue)

# alpha & w calculation
W = np.linalg.inv(l * np.identity(N) + phi.T @ phi) @ phi.T @ t_arr
alp = sum((t_arr - phi @ W)**2) / len(t_arr)
print(alp)

Output

2.090565904553811

Conclusion

Fixed basis functions are a popular choice while modeling linear equations that use a combination of non-linear basis functions to model a particular task. However, there are a few limitations associated with the usage of fixed-basis functions that they are not useful in cases where there is class overlap and they do sometimes have problems of overfitting because sometimes we may try a lot of them at once.