Torch - Optimizers and Loss Functions

Optimizers and loss functions in Torch are very difficult for training neural networks Common loss function measures specifies the Mean Squared Error(MSE) for the regression task that classifies the Cross-Entry Loss. Every component in the Torch works together in the training loop, where the optimizer re-updates the model parameters based on the gradients that complete the loss function.

Optimizers

Adams and Stochastic optimizers determine parameters to minimize the loss function, which measures the loss function, which measures the difference between the actual and predicted values.

• Stochastic Gradient Descent(SGC): This is an optimization algorithm that is used to minimize the loss function in machine learning models. It completes the gradient using the entire datasets, which updates the model parameters using only a single or a few training iterations.

• Adaptive Moment Estimation(Adam): It combines the other two advantages of extensions of stochastic gradient descent. This completes the adaptive learning rates for each parameter.

• RMSprop:The learning rate of an exponential decaying average divides the squared gradients.

Loss Function

Loss functions are essential in machine learning as they quantify the models predictions that specify the actual data. These measure the predicted output, and the true output returns the optimization process. Common loss function that includes the Mean Squared Error for regression tasks, which calculates the actual and predicted values.

• MSELoss(Mean Squared Error): This measures the average squared difference between the actual and predicted values, that are commonly used for regression tasks.

• CrossEntropyLoss: It combines the NULLoss and LogSoftmax in a single class. That is used for classification tasks. Each loss function is particularly effective for the multi-class problems, where this calculates the difference between the true distribution and predicted probability distribution.

• NULLoss(Negative Log Likelihood): This is used for classification problems where output is the probability distribution. The loss function is particularly effective while dealing with the multi-classification tasks.

Commonly Used Machines

Torch has been developed using an object-oriented paradiagram, which is implemented using the modifications in the existing algorithms or the design of the new algorithms. Following are the commonly used methods −

Gradient Machines

This important technique in machine learning is introduced by back-propogation algorithm. This is the application of simple gradient descent to complex the derivable functions. Torch determines the trained gradient descent function.

Mathematical representation is as follows −

$$\mathrm{f_{(w)}(x)=v_0+\sum^{N}{j=1}v_j\:tanh\:(v{j0}+\sum^{d}{i=1}u{ji}x^{i})}$$

This formulae optimizes the weight by updating the function iteratively for each example in the training set. This computes the derivation of the cost function. The text specifies the modularity of Torch by determining different gradient machines and cost functions to be implemented. This approach simplifies the complex model creation.

Support Vector Machines(SVM)

Support Vector Machines are very powerful machine learning algorithms those are widely used for classification tasks. These demonstrated good performance across different classification problems.

Mathematical representation is as follows −

$$\mathrm{k(x,x_{j})=exp(\:(−\gamma)\lVert x_{i}−x_{j}\lVert^{2})(\gamma \:\epsilon \:R)}$$

Training an SVM involves solving an optimization problem i.e., represented as −

$$\mathrm{y=sign (\sum^{T}_{i=1}y_i\:\alpha_i\:K(x,x_i)+b)}$$

Most commonly used Gaussian kernal −

$$\mathrm{Q(\alpha)=−\sum^{T}{i=1}\alpha{i}+\frac{1}{2}\sum^{T}{i=1}\sum^{T}{j=1}\alpha_i \alpha_j y_iy_jk(x_i x_j)}$$

Distributions

A Torch distribution is an object, such as Gaussian for instance that computes the probability, likelihood or density of a data set. The parameters in this distribution can be determined using various training algorithms such as Exception-Maximization or Viterbi algorithm. Torch distribution is a specified trained gradient machine to optimize different criterion's or we can create very complex machines.