Torch - Recurrent Neural Networks

Quiz

A Recurrent Neural Network(RNN) in Torch is specified to handle sequential data by classifying a hidden state that captures information from the previous inputs. In Torch we can use the torch.nn.RNN module to create a RNNs. This helps us to understand the input size, number of layers, non-linearity and hidden size. RNNs are specified for tasks like natural language processing and series prediction where the sequence of data is crucial. RNN can handle variable-length sequence as input and provide variable-length sequence as output.

Neural Networks Handling Vectors and Sequences

In Torch, neural networks are built using the torch.nn module. This provides a flexible and efficient way to build and train neural networks.

Vectors: A vector is a traditional neural network, like feed forward networks, that handles the fixed size of the input vectors. Each layer in the vector transforms the input vector into another vector from a series of linear and non-linear operations.
Sequences: RNNs are specified to handle sequential data, where the data is in the order of data points that matters. These maintain a hidden state that evolves over time that captures the information from the other inputs.

The code below allows the RNN to handle sequences and captures temporal dependencies, making it suitable for various tasks. Implementing RNNs in Torch −

import torch
import torch.nn as nn

class SimpleRNN(nn.Module):
    def __init__(self,int_size, hdn_size, ott_size):
        super(SimpleRNN, self).__init__()
        self.rnn = nn.RNN(int_size, hdn_size, batch_first=True)
        self.fc = nn.Linear(hdn_size, ott_size)

    def forward(self, x):
        h0 = torch.zeros(1, x.size(0), hdn_size)  
        out, _ = self.rnn(x, h0)
        out = self.fc(out[:, -1, :])  
        return out

int_size = 20  #input
hdn_size = 30  #hidden
ott_size = 2   #output

model = SimpleRNN(int_size, hdn_size, ott_size)

y = torch.randn(2, 4, input_size)

output = model(y)
print(output)

Forward and Backward Pass in Neural Network

In neural network training, the forward pass involves feeding input data through the network to complete the predictions and loss. The backward pass calculates the gradients of the loss with respect to a model parameters using backpropagation. These gradients are even used by the optimizer to update the model parameters that optimizes the loss over iterations.

Forward Pass

In forward pass, input data is fed through the neural network layer by layer. Each layer applies transformation to produce an output. The final layer generates predictions, which are compared to the true labels using a loss function to complete the error

Input Data: The data is fed into the neural networks.
Layer-by-Layer Computation: The data passes through each layer of the network sequentially.
Output and Loss Calculation: Final layer produces the network's output and loss function compares the predictions of the true labels and completes the loss, which quantifies the error.

Backward Pass

In backward pass, the loss function is propagated back through the network to complete gradients of the loss with respect to each parameter. This is done using backpropagation. These gradients are used by the optimizer to update the model parameters, that reduces the loss in subsequent iteration.

# Compute gradients
loss.backward() 
# Updates the parameters
optimizer.step() 
# Zero gradients
optimizer.zero_grad()

nngraph Package

The nngraph package extends the capabilities of the nn pachakge in Torch by allowing users to define neural networks as computational graphs. This approach provides greater clarity and flexibility for the complex architecture like recurrent and convolutional neural networks. Each nn module is represented as a node in a graph. The visualization package supports the computational graph using tools like grapgviz. This helps us to understand and debug the network structure. It allows the flexibility for the creation of networks with multiple inputs and outputs. This supports complex architectures that are difficult to implement using sequential models.

To use nngraph we need to install it along with graphviz for visualization −

brew install graphviz # Mac users

sudo apt-get install graphviz -yellow #Ubuntu users

Print Page