Tensor Operations on PyTorch

PyTorch, an open-source machine learning framework, is widely used in academic research and industries. It offers ample tools and libraries to build and train neural networks effectively.

Tensors in PyTorch

In PyTorch, a tensor is a multidimensional array of values that can be used to represent data for machine learning models. Dimensions may vary as 1D, 2D, 3D, and so on, totally depending on the complexity of the data they represent.

Let's take an example, a 1D tensor can be used to represent a sequence of values, such as time-series data, whereas a 2D tensor can be used to represent an image, in which each element of the tensor corresponds to a pixel in the image.

Tensor Operations

Tensor operations are mathematical operations that can be performed on tensors to manipulate and transform their values. PyTorch provides a wide range of tensor operations that can be used to perform basic operations like arithmetic, statistical, and logical operations on tensors.

These operations are implemented as functions that take one or more tensors as input and return a new tensor as output.

Example

In PyTorch, we use the following example explains creating a tensor, it will take a list or tuple of values as input.

import torch
# create a 2D tensor using a nested list
a = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]])
# print the tensor
print(a)

Output

tensor([[1, 2, 3, 4],
         [5, 6, 7, 8]])

Addition

Tensor addition can be performed using the torch.add() function or the + operator.

Example

import torch
# create a 2D tensor using a nested list
a = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]])
b = torch.add(a, 5)
c = a + b
print(b)
print(c)

Output

tensor([[6, 7, 8, 9],
        [10, 11, 12, 13]])

Subtraction

Tensor subtraction can be performed using the torch.sub() function or the - operator.

Example

# subtract a scalar value of 2 from each element of the tensor to create a new tensor
d = torch.sub(a, 2)
# subtract the modified tensor from the original tensor to create a new tensor
e = a - d
print(d)
print(e)

Output

tensor([[-1, 0, 1, 2],  [3, 4, 5, 6]])
tensor([[2, 2, 2, 2],  [2, 2, 2, 2]])

Multiplication

Tensor multiplication can be performed using the torch.mul() function or the * operator.

Example

# multiply each element of the tensor by a scalar value of 2 to create a new tensor
f = torch.mul(a, 2)
g = a * f
print(f)
print(g)

Output

tensor([[2, 4, 6, 8],
        [10, 12, 14, 16]])
tensor([[2, 8, 18, 32],
        [50, 72, 98,128]])

Division

Tensor division can be performed using the torch.div() function or the / operator.

Example

# divide each element of the tensor by a scalar value of 2 to create a new tensor
h = torch.div(a, 2)

# divide the original tensor element-wise by the modified tensor to create a new tensor
i = a / h
print(h)
print(i)

Output

tensor([[0.5000, 1.0000, 1.5000, 2.0000],
        [2.5000, 3.0000, 3.5000, 4.0000]])
tensor([[2., 2., 2., 2.],
        [2., 2., 2., 2.]])

Advanced Tensor Operations

Advanced tensor operations include matrix multiplication, transposition, reshaping, and concatenation that basically deals with 2D tensors.

Matrix Multiplication

We can perform Matrix multiplication using the torch.mm() function or the @ operator.

Example

# create two 2D tensors
A = torch.tensor([[1, 2], [3, 4]])
B = torch.tensor([[5, 6], [7, 8]])

# perform matrix multiplication using torch.mm() function
C = torch.mm(A, B)

# we can use the @ operator for matrix multiplication
D = A @ B
print(C)
print(D)

Output

tensor([[19, 22],
        [43, 50]])
tensor([[19, 22],
        [43, 50]])

Transposition

Transposition in tensor operations is the process of flipping the axes of a tensor. It involves exchanging the rows and columns of a 2D tensor or more generally, the axes of a tensor of any dimension.

We can perform Transposition using the torch.t() function.

Example

# create a 2D tensor
E = torch.tensor([[1, 2], [3, 4], [5, 6]])

# transpose the tensor using torch.t() function
F = torch.t(E)
print(E)
print(F)

Output

tensor([[1, 2],
        [3, 4],
        [5, 6]])
tensor([[1, 3, 5],
        [2, 4, 6]])

Reshaping

Reshaping in tensor operations is the process of changing the shape or dimensions of a tensor while preserving its underlying data. It involves rearranging the elements of a tensor to fit a new shape, without changing the total number of elements.

We can perform Reshaping using the torch.reshape() function or the .view() method.

Example

# create a 2D tensor
G = torch.tensor([[1, 2, 3], [4, 5, 6]])

# reshape the tensor using torch.reshape() function
H = torch.reshape(G, (3, 2))

# reshape the tensor using the .view() method
I = G.view(3, 2)

Output

tensor([[1, 2, 3],
        [4, 5, 6]])
tensor([[1, 2],
        [3, 4],
        [5, 6]])
tensor([[1, 2],
        [3, 4],
        [5, 6]])

Concatenation

Concatenation in tensor operations is the process of joining two or more tensors along a specific dimension to form a larger tensor. The resulting tensor has a new dimension that is the concatenation of the original dimensions of the input tensors.

It can be performed using the torch.cat() function.

Example

# create two tensors
J = torch.tensor([[1, 2], [3, 4]])
K = torch.tensor([[5, 6]])

# concatenate the tensors along the first axis
L = torch.cat((J, K), dim=0)
print(J)
print(K)
print(L)

Output

tensor([[1, 2],
        [3, 4]])
tensor([[5, 6]])
tensor([[1, 2],
        [3, 4],
       [5, 6]])

Conclusion

PyTorch use areas include computer vision, natural language processing, speech recognition, recommendation systems, and autonomous vehicles.

Facebook AI Research (FAIR) uses PyTorch extensively for the research and development of state-of-the-art deep learning models. NVIDIA, Tesla and IBM also use PyTorch for developing autonomous driving systems, in various computer vision applications, and also in natural language processing models.