How to perform in-place operations in PyTorch?


In-place operations directly change the content of a tensor without making a copy of it. Since it does not create a copy of the input, it reduces the memory usage when dealing with high-dimensional data. An in-place operation helps to utilize less GPU memory.

In PyTorch, in-place operations are always post-fixed with a "_", like add_(), mul_(), etc.

Steps

To perform an in-place operation, one could follow the steps given below −

  • Import the required library. The required library is torch.

  • Define/create tensors on which in-place operation is to be performed.

  • Perform both normal and in-place operations to see the clear difference between them.

  • Display the tensors obtained in normal and in-place operations.

Example 1

The following Python program highlights the difference between a normal addition and an in-place addition. In in-place addition, the value of the first operand "x" is changed; while in normal addition, it remains unchanged.

# import required library
import torch

# create two tensors x and y
x = torch.tensor(4)
y = torch.tensor(3)
print("x=", x.item())
print("y=", y.item())

# Normal addition
z = x.add(y)
print("Normal Addition x:",x.item())

# In-place addition
z = x.add_(y)
print("In-place Addition x:",x.item())

Output

x = 4
y = 3
Normal Addition x: 4
In-place Addition x: 7

In the above program, two tensors x and y are added. In normal addition operation, the value of x is not changed, but in in-place addition operation, it's changed.

Example 2

The following Python program shows how the normal addition and in-place addition operations are different in terms of memory allocation.

# import required library
import torch

# create two tensors x and y
x = torch.tensor(4)
y = torch.tensor(3)
print("id(x)=", id(x))

# Normal addition
z = x.add(y)
print("Normal Addition id(z):",id(z))

# In-place addition
z = x.add_(y)
print("In-place Addition id(z):",id(z))

Output

id(x)= 63366656
Normal Addition id(z): 63366080
In-place Addition id(z): 63366656

In the above program, normal operation allocates new memory location for "z", whereas in-place operation does not allocate new memory.

Updated on: 06-Dec-2021

1K+ Views

Kickstart Your Career

Get certified by completing the course

Get Started
Advertisements