Apache MXNet - NDArray

---

---

In this chapter, we will be discussing about MXNet’s multi-dimensional array format called ndarray.

Handling data with NDArray

First, we are going see how we can handle data with NDArray. Following are the prerequisites for the same −

Prerequisites

To understand how we can handle data with this multi-dimensional array format, we need to fulfil the following prerequisites:

• MXNet installed in a Python environment

• Python 2.7.x or Python 3.x

Implementation Example

Let us understand the basic functionality with the help of an example given below −

First, we need to import MXNet and ndarray from MXNet as follows −

import mxnet as mx
from mxnet import nd

Once we import the necessary libraries, we will go with the following basic functionalities:

A simple 1-D array with a python list

Example

x = nd.array([1,2,3,4,5,6,7,8,9,10])
print(x)

Output

The output is as mentioned below −

[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]
<NDArray 10 @cpu(0)>

A 2-D array with a python list

Example

y = nd.array([[1,2,3,4,5,6,7,8,9,10], [1,2,3,4,5,6,7,8,9,10], [1,2,3,4,5,6,7,8,9,10]])
print(y)

Output

The output is as stated below −

[[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]
[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]
[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]]
<NDArray 3x10 @cpu(0)>

Creating an NDArray without any initialisation

Here, we will create a matrix with 3 rows and 4 columns by using .empty function. We will also use .full function, which will take an additional operator for what value you want to fill in the array.

Example

x = nd.empty((3, 4))
print(x)
x = nd.full((3,4), 8)
print(x)

Output

The output is given below −

[[0.000e+00 0.000e+00 0.000e+00 0.000e+00]
 [0.000e+00 0.000e+00 2.887e-42 0.000e+00]
 [0.000e+00 0.000e+00 0.000e+00 0.000e+00]]
<NDArray 3x4 @cpu(0)>

[[8. 8. 8. 8.]
 [8. 8. 8. 8.]
 [8. 8. 8. 8.]]
<NDArray 3x4 @cpu(0)>

Matrix of all zeros with the .zeros function

Example

x = nd.zeros((3, 8))
print(x)

Output

The output is as follows −

[[0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0.]]
<NDArray 3x8 @cpu(0)>

Matrix of all ones with the .ones function

Example

x = nd.ones((3, 8))
print(x)

Output

The output is mentioned below −

[[1. 1. 1. 1. 1. 1. 1. 1.]
   [1. 1. 1. 1. 1. 1. 1. 1.]
   [1. 1. 1. 1. 1. 1. 1. 1.]]
<NDArray 3x8 @cpu(0)>

Creating array whose values are sampled randomly

Example

y = nd.random_normal(0, 1, shape=(3, 4))
print(y)

Output

The output is given below −

[[ 1.2673576 -2.0345826 -0.32537818 -1.4583491 ]
 [-0.11176403 1.3606371 -0.7889914 -0.17639421]
 [-0.2532185 -0.42614475 -0.12548696 1.4022992 ]]
<NDArray 3x4 @cpu(0)>

Finding dimension of each NDArray

Example

y.shape

Output

The output is as follows −

(3, 4)

Finding the size of each NDArray

Example

y.size

Output

12

Finding the datatype of each NDArray

Example

y.dtype

Output

numpy.float32

NDArray Operations

In this section, we will introduce you to MXNet’s array operations. NDArray support large number of standard mathematical as well as In-place operations.

Standard Mathematical Operations

Following are standard mathematical operations supported by NDArray −

Element-wise addition

First, we need to import MXNet and ndarray from MXNet as follows:

import mxnet as mx
from mxnet import nd
x = nd.ones((3, 5))
y = nd.random_normal(0, 1, shape=(3, 5))
print('x=', x)
print('y=', y)
x = x + y
print('x = x + y, x=', x)

Output

The output is given herewith −

x=
[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]
<NDArray 3x5 @cpu(0)>
y=
[[-1.0554522 -1.3118273 -0.14674698 0.641493 -0.73820823]
[ 2.031364 0.5932667 0.10228804 1.179526 -0.5444829 ]
[-0.34249446 1.1086396 1.2756858 -1.8332436 -0.5289873 ]]
<NDArray 3x5 @cpu(0)>
x = x + y, x=
[[-0.05545223 -0.3118273 0.853253 1.6414931 0.26179177]
[ 3.031364 1.5932667 1.102288 2.1795259 0.4555171 ]
[ 0.6575055 2.1086397 2.2756858 -0.8332436 0.4710127 ]]
<NDArray 3x5 @cpu(0)>

Element-wise multiplication

Example

x = nd.array([1, 2, 3, 4])
y = nd.array([2, 2, 2, 1])
x * y

Output

You will see the following output−

[2. 4. 6. 4.]
<NDArray 4 @cpu(0)>

Exponentiation

Example

nd.exp(x)

Output

When you run the code, you will see the following output:

[ 2.7182817 7.389056 20.085537 54.59815 ]
<NDArray 4 @cpu(0)>

Matrix transpose to compute matrix-matrix product

Example

nd.dot(x, y.T)

Output

Given below is the output of the code −

[16.]
<NDArray 1 @cpu(0)>

In-place Operations

Every time, in the above example, we ran an operation, we allocated a new memory to host its result.

For example, if we write A = A+B, we will dereference the matrix that A used to point to and instead point it at the newly allocated memory. Let us understand it with the example given below, using Python’s id() function −

print('y=', y)
print('id(y):', id(y))
y = y + x
print('after y=y+x, y=', y)
print('id(y):', id(y))

Output

Upon execution, you will receive the following output −

y=
[2. 2. 2. 1.]
<NDArray 4 @cpu(0)>
id(y): 2438905634376
after y=y+x, y=
[3. 4. 5. 5.]
<NDArray 4 @cpu(0)>
id(y): 2438905685664

In fact, we can also assign the result to a previously allocated array as follows −

print('x=', x)
z = nd.zeros_like(x)
print('z is zeros_like x, z=', z)
print('id(z):', id(z))
print('y=', y)
z[:] = x + y
print('z[:] = x + y, z=', z)
print('id(z) is the same as before:', id(z))

Output

The output is shown below −

x=
[1. 2. 3. 4.]
<NDArray 4 @cpu(0)>
z is zeros_like x, z=
[0. 0. 0. 0.]
<NDArray 4 @cpu(0)>
id(z): 2438905790760
y=
[3. 4. 5. 5.]
<NDArray 4 @cpu(0)>
z[:] = x + y, z=
[4. 6. 8. 9.]
<NDArray 4 @cpu(0)>
id(z) is the same as before: 2438905790760

From the above output, we can see that x+y will still allocate a temporary buffer to store the result before copying it to z. So now, we can perform operations in-place to make better use of memory and to avoid temporary buffer. To do this, we will specify the out keyword argument every operator support as follows −

print('x=', x, 'is in id(x):', id(x))
print('y=', y, 'is in id(y):', id(y))
print('z=', z, 'is in id(z):', id(z))
nd.elemwise_add(x, y, out=z)
print('after nd.elemwise_add(x, y, out=z), x=', x, 'is in id(x):', id(x))
print('after nd.elemwise_add(x, y, out=z), y=', y, 'is in id(y):', id(y))
print('after nd.elemwise_add(x, y, out=z), z=', z, 'is in id(z):', id(z))

Output

On executing the above program, you will get the following result −

x=
[1. 2. 3. 4.]
<NDArray 4 @cpu(0)> is in id(x): 2438905791152
y=
[3. 4. 5. 5.]
<NDArray 4 @cpu(0)> is in id(y): 2438905685664
z=
[4. 6. 8. 9.]
<NDArray 4 @cpu(0)> is in id(z): 2438905790760
after nd.elemwise_add(x, y, out=z), x=
[1. 2. 3. 4.]
<NDArray 4 @cpu(0)> is in id(x): 2438905791152
after nd.elemwise_add(x, y, out=z), y=
[3. 4. 5. 5.]
<NDArray 4 @cpu(0)> is in id(y): 2438905685664
after nd.elemwise_add(x, y, out=z), z=
[4. 6. 8. 9.]
<NDArray 4 @cpu(0)> is in id(z): 2438905790760

NDArray Contexts

In Apache MXNet, each array has a context and one context could be the CPU, whereas other contexts might be several GPUs. The things can get even worst, when we deploy the work across multiple servers. That’s why, we need to assign arrays to contexts intelligently. It will minimise the time spent transferring data between devices.

For example, try initialising an array as follows −

from mxnet import nd
z = nd.ones(shape=(3,3), ctx=mx.cpu(0))
print(z)

Output

When you execute the above code, you should see the following output −

[[1. 1. 1.]
 [1. 1. 1.]
 [1. 1. 1.]]
<NDArray 3x3 @cpu(0)>

We can copy the given NDArray from one context to another context by using the copyto() method as follows −

x_gpu = x.copyto(gpu(0))
print(x_gpu)

NumPy array vs. NDArray

We all the familiar with NumPy arrays but Apache MXNet offers its own array implementation named NDArray. Actually, it was initially designed to be similar to NumPy but there is a key difference −

The key difference is in the way calculations are executed in NumPy and NDArray. Every NDArray manipulation in MXNet is done in asynchronous and non-blocking way, which means that, when we write code like c = a * b, the function is pushed to the Execution Engine, which will start the calculation.

Here, a and b both are NDArrays. The benefit of using it is that, the function immediately returns back, and the user thread can continue execution despite the fact that the previous calculation may not have been completed yet.

Working of Execution Engine

If we talk about the working of execution engine, it builds the computation graph. The computation graph may reorder or combine some calculations, but it always honors dependency order.

For example, if there are other manipulation with ‘X’ done later in the programming code, the Execution Engine will start doing them once the result of ‘X’ is available. Execution engine will handle some important works for the users, such as writing of callbacks to start execution of subsequent code.

In Apache MXNet, with the help of NDArray, to get the result of computation we only need to access the resulting variable. The flow of the code will be blocked until the computation results are assigned to the resulting variable. In this way, it increases code performance while still supporting imperative programming mode.

Converting NDArray to NumPy Array

Let us learn how can we convert NDArray to NumPy Array in MXNet.

Combining higher-level operator with the help of few lower-level operators

Sometimes, we can assemble a higher-level operator by using the existing operators. One of the best examples of this is, the np.full_like() operator, which is not there in NDArray API. It can easily be replaced with a combination of existing operators as follows:

from mxnet import nd
import numpy as np
np_x = np.full_like(a=np.arange(7, dtype=int), fill_value=15)
nd_x = nd.ones(shape=(7,)) * 15
np.array_equal(np_x, nd_x.asnumpy())

Output

We will get the output similar as follows −

True

Finding similar operator with different name and/or signature

Among all the operators, some of them have slightly different name, but they are similar in the terms of functionality. An example of this is nd.ravel_index() with np.ravel() functions. In the same way, some operators may have similar names, but they have different signatures. An example of this is np.split() and nd.split() are similar.

Let’s understand it with the following programming example:

def pad_array123(data, max_length):
data_expanded = data.reshape(1, 1, 1, data.shape[0])
data_padded = nd.pad(data_expanded,
mode='constant',
pad_width=[0, 0, 0, 0, 0, 0, 0, max_length - data.shape[0]],
constant_value=0)
data_reshaped_back = data_padded.reshape(max_length)
return data_reshaped_back
pad_array123(nd.array([1, 2, 3]), max_length=10)

Output

The output is stated below −

[1. 2. 3. 0. 0. 0. 0. 0. 0. 0.]
<NDArray 10 @cpu(0)>

Minimising impact of blocking calls

In some of the cases, we have to use either .asnumpy() or .asscalar() methods, but this will force MXNet to block the execution, until the result can be retrieved. We can minimise the impact of a blocking call by calling .asnumpy() or .asscalar() methods in the moment, when we think the calculation of this value is already done.

Implementation Example

Example

from __future__ import print_function
import mxnet as mx
from mxnet import gluon, nd, autograd
from mxnet.ndarray import NDArray
from mxnet.gluon import HybridBlock
import numpy as np

class LossBuffer(object):
   """
   Simple buffer for storing loss value
   """
   
   def __init__(self):
      self._loss = None

   def new_loss(self, loss):
      ret = self._loss
      self._loss = loss
      return ret

      @property
      def loss(self):
         return self._loss

net = gluon.nn.Dense(10)
ce = gluon.loss.SoftmaxCELoss()
net.initialize()
data = nd.random.uniform(shape=(1024, 100))
label = nd.array(np.random.randint(0, 10, (1024,)), dtype='int32')
train_dataset = gluon.data.ArrayDataset(data, label)
train_data = gluon.data.DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=2)
trainer = gluon.Trainer(net.collect_params(), optimizer='sgd')
loss_buffer = LossBuffer()
for data, label in train_data:
   with autograd.record():
      out = net(data)
      # This call saves new loss and returns previous loss
      prev_loss = loss_buffer.new_loss(ce(out, label))
   loss_buffer.loss.backward()
   trainer.step(data.shape[0])
   if prev_loss is not None:
      print("Loss: {}".format(np.mean(prev_loss.asnumpy())))

Output

The output is cited below:

Loss: 2.3373236656188965
Loss: 2.3656985759735107
Loss: 2.3613128662109375
Loss: 2.3197104930877686
Loss: 2.3054862022399902
Loss: 2.329197406768799
Loss: 2.318927526473999