How can functional API be used to work with residual connections in Python?

KerasPythonServer Side ProgrammingProgramming

Keras is present within the Tensorflow package. It can be accessed using the below line of code.

import tensorflow
from tensorflow import keras

The Keras functional API helps create models that are more flexible in comparison to models created using sequential API. The functional API can work with models that have non-linear topology, can share layers and work with multiple inputs and outputs. A deep learning model is usually a directed acyclic graph (DAG) that contains multiple layers. The functional API helps build the graph of layers.

We are using the Google Colaboratory to run the below code. Google Colab or Colaboratory helps run Python code over the browser and requires zero configuration and free access to GPUs (Graphical Processing Units). Colaboratory has been built on top of Jupyter Notebook. Following is the code snippet;

Example

print("Toy ResNet model for CIFAR10")
print("Layers generated for model")
inputs = keras.Input(shape=(32, 32, 3), name="img")
x = layers.Conv2D(32, 3, activation="relu")(inputs)
x = layers.Conv2D(64, 3, activation="relu")(x)
block_1_output = layers.MaxPooling2D(3)(x)

x = layers.Conv2D(64, 3, activation="relu", padding="same")(block_1_output)
x = layers.Conv2D(64, 3, activation="relu", padding="same")(x)
block_2_output = layers.add([x, block_1_output])

x = layers.Conv2D(64, 3, activation="relu", padding="same")(block_2_output)
x = layers.Conv2D(64, 3, activation="relu", padding="same")(x)
block_3_output = layers.add([x, block_2_output])

x = layers.Conv2D(64, 3, activation="relu")(block_3_output)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(256, activation="relu")(x)
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(10)(x)

model = keras.Model(inputs, outputs, name="toy_resnet")
print("More information about the model")
model.summary()

Code credit − https://www.tensorflow.org/guide/keras/functional

Output

Toy ResNet model for CIFAR10
Layers generated for model
More information about the model
Model: "toy_resnet"
________________________________________________________________________________
__________________
Layer (type)          Output Shape          Param #       Connected to
================================================================================
==================
img (InputLayer)       [(None, 32, 32, 3)]    0
________________________________________________________________________________
__________________
conv2d_32 (Conv2D)    (None, 30, 30, 32)     896          img[0][0]
________________________________________________________________________________
__________________
conv2d_33 (Conv2D)    (None, 28, 28, 64)    18496         conv2d_32[0][0]
________________________________________________________________________________
__________________
max_pooling2d_8 (MaxPooling2D) (None, 9, 9, 64) 0          conv2d_33[0][0]
________________________________________________________________________________
__________________
conv2d_34 (Conv2D)       (None, 9, 9, 64)       36928       max_pooling2d_8[0][0]
________________________________________________________________________________
__________________
conv2d_35 (Conv2D)       (None, 9, 9, 64)       36928       conv2d_34[0][0]
________________________________________________________________________________
__________________
add_12 (Add)             (None, 9, 9, 64)          0       conv2d_35[0][0]
                                          max_pooling2d_8[0][0]
________________________________________________________________________________
__________________
conv2d_36 (Conv2D)          (None, 9, 9, 64)    36928       add_12[0][0]
________________________________________________________________________________
__________________
conv2d_37 (Conv2D)          (None, 9, 9, 64)    36928       conv2d_36[0][0]
________________________________________________________________________________
__________________
add_13 (Add)                (None, 9, 9, 64)       0       conv2d_37[0][0]
                                       add_12[0][0]
________________________________________________________________________________
__________________
conv2d_38 (Conv2D)          (None, 7, 7, 64)    36928       add_13[0][0]
________________________________________________________________________________
__________________
global_average_pooling2d_1    (Glo (None, 64)      0       conv2d_38[0][0]
________________________________________________________________________________
__________________
dense_40 (Dense)             (None, 256)          16640    global_average_pooling2d_1[0][0]
________________________________________________________________________________
__________________
dropout_2 (Dropout)          (None, 256)          0          dense_40[0][0]
________________________________________________________________________________
__________________
dense_41 (Dense)             (None, 10)          2570       dropout_2[0][0]
================================================================================
==================
Total params: 223,242
Trainable params: 223,242
Non-trainable params: 0
________________________________________________________________________________
__________________

Explanation

  • The model has multiple inputs and outputs.

  • The functional API makes it easy to work with non-linear connectivity topologies.

  • This model with layers is not connected sequentially, hence the ‘Sequential’ API can’t work with it.

  • This is where residual connections come into the picture.

  • A sample ResNet model using CIFAR10 is built to demonstrate the same.

raja
Published on 18-Jan-2021 11:44:48
Advertisements