How can Keras be used to implement ensembling in Python?

Ensemble learning combines multiple models to create a stronger predictor than any individual model. In Keras, we can implement ensembling using the Functional API to create models that average predictions from multiple sub-models.

What is Ensembling?

Ensembling is a machine learning technique where multiple models are trained independently and their predictions are combined (usually averaged) to make final predictions. This approach often reduces overfitting and improves model performance.

Setting Up Keras

Keras is included with TensorFlow and can be imported directly ?

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

Creating Individual Models

First, let's create a function that returns a simple neural network model ?

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

def get_model():
    inputs = keras.Input(shape=(128,))
    outputs = layers.Dense(1)(inputs)
    return keras.Model(inputs, outputs)

print("Creating individual models...")
model_1 = get_model()
model_2 = get_model()
model_3 = get_model()
print("Three models created successfully!")

Creating individual models...
Three models created successfully!

Building the Ensemble Model

Now we combine the three models into a single ensemble that averages their predictions ?

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

def get_model():
    inputs = keras.Input(shape=(128,))
    outputs = layers.Dense(1)(inputs)
    return keras.Model(inputs, outputs)

# Create individual models
model_1 = get_model()
model_2 = get_model()
model_3 = get_model()

# Create ensemble
my_inputs = keras.Input(shape=(128,))
y1 = model_1(my_inputs)
y2 = model_2(my_inputs)
y3 = model_3(my_inputs)

# Average the predictions
my_outputs = layers.average([y1, y2, y3])

# Create the ensemble model
ensemble_model = keras.Model(inputs=my_inputs, outputs=my_outputs)

print("Ensemble model created successfully!")
print(f"Ensemble model summary:")
ensemble_model.summary()

Ensemble model created successfully!
Ensemble model summary:
Model: "model_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_4 (InputLayer)        [(None, 128)]             0         
                                                                 
 model (Functional)          (None, 1)                 129       
                                                                 
 model_1 (Functional)        (None, 1)                 129       
                                                                 
 model_2 (Functional)        (None, 1)                 129       
                                                                 
 average (Average)           (None, 1)                 0         
                                                                 
=================================================================
Total params: 387
Trainable params: 387
Non-trainable params: 0
_________________________________________________________________

How the Ensemble Works

The ensemble model works by:

• Taking the same input and feeding it to all three sub-models
• Each sub-model produces its own prediction
• The layers.average() function combines these predictions by taking their mean
• The final output is the averaged prediction

Benefits of Ensembling

Training the Ensemble

You can train the ensemble model just like any other Keras model ?

# Compile the ensemble model
ensemble_model.compile(optimizer='adam', 
                      loss='mse', 
                      metrics=['mae'])

# Train the model (example with dummy data)
# ensemble_model.fit(X_train, y_train, epochs=10, validation_split=0.2)

Conclusion

Keras makes implementing ensemble models straightforward using the Functional API. By combining multiple models and averaging their predictions, ensembles often achieve better performance and robustness than individual models.