LightGBM - Tree Growth Strategy

Quiz

The gradient boosting framework LightGBM uses "leaf-wise" development, a new approach to tree growth. When it comes to machine learning and data analysis, decision trees are beneficial for both classification and regression. LightGBM's distinguishing characteristic is its leaf-wise tree development method, which differs from the standard level-wise tree growth technique utilized by the majority of decision tree learning algorithms.

In this chapter, we will see the leaf-wise tree growth method, level-wise tree growth method and as well as clarify how it is different from the typical level-wise tree development strategy used by most decision tree learning algorithms.

LightGBM is made to effectively train large data sets and create highly precise prediction models. Let's define a few key terms before we go into level-wise and leaf-wise tree growth:

Gradient Boosting: A machine learning method that builds a powerful predictive model by combining several weak models, usually decision trees.
Decision Tree: A model like a tree that gives labels at the leaf nodes and makes decisions at each inside node based on features.
LightGBM: A gradient boosting framework called LightGBM was developed by Microsoft to efficiently train decision trees for a variety of machine learning uses.

Traditional Level-Wise Tree Growth

Understanding the traditional level-wise tree growth method, which is used in decision trees and different gradient-boosting frameworks, is necessary for leaf-wise development. The tree divides horizontally at each level as it grows deeper, forming a wider, shallower tree. The most popular way for developing decision trees in gradient boosting is the level-wise tree building methodology.

Before going to the next level of the tree, it grows each node at the same level, or depth. The ideal feature and limit that optimizes the goal function divide the root node into two child nodes. After that, the same process is done for every child node until an initial criterion-like a maximum depth, a minimum amount of samples, or a minimal improvement-is satisfied. The level-wise tree development technique makes sure the tree is balanced and that each leaf is the same depth.

Leaf-wise Tree Growth

The Leaf-wise Tree Growth Method is a different method for creating decision trees in gradient boosting.

It works by growing the leaf node that has the most split return among all the leaves on the tree. The root node is split into two child nodes by choosing the feature and level that maximizes the objective function. Next, by selecting one of the child nodes as the next leaf to split, the process is repeated until a stopping criterion (like the maximum depth, maximum number of leaves, or minimal improvement) is reached. The tree may not be balanced or have leaves that are all the same depth even with a leaf-wise tree development strategy.

LightGBM uses leaf-wise growth as compared to depth-wise growth. With this method, a deeper and smaller tree is produced since the algorithm selects the leaf node that offers the greatest loss function decrease. This strategy can result in a more accurate model with fewer nodes than depth-wise growth.

How Leaf-wise Growth Works ?

Let us discuss the leaf-wise growth strategy in in depth −

In LightGBM, all of the training data is originally kept on a single root node.
LightGBM calculates the potential gain, or the improvement in the performance of the model, for each split scenario in which the node is possible.
LightGBM divides the original node into two new leaves by selecting the split with maximum gain.
Unlike level-wise growth which splits all nodes at each level, leaf-wise growth will proceed by selecting the leaf (among all current leaves) with the biggest gain and dividing it. This process continues until a stopping condition (like the maximum tree depth, minimum gain, or minimum number of samples in a leaf) is met.

Advantages of Leaf-wise Growth

Here are some benefits of leaf-wise growth in LightGBM −

Better accuracy: Leaf-wise growth usually delivers better precision as it actively targets the parts of the tree that can offer the greater precision.
Efficiency: LightGBM operates more quickly because it does not waste energy on expanding leaves that have no meaningful impact on loss reduction.

Disadvantages of Leaf-wise Growth

Below are some drawbacks of leaf-wise growth in LightGBM −

Over-fitting: As the tree can grow very deeply on one side, it is more vulnerable to over-fit if the data is uncertain or small. To help minimize this, LightGBM offers parameters like max_depth and min_child_samples.
Unbalanced Memory Usage: Tree growth may be uneven, resulting in varied uses for memory and causing challenges for specific programs.

Key Parameters for Controlling Leaf-wise Growth

Below are some key parameters are listed which you can use in Leaf-Wise tree growth in LightGBM −

num_leaves: Defines the maximum number of leaves that a tree is allowed to have. More leaves can improve accuracy even though they may encourage over-fitting.
min_child_samples: The lowest amount of data required for every leaf. Increasing this value can reduce over-fitting.
max_depth: The maximum depth of a tree. Controls the tree's depth and has the power to prevent it from going too deep.
learning_rate: Controls the size of the training step. Even though it needs more boosting cycles, lower values can produce better results.

Example of LightGBM Leaf-Wise Tree Growth

Here is a simple example to show how LightGBM uses leaf-wise growth −

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Generate synthetic dataset
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=42)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a dataset for LightGBM
train_data = lgb.Dataset(X_train, label=y_train)

# Parameters for LightGBM with leaf-wise growth
params = {
   'objective': 'binary',
   'boosting_type': 'gbdt',  # Gradient Boosting Decision Tree
   'num_leaves': 31,         # Controls the complexity (number of leaves)
   'learning_rate': 0.05,
   'metric': 'binary_logloss',
   'verbose': -1
}

# Train the model
gbm = lgb.train(params, train_data, num_boost_round=100)

# Predict and evaluate
y_pred = gbm.predict(X_test)
y_pred_binary = [1 if p > 0.5 else 0 for p in y_pred]
print(f"Accuracy: {accuracy_score(y_test, y_pred_binary):.4f}")

Output

The result of the above LightGBM model is:

Accuracy: 0.9450

Leaf-wise vs. Level-wise Tree Growth

Below is a comparison between level-wise and leaf-wise tree growth approaches −

Criteria	Level-wise Tree Growth	Leaf-wise Tree Growth (LightGBM)
Growth Pattern	Adds nodes level by level, expanding all leaves of the current depth equally.	Adds nodes to the leaf with the maximum gain, focusing on one leaf at a time.
Tree Structure	Results in a symmetric tree, where all leaves are at the same level.	Results in an asymmetric tree, which can grow deeper on some branches.
Greediness	Less greedy, as it considers all possible splits at each level.	More greedy, as it focuses on the most promising leaf to split next.
Efficiency	Generally more memory-efficient but may take longer to find optimal splits.	More efficient in finding optimal splits, but can use more memory due to deeper trees.
Accuracy	May not find the best splits quickly, potentially leading to lower accuracy.	Often results in better accuracy due to focusing on the most significant splits.
Over-fitting Risk	Lower risk of over-fitting, as the tree grows in a balanced manner.	Higher risk of over-fitting, especially with noisy data, due to deeper growth.
Use Case	Suitable for small datasets or when memory usage is a concern.	Suitable for large datasets where accuracy is the priority.

Print Page