- LightGBM - Home
- LightGBM - Overview
- LightGBM - Architecture
- LightGBM - Installation
- LightGBM - Core Parameters
- LightGBM - Boosting Algorithms
- LightGBM - Tree Growth Strategy
- LightGBM - Dataset Structure
- LightGBM - Binary Classification
- LightGBM - Regression
- LightGBM - Ranking
- LightGBM - Implementation in Python
- LightGBM - Parameter Tuning
- LightGBM - Plotting Functionality
- LightGBM - Early Stopping Training
- LightGBM - Feature Interaction Constraints
- LightGBM vs Other Boosting Algorithms
- LightGBM Useful Resources
- LightGBM - Quick Guide
- LightGBM - Useful Resources
- LightGBM - Discussion
LightGBM - Boosting Algorithms
Before we look at the various boosting algorithms in LightGBM, let us explain what a boosting algorithm is. Boosting is an effective machine learning approach that improves model accuracy. It works by combining multiple weak models (basic models that do not perform well on their own) to create an improved model that can make better predictions.
LightGBM is a popular framework for boosting. It includes a variety of methods for creating powerful predictive models.
LightGBM Boosting Algorithms
LightGBM supports a wide range of boosting techniques. Each has its own method for creating models and making predictions. Here is a list of the main boosting algorithms used in LightGBM −
Gradient Boosting Decision Tree (GBDT)
Random Forest (RF)
DART (Dropouts And Multiple Additive Regression Trees)
Gradient Based One-Side Sampling (GOSS)
Let us go into each of these algorithms −
Gradient Boosting Decision Tree (GBDT)
GBDT is the default and most commonly used algorithm in LightGBM. Here's how it works −
How it works?
GBDT builds a model in stages, with each stage looking for correct errors from the previous level. It uses decision trees to make predictions. A decision tree is similar to a flowchart in that it helps you make decisions based on certain criteria.
GBDT is incredibly powerful and accurate. It is widely used for a range of tasks, like classification and regression.
For example - In a GBDT model the first tree can predict whether a person will buy a product or not. The second tree will learn from the earlier tree's problems and try to solve them, and the cycle will continue.
Advantages of GBDT
Here are the benefits of GBDT algorithm −
High accuracy.
Can handle both numerical and categorical data.
Works well with large datasets.
Random Forest (RF)
Random Forest is another boosting approach that can be used with LightGBM. It is a bit different than GBDT.How it Works?
Random Forest builds many decision trees, each based on a different random sample of data. It then combines all of the trees to get the final prediction. The goal is to minimize over-fitting, which happens when a model performs well on training data but poorly on new, unlabeled data.
Random Forest is useful for creating a model that is more stable and less vulnerable to errors on new data.
Think about the forest of many trees, with each tree representing a unique decision path. The final choice depends on the majority vote of all the trees.
Advantages of Random Forests
Here are the benefits of Random Forests algorithm −
Handles large datasets with high dimensionality (many features).
Less likely to over-fit than a single decision tree.
Good performance in classification and regression challenges.
DART (Dropouts meet Multiple Additive Regression Trees)
DART is an improved version of GBDT with a unique change. Let us see how it works −
How it Works?
DART is like GBDT but adds the concept of "dropouts." Dropouts are random removals from the model's trees during training. This reduces the model's dependency on a single tree, resulting in it being more robust and resistant to over-fitting.
If your GBDT model is over-fitting, look into upgrading to DART. It adds regularization into the model, which enhances its performance on new data.
Suppose you are playing a game in which you have to answer questions, some of which are randomly eliminated. It allows you to pay more attention to the remaining questions, which leads to a better overall performance.
Advantages of DART
Here are the benefits of DART algorithm −
Reduces over-fitting by using the dropout method.
Maintains high accuracy while boosting generalization.
GOSS (Gradient-based One-Side Sampling)
GOSS is a boosting algorithm created for speed and efficiency. GOSS shows the most significant data points to speed up training. It accomplishes this by selecting only the data points with the highest errors and a few data points with lower errors. This reduces the amount of data that needs to be processed allowing training to go faster while retaining high accuracy.
GOSS is great for training your model quickly, mainly with large datasets.
Suppose you are preparing for an exam and choose to focus just on the most difficult questions. This saves time while completing the most challenging places and ensures your performance.
Advantages of GOSS
Here are the benefits of GOSS algorithm −
Faster training speed.
Maintains accuracy by focusing on important data points.
Choose the Right Boosting Algorithm
Choosing the right boosting algorithm is based on your specific requirements.
For great precision, start with the GBDT. It is an ideal default solution for most tasks.
If you have a large dataset and need to train quickly, try GOSS.
DART can help your model prevent over-fitting.
Random Forest is a reliable and straightforward model that generalizes well.