Gradient Boosting Algorithm: A Comprehensive Guide For 2021
Introduction
Gradient boosting is a method that is used to create models in order to use it for prediction. The procedure is used in classification and in regression. The prediction model will usually be presented in the form of a gradient boosting decision tree in order to choose the prediction that is the best. Gradient boosting presents the model building in various stages. This is similar to like in the other methods of boosting. It allows for optimization and generalization of the differentiable loss function.
1) What is GB?
Gradient boosting is a kind of machine learning boosting. It believes in this intuition that when the next best possible model gets combined with the previous model. It helps to minimize the prediction error overall. The real idea is to set the outcome target for the next model which is done in order to minimize any error. Here is how the target is calculated. The outcome target for every case in the data will depend on how much will change the reputation of the case and impact the overall error of prediction.
- If in case there is a small change in the prediction case which causes a large error drop then the next target outcome of the case will be a high value. The prediction from any new model that is close to the target will reduce error.
- If there is a small change in the prediction in the case that does not cause any change in error then the next target outcome will be zero for the case. When there are changes in the prediction it will not decrease the error.
Gradient boosting is the name given to it because the target outcome for every case is set based on the error gradient with respect to the prediction. Every new model will take a step in a direction that will minimize the error of prediction in the space of the possible prediction in each of the training cases.
2) How does gradient boosting work?
Gradient boosting will involve these three elements.
- A loss function that has to be optimized
- A weak learner who will make predictions
- An additive model that helps to add weak learners to minimize the loss function
Loss Function
The loss function that is used will depend on the kind of problem that is being solved. It should be differentiable. However, there are many standard loss functions that are supported and you are free to define your own. Like for example gradient boosting regression could use a squares error and classification could use the logarithmic loss. The reason why the gradient boosting framework is used is that a new algorithm does not need to be derived for every loss function that may want to get used. Instead, the framework is generic enough that can be used by any differentiable loss function.
Weak Learner
The decision tree is used as a weak learner in the gradient boosting. The regression tree is used that outputs the real value for the splits and the outputs of these can be added together. This allows the subsequent model outputs to get added and also correct the residuals that are there in the prediction. The tree gets made in a greedy manner by choosing the split points that are the best which is based on the purity score in order to minimize losses.
The short decision trees could have only one split which is the decision stump. The large trees could be used that may have 4-8 levels. It is possible to contain weak learners in some way. This could be maximizing the layers, splits, nodes, and leaf nodes. This is so that the learner stays weak but can also be constructed in a greedy way.
Additive Model
The trees get added one by one and the existing trees in the model do not change. The procedure of gradient descent minimizes the loss when the trees are added. The gradient descent is used in order to minimize the oarsmen sets. This includes the coefficient in the regression equation or the weight in a neural network. After the error or loss gets calculated the weights get updated in order to minimize any error.
Instead of the parameters, there is a weak learner sub-model or which is more specifically the decision trees. After the loss is calculated in order to perform a gradient descent procedure a tree should be added to the model that helps to reduce loss. This is done by parametrizing the tree and then modifying the tree parameters and moving it in the right direction to reduce the residual loss. This approach is known as functional gradient descent.
The new tree output is then added to the original output in the existing sequence of trees in an effort to improve and correct the final model output. A fixed number of trees get added. The training gets stopped once the loss reaches a level that is acceptable or if it is non-linear and improves on the dataset validation that is external.
3) Why is gradient boosting good?
In the ensemble model, the boosting comes with a gradient boosting algorithm that is easy to read and interpret. This makes it easy to handle the interpretations of the predictions. The prediction capability is done using the clone methods. This could be a random forest or bagging and the decision trees. Gradient boosting is a resilient technique that curbs the overfitting in an easy way. Gradient boosting is good because it is robust and an out of the box gradient boosting classifier. This performs on the dataset where very minimal effort gets spent on cleaning. One can learn the complex decisions of a non-linear form through gradient boosting.
Conclusion
Gradient boosting, as a concept, was originated by Leo Breiman. He discovered that the gradient boosting technique can be applied to the cost functions that are appropriate in the form of an optimization algorithm. This method has gone through various developments in order to optimize the cost functions and this is done by picking the weak function or hypothesis with a gradient that is negative.
If you are interested in making it big in the world of data and evolve as a Future Leader, you may consider our Integrated Program in Business Analytics, a 10-month online program, in collaboration with IIM Indore!
ALSO READ
