In a hurry! XGBoost: Six Easy Steps
What happens in Extreme Gradient Boosting — how it enhances performance
XGBoost is a decision tree-based ensemble machine learning, that uses a boosting framework.
We have explained the Decision Tree and CART technique in the article. Please read it to understand how Decision Tree identifies ways to split the features — Click Here
What is Boosting?
- In boosting each model is built on a sequence.
- Based on the training data set, both techniques create n classifier (random new datasets).
- Means model 1 build based on dataset 1. Then model 2 build based on model 1 (previous classifier success — based on weights) but uses dataset 2. This minimizes the errors from previous models and increases the influence of high performing models.
- Caution, this increases overfitting.
- Boosting uses certain weights to the next classification stage.
Evolution of Decision Tree:
XGBoost works on parallel processing, Fore example there are two loops, One, inner loop and second, the outer loop. Now, if the outer loop creates the leaf nodes one by one. And the inner loop calculates the features. Now, before completion of the inner loop, the outer loop cannot be started. And hence, this limitation is taken care of by parallelization in XGBoost, where to improve run time order of loops are interchangeable. This switch improves algorithm performance.
Similar to CART, we also put criteria to prune tree using max_depth parameter.
XGBoost takes care following inbuilt algorithm enhancements:
- Regularization to overcome overfitting through L1 Lassos and L2 Ridge.
- Sparse features admit, like, missing values, zero values, one-hot encoding naturally.
- Weightage Quintile sketch, to effectively find an optimal split in the database.
- Inbuilt cross-validation
Six Easy Steps to build XGBoost model in R:
Step 1: Install libraries, xgboost, margrittr, Matrix
Step 2: Create a Matrix for train and test datasets- with the use of xgb.DMatrix() function
Step 3: Set parameters, for params and watchlist
Step 4: Build model using xgb.train() function
Step 5: Use xgb.importance() function for feature analysis
Step 6: Make prediction with the use of predict() function
For practical application and how to apply the XGBoost on ‘a car driver accepts/rejects the offer’ prediction case study — Click Here
