Abstract

XGBoost is a very popular and powerful method for prediction. It is an iterative algorithm fitting simple decision trees to the residuals of the previous step. There is an efficient and scalable implementation available. The standard loss-function for XGBoost is the quadratic function, but a Huber loss is also available. In this paper we study the robustness of XGBoost, and show that it is fairly robust, except in presence of vertical outliers. To address this issue, we study other loss functions, corresponding to the S- and tau- estimators from robust regression. It turns out that a two-step procedure, called the MM-XGBoost, provides the best trade-off between robustness and prediction accuracy.