Gradient Descent Boosting-based Decision Tree (GDBT) Learning Algorithm

Context:
- It can (typically) create a Decision Tree Ensemble.
- It can be implemented by a GBDT System (that solves a GBDT task).
- It can range from being a GBDT Classification Algorithm to being a GBDT Regression Algorithm.
- It can handle various data types, including categorical and continuous variables.
- It can be sensitive to the settings of hyperparameters, such as learning rate and tree depth, which require careful tuning for optimal performance.
- It can be computationally intensive, especially with large datasets and many trees.
- ...
Example(s):
- the one implemented in CatBoost, an open-source GBDT library, which improves handling categorical features as described by [[::Liudmila Prokhorenkova]] et al. in 2018.
- the one implemented in TreeBoost, a specific implementation of GBDT, optimized for decision tree models, known for its high predictive accuracy.
- the one proposed for The Gradient Boosting Machine (GBM) framework, by Jerome H. Friedman in 2001.
- ...
Counter-Example(s):
See: Boosting Algorithm, Decision Tree Training Algorithm, Gradient Descent Algorithm.

References

(Prokhorenkova, et al., 2018) ⇒ [[::Liudmila Prokhorenkova]], [[::Gleb Gusev]], [[::Aleksandr Vorobev]], [[::Anna Veronika Dorogush]], and [[::Andrey Gulin]]. ([[::2018]]). “CatBoost: Unbiased Boosting with Categorical Features.” In: Proceedings of advances in neural information processing systems 31

http://www.dtreg.com/treeboost.htm
- TreeBoost - Stochastic Gradient Boosting
- "Boosting" is a technique for improving the accuracy of a predictive function by applying the function repeatedly in a series and combining the output of each function with weighting so that the total error of the prediction is minimized. In many cases, the predictive accuracy of such a series greatly exceeds the accuracy of the base function used alone.
- The TreeBoost algorithm used by DTREG is optimized for improving the accuracy of models built on decision trees. Research has shown that models built using TreeBoost are among the most accurate of any known modeling technique.
- The TreeBoost algorithm is functionally similar to Decision Tree Forests because it creates a tree ensemble, and it uses randomization during the tree creations. However, a random forest builds the trees in parallel and they "vote" on the prediction; whereas TreeBoost creates a series of trees, and the prediction receives incremental improvement by each tree in the series.