Gradient boosting takes a different ensembling strategy than random forests: instead of building many independent trees in parallel, it builds trees sequentially, each new tree specifically correcting the errors made by the trees before it.
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(
n_estimators=100, # number of sequential trees
learning_rate=0.1, # how much each tree corrects the previous ensemble's errors
max_depth=3, # boosted trees are typically kept shallow
random_state=42,
)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
Where random forest trees are trained independently and in any order (well-suited to parallelization), gradient boosting trees must be trained one after another — each tree is trained specifically on the residual errors left by the ensemble so far, gradually reducing the overall error with each addition. learning_rate controls how much each new tree is allowed to correct — a smaller value trains more slowly (needing more trees, n_estimators, to reach the same performance) but generally generalizes better, while a larger value learns faster but risks overshooting and overfitting.
Because each tree in a boosted ensemble is intentionally kept small and weak on its own (hence shallow max_depth, typically 3-6), and correction happens gradually and deliberately, gradient boosting often achieves higher accuracy than random forests on many tabular problems — at the cost of being slower to train (sequential, not parallelizable) and more sensitive to hyperparameter choices, requiring more careful tuning to get right. Modern, highly-optimized implementations — XGBoost, LightGBM, and CatBoost — extend this same core idea with substantial speed and performance improvements, and are the dominant approach in tabular-data machine learning competitions and much real-world production use, worth exploring once scikit-learn's built-in GradientBoostingClassifier feels comfortable.