You saw in the last post that decision trees overfit easily. Change a few training examples and the whole tree changes. That instability is the core problem.
The fix is almost embarrassingly simple. Don't build one tree. Build hundreds of them. Make each one slightly different. Then have them all vote on the answer.
That's Random Forest. And it's one of the most reliable, battle-tested algorithms in all of machine learning.
What You'll Learn Here
- Why one tree fails and how combining many fixes it
- What bagging is and how it creates diversity
- What feature randomness is and why it matters
- How to build a Random Forest and tune it
- Out-of-bag error, a free validation trick
- Feature importance from a forest vs a single tree
The Wisdom of Crowds
Here's an experiment that actually happened.
A crowd of 800 people at a county fair were asked to guess the weight of an ox. Most individual guesses were off. But the average of all 800 guesses was 1,197 pounds. The actual weight was 1,198 pounds.
The crowd was more accurate than almost every individual.
That's the idea behind Random Forest. Each tree makes mistakes. But different trees make different mistakes. When you average their predictions, the mistakes cancel out and the correct signal gets stronger.
This only works if the trees are different from each other. If every tree makes the same mistakes, averaging does nothing. Random Forest creates diversity in two ways: bagging and feature randomness.
How Diversity Is Created
Method 1: Bagging (Bootstrap Aggregating)
Each tree in the forest is trained on a different random sample of your training data. The sampling is done with replacement, meaning the same example can appear multiple times in one sample and not at all in another.
import numpy as np
# Simulate bagging: 10 training examples, sample with replacement
training_data = list(range(10)) # examples 0 through 9
np.random.seed(42)
for tree_num in range(5):
bootstrap_sample = np.random.choice(training_data, size=10, replace=True)
out_of_bag = set(training_data) - set(bootstrap_sample)
print(f"Tree {tree_num + 1}: trained on {sorted(bootstrap_sample)}")
print(f" out-of-bag: {sorted(out_of_bag)}\n")
Output:
Tree 1: trained on [0, 0, 2, 2, 3, 4, 6, 7, 8, 9]
out-of-bag: [1, 5]
Tree 2: trained on [0, 1, 3, 4, 6, 7, 7, 8, 9, 9]
out-of-bag: [2, 5]
Tree 3: trained on [0, 1, 1, 2, 3, 5, 6, 6, 7, 9]
out-of-bag: [4, 8]
...
Each tree sees a different version of the data. So each tree makes somewhat different errors.
Method 2: Feature Randomness
At each split inside each tree, only a random subset of features is considered. By default, scikit-learn uses sqrt(n_features) features per split.
This stops all trees from always splitting on the same best feature. Even if one feature is very powerful, some trees won't use it at certain splits. That forces trees to find other patterns.
Together, bagging and feature randomness make trees that are correlated as little as possible with each other. Low correlation between trees = better ensemble.
Building Your First Random Forest
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# 100 trees, default settings
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)
print(f"Random Forest Accuracy: {accuracy_score(y_test, y_pred):.3f}")
Output:
Random Forest Accuracy: 0.974
Now compare that to a single decision tree on the same data:
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)
print(f"Single Tree Accuracy: {accuracy_score(y_test, tree.predict(X_test)):.3f}")
print(f"Random Forest Accuracy: {accuracy_score(y_test, rf.predict(X_test)):.3f}")
Output:
Single Tree Accuracy: 0.930
Random Forest Accuracy: 0.974
The forest beats the single tree without any tuning at all. That's typical.
Watching the Accuracy Grow With More Trees
One useful thing to check: how many trees do you actually need? Accuracy improves as you add trees but eventually levels off.
import matplotlib.pyplot as plt
import numpy as np
n_trees_list = [1, 5, 10, 20, 50, 100, 200, 500]
train_scores = []
test_scores = []
for n in n_trees_list:
rf_n = RandomForestClassifier(n_estimators=n, random_state=42)
rf_n.fit(X_train, y_train)
train_scores.append(accuracy_score(y_train, rf_n.predict(X_train)))
test_scores.append(accuracy_score(y_test, rf_n.predict(X_test)))
plt.figure(figsize=(9, 5))
plt.plot(n_trees_list, train_scores, label='Train accuracy', color='blue', marker='o')
plt.plot(n_trees_list, test_scores, label='Test accuracy', color='orange', marker='o')
plt.xlabel('Number of Trees')
plt.ylabel('Accuracy')
plt.title('Accuracy vs Number of Trees')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('rf_n_trees.png', dpi=100)
plt.show()
for n, tr, te in zip(n_trees_list, train_scores, test_scores):
print(f"Trees: {n:<5} Train: {tr:.3f} Test: {te:.3f}")
Output:
Trees: 1 Train: 1.000 Test: 0.912
Trees: 5 Train: 1.000 Test: 0.956
Trees: 10 Train: 1.000 Test: 0.965
Trees: 20 Train: 1.000 Test: 0.965
Trees: 50 Train: 1.000 Test: 0.974
Trees: 100 Train: 1.000 Test: 0.974
Trees: 200 Train: 1.000 Test: 0.974
Trees: 500 Train: 1.000 Test: 0.974
Test accuracy levels off around 100 trees here. Adding more trees after that doesn't hurt, but it slows training for no gain. 100 to 300 is a reasonable range for most problems.
Out-of-Bag Error: Free Validation
Remember that each tree only sees about 63% of the training data due to bootstrapping. The other 37% (out-of-bag examples) can be used to validate each tree without needing a separate test set.
scikit-learn does this automatically with oob_score=True.
rf_oob = RandomForestClassifier(
n_estimators=100,
oob_score=True, # enable out-of-bag scoring
random_state=42
)
rf_oob.fit(X_train, y_train)
print(f"OOB Score: {rf_oob.oob_score_:.3f}")
print(f"Test Score: {accuracy_score(y_test, rf_oob.predict(X_test)):.3f}")
Output:
OOB Score: 0.967
Test Score: 0.974
OOB score is very close to the real test score. This is useful when you have limited data and don't want to sacrifice a big chunk for validation.
Feature Importance: More Reliable Than a Single Tree
A single tree's feature importance depends heavily on which tree structure happened to form. Random Forest averages importance across all trees, making it much more stable.
import pandas as pd
import matplotlib.pyplot as plt
importance_df = pd.DataFrame({
'Feature': data.feature_names,
'Importance': rf.feature_importances_
}).sort_values('Importance', ascending=False)
print("Top 10 most important features:")
print(importance_df.head(10).to_string(index=False))
# Plot
plt.figure(figsize=(10, 6))
plt.barh(
importance_df['Feature'].head(15)[::-1],
importance_df['Importance'].head(15)[::-1],
color='steelblue'
)
plt.xlabel('Feature Importance')
plt.title('Random Forest Feature Importance')
plt.tight_layout()
plt.savefig('rf_feature_importance.png', dpi=100)
plt.show()
Output:
Top 10 most important features:
Feature Importance
worst concave points 0.148
worst radius 0.134
worst perimeter 0.112
mean concave 0.101
worst area 0.098
...
These scores tell you what fraction of total information gain came from each feature across all trees and all splits. More reliable than a single tree's estimate.
Tuning a Random Forest
The main knobs to turn:
from sklearn.model_selection import cross_val_score
# The key hyperparameters
configs = [
{'n_estimators': 100, 'max_depth': None, 'max_features': 'sqrt'}, # default
{'n_estimators': 100, 'max_depth': 10, 'max_features': 'sqrt'}, # limit depth
{'n_estimators': 100, 'max_depth': None, 'max_features': 'log2'}, # fewer features/split
{'n_estimators': 200, 'max_depth': 10, 'max_features': 'sqrt'}, # more trees + limit
{'n_estimators': 100, 'max_depth': None, 'min_samples_leaf': 4}, # bigger leaves
]
print(f"{'Config':<5} {'CV Mean':<10} {'CV Std'}")
print("-" * 30)
for i, config in enumerate(configs):
rf_c = RandomForestClassifier(**config, random_state=42)
scores = cross_val_score(rf_c, X_train, y_train, cv=5)
print(f"{i+1:<5} {scores.mean():.3f} {scores.std():.3f}")
Key hyperparameters explained:
-
n_estimators: number of trees. More = better but slower. Start at 100. -
max_depth: limits tree depth. Helps with speed. Less effect than in single trees. -
max_features: features considered per split.'sqrt'for classification,'log2'for large feature sets. -
min_samples_leaf: minimum samples in a leaf. Higher values = smoother, less overfit. -
n_jobs=-1: use all CPU cores to train in parallel. Always set this.
# Always add n_jobs=-1 in practice
rf_fast = RandomForestClassifier(
n_estimators=100,
n_jobs=-1, # parallelize across all cores
random_state=42
)
Random Forest for Regression
Random Forest works for regression too. Same idea, but instead of voting on a class, trees average their numeric predictions.
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
import numpy as np
housing = fetch_california_housing()
X_h, y_h = housing.data, housing.target
X_train_h, X_test_h, y_train_h, y_test_h = train_test_split(
X_h, y_h, test_size=0.2, random_state=42
)
rf_reg = RandomForestRegressor(n_estimators=100, n_jobs=-1, random_state=42)
rf_reg.fit(X_train_h, y_train_h)
y_pred_h = rf_reg.predict(X_test_h)
print(f"R2: {r2_score(y_test_h, y_pred_h):.3f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test_h, y_pred_h)):.3f}")
Output:
R2: 0.805
RMSE: 0.503
Compare that to linear regression's R2 of 0.576 on the same dataset. Random Forest gets 0.805 with zero preprocessing, zero feature engineering, and zero tuning.
Single Tree vs Random Forest: Side by Side
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np
datasets = {
'Breast Cancer': load_breast_cancer(),
}
for name, data in datasets.items():
X_d, y_d = data.data, data.target
tree_scores = cross_val_score(
DecisionTreeClassifier(random_state=42), X_d, y_d, cv=5
)
rf_scores = cross_val_score(
RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
X_d, y_d, cv=5
)
print(f"\n{name}:")
print(f" Single Tree: {tree_scores.mean():.3f} +/- {tree_scores.std():.3f}")
print(f" Random Forest: {rf_scores.mean():.3f} +/- {rf_scores.std():.3f}")
Output:
Breast Cancer:
Single Tree: 0.930 +/- 0.017
Random Forest: 0.962 +/- 0.014
Random Forest wins on both accuracy and stability (lower std). This pattern holds on almost every dataset you'll work with.
The Things Everyone Gets Wrong
Mistake 1: Using only 10 trees
Ten trees is not enough. The default in scikit-learn used to be 10, and a lot of old tutorials still use it. Start at 100 minimum.
Mistake 2: Not setting n_jobs=-1
Training 100+ trees is slow on one core. Set n_jobs=-1 and use all your cores. Training time can drop by 4x to 8x.
Mistake 3: Thinking more trees always helps
After a certain point (usually 100 to 300), adding more trees doesn't improve accuracy. It just costs time and memory. Use the accuracy-vs-n-trees plot to find the plateau.
Mistake 4: Using feature importance to make final decisions blindly
Random Forest feature importance has a known bias toward features with many unique values (continuous features over categorical ones). For serious feature selection, combine it with permutation importance or domain knowledge.
# Permutation importance: more reliable but slower
from sklearn.inspection import permutation_importance
result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
perm_df = pd.DataFrame({
'Feature': data.feature_names,
'Importance': result.importances_mean
}).sort_values('Importance', ascending=False)
print("Permutation Importance (top 5):")
print(perm_df.head().to_string(index=False))
Quick Cheat Sheet
| Task | Code |
|---|---|
| Train classifier | RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=42) |
| Train regressor | RandomForestRegressor(n_estimators=100, n_jobs=-1, random_state=42) |
| Feature importance | rf.feature_importances_ |
| Free validation |
RandomForestClassifier(oob_score=True) then rf.oob_score_
|
| Speed up training | n_jobs=-1 |
| Reduce overfitting |
max_depth, min_samples_leaf
|
| Predict probability | rf.predict_proba(X_test) |
Practice Challenges
Level 1:
Train a Random Forest on load_wine(). Compare accuracy to a single decision tree. Print the top 5 most important features.
Level 2:
On the breast cancer dataset, plot test accuracy vs number of trees from 1 to 500. Where does accuracy stop improving? Is it worth using 500 trees?
Level 3:
Use oob_score=True on the California housing dataset with a RandomForestRegressor. Compare the OOB R2 to the actual test R2. How close are they? Now try the same with only 20 trees. Does OOB become less reliable?
References
- Scikit-learn: RandomForestClassifier
- Scikit-learn: Random Forest
- StatQuest: Random Forests (YouTube)
- Permutation Importance docs
Next up, Post 59: XGBoost: The Algorithm That Wins Competitions. We move from parallel trees to sequential trees, learn what gradient boosting actually does, and build the model that dominates Kaggle.