Ensemble learning is a supervised learning technique used in machine learning to improve overall performance by combining the predictions from multiple models.
Each model can be a different classifier:
How does Ensemble Learning Work?
Ensemble learning works on the principle of the “wisdom of the crowd”. By combing multiple models, we can improve the accuracy of the predictions.
Types of Ensemble Methods
- Bootstrap aggregation (bagging)
- Random Forests
- Stacked Generalization (Blending)
Voting is an ensemble machine learning algorithm that involves making a prediction that is the average (regression) or the sum (classification) of multiple machine learning models.
- Same training sets
- Different algorithms
Bootstrap Aggregation (Bagging)
Bagging, or bootstrap aggregation, is an ensemble method that reduces the variance of individual models by fitting a decision tree on different bootstrap samples of a training set.
- Different training sets
- Same algorithm
- Two models from
Random forests use decision trees as the base estimator for the predictions and improve the performance of models by calculating the majority voting / average prediction of multiple decision trees.
Random forest is both a supervised learning algorithm and an ensemble algorithm.
- Base estimator is a decision tree
- Each estimator uses a different bootstrap sample of the training set
- Two models from
Boosting is an ensemble method that converts weak learners into strong learners by having each predictor fix the errors of its predecessor.
Boosting can be used in classification and regression problems.
Boosting machine learning algorithms work by:
- Instantiating a weak learner (e.g. CART with
- Making a prediction and passing the wrong predictions to the next predictor
- Paying more and more attention at each iteration to the observations. having prediction errors
- Making new prediction until the limit is reached or the higher accuracy is achieved.
Multiple boosting Algorithms
- Gradient Boosting: Gradient boosting machines, Gradient Boosted Regression Trees
Stacked Generalization (Blending)
Stacking, also known as Stacked Generalization, is an ensemble technique that improves the accuracy of the models by combining predictions of multiple classification or regression machine learning models.
Stacking machine learning algorithms work by:
- Using multiple first level models to predict on a training set
- Combining (stacking) the predictions to generate a new training set
- Fitting and predicting a second level model on the generated training set
Interesting Resources from the Community
- Understand Stacked Generalization (blending) in depth with code demonstration by Siddharth Agarwal
- How to Develop Voting Ensembles With Python by Jason Brownlee
- Ensemble Learning in Python by Sayak Paul
- A Comprehensive Guide to Ensemble Learning – What Exactly Do You Need to Know by Derrick Mwiti
- Marketing – Conversion Rate Analytics with RandomForestClassifier – by Michael Fuchs
- How to Build a Simple Marketing Mix Model with Python by Terence Shin
This concludes the introduction of ensemble machine learning algorithms. We have covered how ensemble learning works and provided an overview of the most common machine learning models.
The next step is to learn how to use Scikit-learn to train each ensemble machine learning models on real data.
SEO Strategist at Tripadvisor, ex- Seek (Melbourne, Australia). Specialized in technical SEO. In a quest to programmatic SEO for large organizations through the use of Python, R and machine learning.