Ensemble classifier with wrapper method

Question

I'm trying to combine multiple classifiers (ANN, SVM, kNN, ... etc.) using ensemble learning (viting, stacking ...etc.) .

In order to make a classifier, I'm using more than 20 types of explanatory variables. However, each classifier has the best subset of explanatory variables. Thus, seeking the best combination of explanatory variables for each classifier in wrapper method, I would like to combine multiple classifiers (ANN, SVM, kNN, ... etc.) using ensemble learning (viting, stacking ...etc.) .

By using the meta-learning with weka, I should be able to use the ensemble itself. But I can not obtain the best combination of explanatory variables since wrapper method summarizes the prediction of each classifier.

I am not stick to weka if it can be solved easier in maybe matlab or R.

Answer 1

With ensemble approaches, best results have been achieved with very simple classifiers. Which on the other hand can be pretty fast, to make up for the ensemble cost.

This may seem counterintuitive at first: one would exepect a better input classifier to produce a better output. However, there are two reasons why this does not work.

First of all, with simple classifiers, you can usually tweak them more to get a diverse set of input classifiers. A full-dimensional method + feature bagging gives you a diverse set of classifiers. A classifier that internally does feature selection or reduction makes feature bagging largely disfunct for getting variety. Secondly, a complex method such as SVM is more likely to optimize/converge towards the very same result. After all, the complex methods are supposed to go through a much larger search space and find the best result in this search space. But that also means, you are more likely to get the same result again. Last but not least, when using very primivite classifiers, the errors are better behaved and more likely to even out on ensemble combination.