How calculate average output probabilities in MLP or SVM in MATLAB

Question

I have a system that find best model (best inputs and parameters of MLP/SVM) model in a financial problem for every inserted database and create a specific model for a specific data sample. I'm using 5-fold cross validation that repeats for 5 times (for increasing reliability of my system) for cost function of my optimization algorithm so In the end I will find best 25 models that now will use for out-sample data.

When I insert out sample data to these 25 models in some cases I have slightly different outputs ( I have a classification MLP model that can generate probability). This is one of the outputs :

0.380032703, 0.356491904, 0.369914169, 0.377588996, 0.425321986 ,0.382467546, 0.341404809, 0.538460421, 0.353585952, 0.332340499, 0.36827261, 0.369171894, 0.329522399, 0.398054125, 0.360501952, 0.406897298, 0.480889919, 0.403619213, 0.406236551, 0.422529236, 0.393108809, 0.309808029, 0.357776713, 0.337373411, 0.397259506

As you know these outputs have difference parts of data as training sets and testing sets for creating model. Other things like inputs and parameters are the same.I should have only one probability for any out sample. My fist approach was using average. I want your suggestions to create one robust probability from these probabilities. When I check other out sample outputs, In a lot of cases difference is low but in some cases difference between probabilities is high.

What is your idea about this problem?

Thanks.

Answer 1

You should weight the outputs according to how much you believe that the model which generated the output is correct. If you have no reason to prefer any of the models, then they all have the same weight (namely 1/(number of models)), and the combined output is just the mean of the individual outputs.