Neural Network predictions are always the same while testing an fMRI dataset with pyBrain. Why?

Question

I am quite new to fMRI analysis. I am trying to determine which object (out of 9 objects) a person is thinking about just by looking at their Brain Images. I am using the dataset on https://openfmri.org/dataset/ds000105/ . So, I am using a neural network by inputting 2D slices of brain images to get the output as 1 of the 9 objects. There are details about every step and the images in the code below.

import os, mvpa2, pyBrain
    import numpy as np
    from os.path import join as opj
    from mvpa2.datasets.sources import OpenFMRIDataset
    from pybrain.datasets import SupervisedDataSet,classification

path = opj(os.getcwd() , 'datasets','ds105')

of = OpenFMRIDataset(path)

#12th run of the 1st subject
ds = of.get_model_bold_dataset(model_id=1, subj_id=1,run_ids=[12])

#Get the unique list of 8 objects (sicissors, ...) and 'None'.
target_list = np.unique(ds.sa.targets).tolist()

#Returns Nibabel Image instance
img = of.get_bold_run_image(subj=1,task=1,run=12)

# Getting the actual image from the proxy image
img_data = img.get_data()

#Get the middle voxelds of the brain samples
mid_brain_slices = [x/2 for x in img_data.shape]

# Each image in the img_data is a 3D image of 40 x 64 x 64 voxels, 
# and there are 121 such samples taken periodically every 2.5 seconds.
# Thus, a single person's brain is scanned for about 300 seconds (121 x 2.5).
# This is a 4D array of 3 dimensions of space and 1 dimension of time, 
# which forms a matrix of (40 x 64 x 64 x 121) 

# I only want to extract the slice of the 2D images brain in it's top view 
# i.e. a series of 2D images 40 x 64
# So, i take the middle slice of the brain, hence compute the middle_brain_slices

DS = classification.ClassificationDataSet(40*64, class_labels=target_list)

# Loop over every brain image
for i in range(0,121):

    #Image of brain at i th time interval
    brain_instance = img_data[:,:,:,i]

    # We will slice the brain to create 2D plots and use those 'pixels'
    # as the features

    slice_0 = img_data[mid_brain_slices[0],:,:,i] #64 x 64
    slice_1 = img_data[:,mid_brain_slices[1],:,i] #40 x 64
    slice_2 = img_data[:,:,mid_brain_slices[2],i] #40 x 64

    #Note : we may actually only need one of these slices (the one with top view)

    X = slice_2 #Possibly top view

    # Reshape X from 40 x 64 to 1D vector 2560 x 1
    X = np.reshape(X,40*64)

    #Get the target at this intance (y)
    y = ds.sa.targets[i]
    y = target_list.index(y)

    DS.appendLinked(X,y)


print DS.calculateStatistics()
print DS.classHist
print DS.nClasses
print DS.getClass(1)

# Generate y as a 9 x 1 matrix with eight 0's and only one 1 (in this training set)
DS._convertToOneOfMany(bounds=[0, 1])

#Split into Train and Test sets
test_data, train_data = DS.splitWithProportion( 0.25 )
#Note : I think splitWithProportion will also internally shuffle the data

#Build neural network
from pybrain.tools.shortcuts import buildNetwork
from pybrain.structure.modules   import SoftmaxLayer
nn = buildNetwork(train_data.indim, 64, train_data.outdim, outclass=SoftmaxLayer)

from pybrain.supervised.trainers import BackpropTrainer
trainer = BackpropTrainer(nn, dataset=train_data, momentum=0.1, learningrate=0.01 , verbose=True, weightdecay=0.01) 
trainer.trainUntilConvergence(maxEpochs = 20)

The line nn.activate(X_test[i]) should take the 2560 inputs and generate a probability output, right? in the predicted y vector (shape 9 x 1 )

So, I assume the highest of the 9 values should be assigned answer. But it is not the case when I verify it with y_test[i]. Furthermore, I get similar values for X_test for every test sample. Why is this so?

 #Just splitting the test and trainset 
 X_train = train_data.getField('input')
 y_train = train_data.getField('target')
 X_test = test_data.getField('input')
 y_test = test_data.getField('target')

 #Testing the network
  for i in range(0,len(X_test)):
     print nn.activate(X_test[i])
     print y_test[i]

When I include the code above, here are some values of X_test :

.
.
.

nn.activated =  [ 0.44403205  0.06144328  0.04070154  0.09399672  0.08741378  0.05695479 0.08178353  0.0623408   0.07133351]
y_test [0 1 0 0 0 0 0 0 0]

nn.activated =  [ 0.44403205  0.06144328  0.04070154  0.09399672  0.08741378  0.05695479 0.08178353  0.0623408   0.07133351]
y_test [1 0 0 0 0 0 0 0 0]

nn.activated =  [ 0.44403205  0.06144328  0.04070154  0.09399672  0.08741378  0.05695479 0.08178353  0.0623408   0.07133351]
y_test [0 0 0 0 0 0 1 0 0]
.
.
.

So the probability of the test sample being index 0 in every case id 44.4% irrespective of the sample value. The actual values keep varying though.

print 'print predictions: ' , trainer.testOnClassData (dataset=test_data)

x = []
for item in y_test:
    x.extend(np.where(item == 1)[0])
print 'print actual: ' , x   

Here, the output comparison is :

print predictions:  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
print actual:  [7, 0, 4, 8, 2, 0, 2, 1, 0, 6, 1, 4]

All the predictions are for the first item. I don't know what the problem is. The total error seems to be decreasing, which is a good sign though :

Total error:  0.0598287764931
Total error:  0.0512272330797
Total error:  0.0503835076374
Total error:  0.0486402801867
Total error:  0.0498354140541
Total error:  0.0495447833038
Total error:  0.0494208449895
Total error:  0.0491162599037
Total error:  0.0486775862084
Total error:  0.0486638648161
Total error:  0.0491337891419
Total error:  0.0486965691406
Total error:  0.0490016912735
Total error:  0.0489939195858
Total error:  0.0483910986235
Total error:  0.0487459940103
Total error:  0.0485516142106
Total error:  0.0477407360102
Total error:  0.0490661144891
Total error:  0.0483103097669
Total error:  0.0487965594586

Answer 1

I can't be sure -- because I haven't used all of these tools together before, or worked specifically in this kind of project -- but I would look at the documentation and be sure that your nn is being created as you expect it to.

Specifically, it mentions here:

http://pybrain.org/docs/api/tools.html?highlight=buildnetwork#pybrain.tools.shortcuts.buildNetwork

that "If the recurrent flag is set, a RecurrentNetwork will be created, otherwise a FeedForwardNetwork.", and you can read here:

http://pybrain.org/docs/api/structure/networks.html?highlight=feedforwardnetwork

that "FeedForwardNetworks are networks that do not work for sequential data. Every input is treated as independent of any previous or following inputs.".

Did you mean to create a "FeedForward" network object?

You're testing by looping over an index and activating each "input" field that's based off the instantiation of a FeedForwardNetwork object, which the documentation suggests are treated as independent of other inputs. This may be why you're getting such similar results each time, when you are expecting better convergences.

You initialize your dataset ds object with the parameters model_id=1, subj_id=1,run_ids=[12] , suggesting that you're only looking at a single subject and model, but 12 "runs" from that subject under that model, right?

Most likely there's nothing semantically or grammatically wrong with your code, but a general confusion from the PyBrain library's presumed and assumed models, parameters, and algorithms. So don't tear your hair out looking for code "errors"; this is definitely a common difficulty with under-documented libraries.

Again, I may be off base, but in my experience with similar tools and libraries, it's most often that the benefit of taking an extremely complicated process and simplifying it to just a couple dozen lines of code, comes with a TON of completely opaque and fixed assumptions.

My guess is that you're essentially re-running "new" tests on "new" or independent training data, without all the actual information and parameters that you thought you had setup in the previous code lines. You are exactly correct that the highest value (read: largest probability) is the "most likely" (that's exactly what each value is, a "likeliness") answer, especially if your probability array represents a unimodal distribution .

Because there are no obvious code syntax errors -- like accidentally looping over a range iterator equivalent to the list [0,0,0,0,0,0] ; which you can verify because you reuse the i index integer in printing y_test which varies and the result of nn.activate(X_test[i]) which isn't varying -- then most likely what's happening is that you're basically restarting your test every time and that's why you're getting an identical result, not just similar but identical for every printout of that nn.activate(...) method's results.

This is a complex, but very well written and well illustrated question, but unfortunately I don't think there will be a simple or blatantly obvious solution.

Again, you're getting the benefits of PyBrain's simplificaiton of neural networks, data training, heuristics, data reading, sampling, statistical modelling, classification, and so on and so forth, all reduced into single line or two line commands. There are assumptions being made, TONS of them. That's what the documentation needs to be illuminating, and we have to be very very careful when we use tools like these that it's not just a matter of correct syntax, but an actually correct (read: expected) algorithm, assumptions and all.

Good luck!

(PS -- Open source libraries also, despite a lack of documentation, give you the benefit of checking the source code to see [assumptions and all] what they're actually doing: https://github.com/pybrain/pybrain )