如何处理数据以允许在其上运行随机森林？

Question

I want to train a random forest on a bunch of matrices (first link below for an example). I want to classify them as either "g" or "b" (good or bad, a or b, 1 or 0, it doesn't matter).

I've called the script randfore.py. I am currently using 10 examples, but I will be using a much bigger data set once I actually get this up and running.

Here is the code:

# -*- coding: utf-8 -*-
import numpy as np
import pandas as pd
import os

import sklearn
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

working_dir = os.getcwd() # Grabs the working directory

directory = working_dir+"/fakesourcestuff/" ## The actual directory where the files are located

sources = list() # Just sets up a list here which is going to become the input for the random forest

for i in range(10):
    cutoutfile = pd.read_csv(directory+ "image2_with_fake_geotran_subtracted_corrected_cutout_" + str(i) +".dat", dtype=object) ## Where we get the input data for the random forest from
    sources.append(cutoutfile) # add it to our sources list

targets = pd.read_csv(directory + "faketargets.dat",sep='\n',header=None, dtype=object) # Reads in our target data... either "g" or "b" (Good or bad)


sources = pd.DataFrame(sources) ## I convert the list to a dataframe to avoid the "ValueError: cannot copy sequence with size 99 to array axis with dimension 1" error. Necessary?

# Training sets
X_train = sources[:8] # Inputs
y_train = targets[:8] # Targets

# Random Forest
rf = RandomForestClassifier(n_estimators=10)
rf_fit = rf.fit(X_train, y_train)

Here is the current error output:

Traceback (most recent call last):
  File "randfore.py", line 31, in <module>
    rf_fit = rf.fit(X_train, y_train)
  File "/home/ithil/anaconda2/envs/iraf27/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 247, in fit
    X = check_array(X, accept_sparse="csc", dtype=DTYPE)
  File "/home/ithil/anaconda2/envs/iraf27/lib/python2.7/site-packages/sklearn/utils/validation.py", line 382, in check_array
    array = np.array(array, dtype=dtype, order=order, copy=copy)
ValueError: setting an array element with a sequence.

I tried making the dtype = object, but it hasn't helped. I'm just not sure what sort of manipulation I need to perform to have this work.

I think the problem is because the files I appending to sources aren't just numbers but a mix of numbers, commas, and various square brackets (it's basically a big matrix). Is there a natural way to import this? The square brackets in particular are probably an issue.

Before I converted sources to a DataFrame I was getting the following error:

ValueError: cannot copy sequence with size 99 to array axis with dimension 1 This is due to the dimensions of my input (100 lines long) and my target which has 10 rows and 1 column.

Here is the contents of the first file that's read into cutouts (they're all the exact same style) to be used as the input: https://pastebin.com/tkysqmVu

And here is the contents of faketargets.dat, the targets: https://pastebin.com/632RBqWc

Any ideas? Help greatly appreciated. I am sure there is a lot of fundamental confusion going on here.

Answer 1

Try writing:

X_train = sources.values[:8] # Inputs
y_train = targets.values[:8] # Targets

I hope this will solve your problem!