在Array Appending方面遇到问题

Question

I feel kinda silly being stuck on something that seems so simple for so long but since Im about to put my head through the wall, I figured I'd ask for some help.

I have a loop that splits my data into smaller subsets and loops through each one. For each loop, it produces a y_test and a y_pred array. It'll be variable size but the shape is (X,). In order to plot the the two arrays vs each other, i just assigned the arrays to an empty dataframe and just used matplotlib to plot.

Now, i'd like to just be able to also keep a running total of the y_pred and y_test so I can see a plot of the entire data set.

What I've tried:

Initially, I just tried creating another empty data frame outside my loop and thought to just append the arrays to the end of my columns of the dataframe but i found appending arrays to dataframe was not possible.

Then i thought I'll just append to an empty array for each time through the loop and convert to a dataframe at the end to plot but Im not having much luck there either and if I understand correctly - np.append is creating a new array of the appended data every time I append? Wasn't sure if this would get memory intensive.

I was wondering what is the best way to do this?

Here is my code (I tried to remove a lot of the lines that weren't necessary to the problem to make it easier to follow):

continuous_results = pd.Dataframe()
        tscv = TimeSeriesSplit(n_splits=self.no_splits)
        for train_index, test_index in tqdm(tscv.split(X)):
            X_train, X_test = X.iloc[train_index], X.iloc[test_index]
            y_train, y_test = y.iloc[train_index], y.iloc[test_index]



            self.regressor.fit(X_train, y_train.ravel())

            # predict y values
            y_pred = self.regressor.predict(X_test)


            # plot y_pred vs y_test
            y_df = pd.DataFrame()
            y_pred = y_pred.reshape(len(y_pred), )
            y_test = y_test.reshape(len(y_test), )
            y_df['y_pred'] = y_pred
            y_df['y_test'] = y_test
# failed attempts at continuous dataframe
            continuous_results = continuous_results['Model'].append(y_pred[:,:])
            continuous_results = continuous_results['Actual'].append(y_test)


            y_df.plot()

Answer 1

It is possible to create DataFrames from numpy arrays and vice versa:

# If you already have data as an array
data = np.random.random((10,5))
# Create a dataframe from a numpy array
df = pd.DataFrame(data)
# Create a numpy array from a dataframe
as_array = df.to_numpy()

If you want/have to loop, you can do this with both numpy arrays and DataFrames. It is more efficient to construct a numpy from a list than to concatenate arrays in a loop:

# Looping - arrays can handle n dimensions
data = []
for i in range(10):
    row = np.random.random((1,1,1,1,1))
    # Add a second dimension
    row = row[:,np.newaxis]
    # Remove the second dimension
    row = row[:,-1]
    # A list can hold anything
    data.append(row)
# Construct an array from a list of arrays
array = np.array(data)

DataFrames can do this as well, but a dataframe row can only have one dimension.

# looping - dataframes can work with only one dimension per row
data = []
for i in range(10):
    data.append(np.random.random(5))
# Construct a DataFrame from a list of values
df = pd.DataFrame(data)

In order to append to an existing DataFrame, a Series or a DataFrame needs to be created from the data first.

df = pd.DataFrame()
for i in range(10):
    n = np.random.random(1)
    # To append to a DataFrame, first create a Series (a row or a column) or a DataFrame
    row = pd.Series(n, name=i)
    # append a Series (or a DataFrame) to the "bottom" of another DataFrame
    df = df.append(row)