如何保存numpy数组，以便以后可以适当地加载它们？

Question

I have a code which outputs an N-length Numpy array at every iteration.

Eg. -- theta = [ 0, 1, 2, 3, 4 ]

I want to be able to save the arrays to a text file or .csv file dynamically such that I can load the data file later and extract appropriately which array corresponds to which iteration. Basically, it should be saved in an ordered fashion.

I am assuming the data file would look something like this:-

0 1 2 3 4

1 2 3 4 5

2 3 4 5 6 ... (Random output)

I thought of using np.c_ but I don't want to overwrite the file at every iteration and if I simply save the terminal output as > output.txt, it saves as arrays including the brackets. I don't know how to read such a text file.

Is there a proper method to do this, ie write and read the data?

Answer 1

I use pandas and openpyxl in order to store data I calculated with Python.

Here an example in which I generate a numpy array, I convert it to a pandas dataframe and then I save it into an excel file located in "path" (remember that if you are working on Windows you need "r" at the beginning of the file path in order to convert the symbol "\\" to the symbol "\\\\")

import numpy as np
import pandas as pd
from openpyxl import load_workbook

path = r"C:\Users\fedel\Desktop\excelData\fileName.xlsx"

data = np.random.randn(100)
data = pd.DataFrame(data)
n = 0
data.to_excel(path, sheet_name = 'sheet number ' + str(n)) # save data in an excel worksheet

Now you can open the "fileName.xlsx" file and see that you stored data on a work sheet named "sheet number 0".

Next step is to generate other data and save them in other worksheets, without deleting the first one

book = load_workbook(path) #load excel file 
writer = pd.ExcelWriter(path, engine = 'openpyxl') # use pandas to write in the some excel file
writer.book = book # write what you saved before in order to avoid "overwriting"

for n in range(1, 10):
    data = np.random.randn(100)
    data = pd.DataFrame(data)
    data.to_excel(writer, sheet_name = 'sheet number ' + str(n) ) # iteratively save data on different excel worksheets

writer.save()
writer.close()

When you want to open and analyze the data you stored then I suggest you to type

xls = pd.ExcelFile(path)
df1 = xls.parse(0)

if you want to open data from the first worksheet or

xls = pd.ExcelFile(path)
df1 = xls.parse(1)

if you want to open data from the second one, you may even write

xls = pd.ExcelFile(path)
dataNames = xls.sheet_names
df1 = xls.parse(dataNames[0])

Answer 2

Are you looking for something like np.savetxt ?

If you want to append data to an existing file, you can open the file with append mode.

with open('data.txt', 'a') as f:
    np.savetxt(f, newdata)

Check out this post Appending a matrix to an existing file using numpy

You can read the text file using np.loadtxt

Answer 3

How about ndarray 's .tofile() method? To read use numpy.fromfile() .