Convert .xsf (or .dat) file to array with np.loadtxt Python

Question

I have been checking some guides and also some of the questions that have been posted here but nothing has been working for me so far. I have an.xsf file which contains the first 57 lines as general instructions and then ca. 3*10^6 numbers. I want to load those numbers into a np.array and figured that the command

data = np.loadtxt('filename.xsf', skiprows = 57)

would do the trick.

That actually does not work because the data between line 58 and 531509 are organised as following

0.362077E+02  0.960500E+00  0.600950E+00  0.901461E-01  0.478295E-02  0.710280E-01

whereas the last line only contains one element. The error I get is

ValueError: Wrong number of columns at line 531510

I figured then to specify a delimiter (the double space)

data = np.loadtxt('filename.xsf', delimiter ='  ' ,skiprows = 57)

this results in the inability of reading the file.

From my understanding my first attempt results in something which is not an array of floats but rather an array where every element is a list (taken from each line as a whole) of floats. Beeing the last line of the file a single number it does not match the format of the rest of the array. In the second case scenario I am struggling with the definition of the delimiter.

I know this is a often asked and answered question, but none of the methods i tried has been working. I am hence trying to provide as much contest as possible as to my problem. Thanks to everyone who is willing to contribute

Answer 1

It took me some time but I have seem to been able to find an answer to my question... which I am posting in order to get possible corrections

1- I have converted my file to a csv (probably not necessary)

2-

import itertools

data = []
with open('filename.csv') as f:
    for line in f:
        data.append(line.strip().split(','))
#this returns a list of lists each of which is a line from the file

data= list(itertools.chain.from_iterable(data))
#merges the list of list into a single list