Python: Reading fortran file from url

Question

I would like to do the following in Python 3: Read in a FortranFile, but from an URL rather than a local file. The reason is that for my concrete example there are a lot of files and I want to avoid having to download them all first.

I have managed to

a) read in a simple.txt file from an URL

import urllib
from urllib.request import urlopen
url='http://www.deus-consortium.org/deus-library/filelist/deus_file_list_501.txt'
data=urllib.request.urlopen(url)
i=0
for line in data: # files are iterable
    print(i,line)
    i+=1
#alternative: data.read()

b) read in a local FortranFile (binary little endian unformated Fortran file) like this:

The file is from: http://www.deus-consortium.org/deus-library/efiler1/Babel_le/boxlen648_n2048_lcdmw7/post/fof/output_00090/fof_boxlen648_n2048_lcdmw7_masst_00000

from scipy.io import FortranFile
filename='../../Downloads/fof_boxlen648_n2048_rpcdmw7_masst_00000'
ff = FortranFile(filename, 'r')
nhalos=ff.read_ints(dtype=np.int32)[0]
print('number of halos in file',nhalos)

Is there any way to avoid downloading and reading FortranFiles directly from the URL? I tried

import urllib
from urllib.request import urlopen
url='http://www.deus-consortium.org/deus-library/efiler1/Babel_le/boxlen648_n2048_lcdmw7/cube_00090/fof_boxlen648_n2048_lcdmw7_cube_00000'
pathname = urllib.request.urlopen(url)  
ff = FortranFile(pathname, 'r')
ff.read_ints()

gives "OSError: obtaining file position failed". pathname.read() doesn't work either because it's a fortran file.

Any ideas? Thanks in advance!

Answer 1

Maybe you can use tempfile module to download and read the data?

For example:

import urllib
import tempfile
from scipy.io import FortranFile
from urllib.request import urlopen

url='http://www.deus-consortium.org/deus-library/efiler1/Babel_le/boxlen648_n2048_lcdmw7/cube_00090/fof_boxlen648_n2048_lcdmw7_cube_00000'

with tempfile.TemporaryFile() as fp:
    fp.write(urllib.request.urlopen(url).read())
    fp.seek(0)

    ff = FortranFile(fp, 'r')
    info = ff.read_ints()
    print(info)

Prints:

[12808737]