Loading numpy array from http response without saving a file

Question

I have a bunch of files containing numpy arrays at some url (eg, https://my_url/my_np_file.npy ) and I am trying to load them in my computer.

If I download the file manually, I can properly load the numpy array using np.load('file_path') . If I take the url reponse (using the code below), save the content to a file and then use np.load() , it also works.

response, content = http.request('https://my_url/my_np_file.npy')

If I try to load the array from the content string, I get the error bellow. This is likely because the np.load is interpreting the string input as a name for the file, rather than the data itself.

File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 370, in load fid = open(file, "rb") TypeError: file() argument 1 must be encoded string without null bytes, not str

Is there any way to load the array without having to save a file?

Answer 1

You are missing io.BytesIO to make the string appear like a file object to np.load !

The following is what you're looking for:

import requests
import io

response = requests.get('https://my_url/my_np_file.npy')
response.raise_for_status()
data = np.load(io.BytesIO(response.content))  # Works!

Answer 2

Can you add an example of the HTTP response? (from https://my_url/my_np_file.npy )

Maybe you could try np.fromstring

This is an example from the above reference.

>>> np.fromstring('1 2', dtype=int, sep=' ')
array([1, 2])
>>> np.fromstring('1, 2', dtype=int, sep=',')
array([1, 2])

Answer 3

I just worked out a workaround using shutil

import requests
import shutil
response = requests.get('http://path/to/test.npy', stream=True)

with open('haha.npy', 'wb') as fin:
    shutil.copyfileobj(response.raw, fin)

np.load('haha.npy') # Works!

This will first download the file itself though, but in an automatic fashion.