如何将 .npy 文件作为 numpy 数组加载到 Google Colab 上的虚拟机中

Question

I have some datasets and labels that are basically numpy saved files with the extension .npy.

I have saved train.npy and train_labels.npy in my google drive.

While using Google Colab, I have to use that data. I am able to find folder and the id of the data files in my drive. How do I load those data files into the memory of my virtual machine that Google Colab uses?

Answer 1

Solved it.

First do the simple authentication as stated in the doc

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# 1. Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

I created some helper function which gets the file id if you know the file name and folder id where the file is present on drive. Folder id is the last part of the link in drive.google.com/../../folders/

def get_file_from_drive(folder_id, file_name):
  file_list = drive.ListFile({'q': "'" + folder_id + "' in parents and 
trashed=false"}).GetList()
  for file in file_list:
    if file['title'] == file_name:
      return file['id']

def upload_file_to_drive(file_name, file_data):
  uploaded = drive.CreateFile({'title': file_name})
  uploaded.SetContentString(file_data)
  uploaded.Upload()
  print('Uploaded file with ID {}'.format(uploaded.get('id')))

drive_folder_id = '<Folder ID>'

This function uploads the file from google drive into the virtual system that colab allows you to use.

def upload_data_system():
  downloaded = drive.CreateFile({'id': get_file_from_drive(drive_folder_id, 'train.npy')})
  downloaded.GetContentFile('train.npy') 

  downloaded = drive.CreateFile({'id': get_file_from_drive(drive_folder_id, 'train_labels.npy')})
  downloaded.GetContentFile('train_labels.

upload_data_system()

Viola! Your files are uploaded to the file system and can be loaded into memory using simple python as it was done locally. To verify, run this on colab. You should see your files

import os
from os import listdir

for f in os.listdir('.'):
  if os.path.isfile(f):
    print(f)

Now load your numpy file as np.load(path_to_file_in_filesystem)