无法使用python在谷歌驱动器上上传大文件

Question

我正在尝试使用以下代码通过 Google API 将文件上传到 Google Drive

import httplib2
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools
try:
    import argparse
    flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
    flags = None

SCOPES =['https://www.googleapis.com/auth/drive','https://www.googleapis.com/auth/drive.file','https://www.googleapis.com/auth/drive.appdata', 'https://www.googleapis.com/auth/drive.apps.readonly']
store = file.Storage('scope.json')
creds = store.get()
if not creds or creds.invalid:
    flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
    creds = tools.run_flow(flow, store, flags) if flags else tools.run(flow, store)
    DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
else:
    credentials = creds
    http = credentials.authorize(httplib2.Http())
    DRIVE = discovery.build('drive', 'v3', http=http)

FILES = (
    ('/home/vkm/mayur/Demo_Google_API.zip', 'application/vmd.google-apps.document'),
)

for filename, mimeType in FILES:
    metadata = {'name': filename}
    if mimeType:
        metadata['mimeType'] = mimeType
    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
    if res:
        print('Uploaded "%s" (%s)' % (filename, res['mimeType']))

我可以上传小文件，但是当我尝试使用 8GB 的​​文件时，它给出了 MemorryErro。请找到我收到的错误消息。

Traceback (most recent call last):
  File "demo.py", line 46, in <module>
    res = DRIVE.files().create(body=metadata, media_body=filename).execute()
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/discovery.py", line 853, in method
    payload = media_upload.getbytes(0, media_upload.size())
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 482, in getbytes
    return self._fd.read(length)
MemoryError

Answer 1

Vikram的评论揭示了mhawke的答案中的一个问题：需要根据以下返回值调用next_chunk ：

request = DRIVE.files().create(body=metadata, media_body=media)

不在request.execute()的返回值上。

这是经过验证的Python代码段，该代码可处理最大10MB的Google云端硬盘帐户文件：

# Upload some file that just happens to be binary (we
# don't care about metadata, just upload it without
# translation):
the_file_to_upload = 'some_binary_file'
metadata = {'name': the_file_to_upload}
# Note the chunksize restrictions given in
# https://developers.google.com/api-client-library/python/guide/media_upload
media = MediaFileUpload(the_file_to_upload,
                        chunksize=1024 * 1024,
                        # Not sure whether or not this mimetypes is necessary:
                        mimetype='text/plain',
                        resumable=True)
request = drive_service.files().create(body=metadata, media_body=media)
response = None
while response is None:
    status, response = request.next_chunk()
    if status:
        print("Uploaded %d%%." % int(status.progress() * 100))
print("Upload of {} is complete.".format(the_file_to_upload))

这是一段Python代码，可下载相同的文件，但下载到不同的文件，因此我可以使用sha1sum来验证文件是否已被Google Drive移入移出。

# Verify downloading works without translation:
request = drive_service.files().get_media(fileId=response['id'])
# Use io.FileIO. Refer to:
# https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.MediaIoBaseDownload-class.html
out_filename = the_file_to_upload + ".out"
fh = io.FileIO(out_filename, mode='wb')
downloader = MediaIoBaseDownload(fh, request, chunksize=1024 * 1024)
done = False
while done is False:
    status, done = downloader.next_chunk()
    if status:
        print("Download %d%%." % int(status.progress() * 100))
print("Download Complete!")

Answer 2

您可以使用可恢复的媒体上传来上传文件。 这将以块的形式发送文件，并且不应使内存最大用尽，我认为这种情况正在发生，因为您的客户端正在尝试一次发送整个文件。

为此，您需要将MediaFileUpload对象传递给create()方法，在该方法中， resumable标志设置为True 。 您也可以选择设置chunksize 。

metadata = {'name': filename}
media = MediaFileUpload(filename, mimetype=mimetype, resumable=True)

request = DRIVE.files().create(body=metadata, media_body=media)
response = None
while response is None:
  status, response = request.next_chunk()
  if status:
    print "Uploaded %d%%." % int(status.progress() * 100)
print "Upload Complete!"

尝试减少chunksize如果需要的话。

Answer 3

使用 python 将大文件上传到 Google Drive 的最简单方法是添加 resumable=True

from googleapiclient.http import MediaFileUpload    
media = MediaFileUpload(filename, resumable=True)