將文件和對象寫入amazon s3

Question

我正在使用amazon S3將動態生成的文件分發給S3。

在本地服務器上，我可以使用

destination = open(VIDEO_DIR + newvideo.name, 'wb+')

將生成的視頻存儲到VIDEO_DIR.newvideo.name位置

是否有可行的方法將VIDEO_DIR更改為S3端點位置。 那么動態生成的視頻可以直接寫入S3服務器嗎？

另一個問題是：有沒有可行的方法直接將對象寫入S3？ 例如，一個chunklet=Chunklet() ，如何直接將這個chunklet=Chunklet()對象寫入S3服務器？

我可以先創建一個本地文件並使用S3 API。 例如，

mime = mimetypes.guess_type(filename)[0]
k = Key(b)
k.key = filename
k.set_metadata("Content-Type", mime)
k.set_contents_from_filename(filename)
k.set_acl('public-read')

但我想提高效率。 使用Python。

Answer 1

使用boto庫訪問S3存儲。 您仍然必須先將數據寫入（臨時）文件，然后才能發送它，因為尚未實現流寫入方法。

我將使用上下文管理器來解決這個限制：

import tempfile
from contextlib import contextmanager

@contextmanager
def s3upload(key):
    with tempfile.SpooledTemporaryFile(max_size=1024*10) as buffer:  # Size in bytes
        yield buffer  # After this, the file is typically written to
        buffer.seek(0)  # So that reading the file starts from its beginning
        key.set_contents_from_file(buffer)

將其用作上下文托管文件對象：

k = Key(b)
k.key = filename
k.set_metadata("Content-Type", mime)

with s3upload(k) as out:
    out.write(chunklet)

Answer 2

Martijn的解決方案很棒，但它會強制您在上下文管理器中使用該文件（您不能執行out = s3upload(…)並print >> out, "Hello" ）。 以下解決方案類似地工作（內存存儲直到一定大小），但作為上下文管理器和常規文件（您可以with S3WriteFile(…)和out = S3WriteFile(…); print >> out, "Hello"; out.close() ）：

import tempfile
import os

class S3WriteFile(object):
    """
    File-like context manager that can be written to (and read from),
    and which is automatically copied to Amazon S3 upon closing and deletion.
    """

    def __init__(self, item, max_size=10*1024**2):
        """
        item -- boto.s3.key.Key for writing the file (upon closing). The
        name of the object is set to the item's name (key).

        max_size -- maximum size in bytes of the data that will be
        kept in memory when writing to the file. If more data is
        written, it is automatically rolled over to a file.
        """

        self.item = item

        temp_file = tempfile.SpooledTemporaryFile(max_size)

        # It would be useless to set the .name attribute of the
        # object: when using it as a context manager, the temporary
        # file is returned, which as a None name:
        temp_file.name = os.path.join(
            "s3://{}".format(item.bucket.name),
            item.name if item.name is not None else "<???>")

        self.temp_file = temp_file

    def close(self):
        self.temp_file.seek(0)
        self.item.set_contents_from_file(self.temp_file)
        self.temp_file.close()

    def __del__(self):
        """
        Write the file contents to S3.
        """
        # The file may have been closed before being deleted:
        if not self.temp_file.closed:
            self.close()

    def __enter__(self):
        return self.temp_file

    def __exit__(self, *args, **kwargs):
        self.close()
        return False

    def __getattr__(self, name):
        """
        Everything not specific to this class is delegated to the
        temporary file, so that objects of this class behave like a
        file.
        """
        return getattr(self.temp_file, name)

（實現說明：不是將很多東西委托給self.temp_file以便生成的類的行為就像一個文件，繼承SpooledTemporaryFile原則上會工作。但是，這是一個舊式的類，所以不調用__new__() ， ，據我所見，無法設置臨時數據的非默認內存大小。）