如何在 S3 中模擬 ZipFile object？

Question

我想測試將檔案上傳到 S3 存儲桶。 此存檔已解壓縮，因此我也想測試提取的 JSON 文件。 這是我要測試的內容：

def get_files_from_s3(bucket_name, s3_prefix):
    files = []
    s3_resource = boto3.resource("s3")
    bucket = s3_resource.Bucket(bucket_name)
    response = bucket.objects.filter(Prefix=s3_prefix)
    for obj in response:
        if obj.key.endswidth('.zip'):
            buffer = BytesIO(obj.get()["Body"].read())
            print("Before unzipping the file")
            z = ZipFile(buffer)
            for filename in z.namelist():
                print("After unzipping the file")
                if filename == "files/file1.json":
                    for dict in self.parse_json(z, filename):
                    # store some data from the json
            data.append(...)
    return data

def parse_json(obj, file):
    with obj.open(file, "r") as f:
        data = json.loads(f.read())
    return data["some_key"]

這是我在這里得到上一個問題的答案后在單元測試中嘗試過的：

from moto import mock_s3

@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials, to ensure we're not touching AWS directly"""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'


@mock_s3
def test_get_files_from_s3(self, aws_credentials):
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(self.bucket_name)
    # Create the bucket first, as we're interacting with an empty mocked 'AWS account'
    bucket.create()

    # Create some example files that are representative of what the S3 bucket would look like in production
    client = boto3.client('s3')
    client.put_object(Bucket=self.bucket_name, Key=s3_prefix + "file.zip", Body=open('local_file.zip', 'r'))
    client.put_object(Bucket=self.bucket_name, Key=s3_prefix + "file.nonzip", Body="...")

    # Retrieve the files again using whatever logic
    files = module.get_files_from_s3(BUCKET_NAME, S3_PREFIX)
    self.assertEqual(['file.zip'], files)

所以我關心的是如何在傳遞給 moto 時模擬 ZipFile object？ 因為當我使用要發送到 S3 客戶端的實際 zip 文件傳遞Body參數時，我發現了這個錯誤： E UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9b in position 10: invalid start byte

另一個問題是如何將 object 傳遞給另一個 function 像parse_json() ？

謝謝

Answer 1

Body參數只接受字節，這意味着open方法需要使用rb參數返回文件的字節內容：

client.put_object(Bucket=self.bucket_name, Key=s3_prefix + "file.zip", Body=open('local_file.zip', 'rb'))

請注意，這不是 Zipfile/mocking/moto 問題，而是 boto3 限制。
如果第二個參數只是r ，則內容將作為字符串返回。 嘗試上傳時，AWS 將失敗並顯示以下錯誤消息： TypeError: Unicode-objects must be encoded before hashing