单元测试：使用 Python 模拟运行“aws s3 同步”的子进程

Question

My project needs to download quite a few files regularly before doing treatment on them. I tried coding it directly in Python but it's horribly slow considering the amount of data in the buckets.

I decided to use a subprocess running aws-cli because boto3 still doesn't have a sync functionality. I know using a subprocess with aws-cli is not ideal, but it really is useful and works extremely well out of the box.

One of the perks of aws-cli is the fact that I can see the progress in stdout , which I am getting with the following code:

def download_bucket(bucket_url, dir_name, dest):
"""Download all the files from a bucket into a directory."""
path = Path(dest) / dir_name
bucket_dest = str(os.path.join(bucket_url, dir_name))
with subprocess.Popen(["aws", "s3", "sync", bucket_dest, path], stdout=subprocess.PIPE, bufsize=1, universal_newlines=True) as p:
    for b in p.stdout:
        print(b, end='')

if p.returncode != 0:
    raise subprocess.CalledProcessError(p.returncode, p.args)

Now, I want to make sure that I test this function but I am blocked here because:

1. I don't know the best way to test this kind of freakish behavior:
   • Am I supposed to actually create a fake local s3 bucket so that aws s3 sync can hit it?
   • Am I supposed to mock the subprocess call and not actually call my download_bucket function?

Until now, my attempt was to create a fake bucket and to pass it to my download_bucket function. This way, I thought that aws s3 sync would still be working, albeit locally:

def test_download_s3(tmpdir):
tmpdir.join(f'frankendir').ensure()
with mock_s3():
    conn = boto3.resource('s3', region_name='us-east-1')
    conn.create_bucket(Bucket='cool-bucket.us-east-1.dev.000000000000')

    s3 = boto3.client('s3', region_name="us-east-1")
    s3.put_object(Bucket='cool-bucket.us-east-1.dev.000000000000', Key='frankendir', Body='has no files')

    body = conn.Object('cool-bucket.us-east-1.dev.000000000000', 'frankendir').get()[
        'Body'].read().decode("utf-8")

    download_bucket('s3://cool-bucket.us-east-1.dev.000000000000', 'frankendir', tmpdir)

    #assert tmpdir.join('frankendir').join('has not files').exists()
    assert body == 'has no files'

But I get the following error fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.

My questions are the following:

1. Should I continue to pursue this creation of a fake local s3 bucket?
   • If so, how am I supposed to get the credentials to work?
2. Should I just mock the subprocess call and how?
   • I am having a hard time understanding how mocking works and how it's supposed to be done. From my understanding, I would just fake a call to aws s3 sync and return some files?
3. Is there another kind of unit test that would be enough that I didn't think of?
   • After all, I just want to know if when I transmit a well-formed s3://bucketurl , a dir in that bucket and a local dir , the files contained within the s3://bucketurl/dir are downloaded to my local dir .

Thank you for your help, I hope that I am not all over the place.

Answer 1

A much better approach is to use moto when faking / testing s3. You can check out their documentation or look at a test code example I did: https://github.com/pksol/pycon-go-beyond-mocks/blob/main/test_s3_fake.py .

If you have a few minutes, you can view this short video of me explaining the benefits of using moto vs trying to mock.