Is aws CLI faster than using boto3?

Question

I have packages stored in s3 bucket. I need to read metadata file of each package and pass the metadata to program. I used boto3.resource('s3') to read these files in python. The code took few minutes to run. While if I use aws cli sync , it downloads these metafiles much faster than boto. My guess was that if I do not download and just read the meta files, it should be faster. But it isn't the case. Is it safe to say that aws cli is faster than using boto?

Answer 1

The AWS Command-Line Interface (CLI) actually uses boto itself, so it is not 'faster'.

However, the aws s3 sync command uses threading to copy multiple files simultaneously, so the copy operation takes less elapsed time.

Answer 2

In my experience I have not seen CLI to be faster than boto. As helloV mentioned CLI was written by pros and is maintained at optimal levels wheras boto is written by users junior to master.

If you need just metadata and want to be fast with a script using boto I highly recommend using client instead of resource and a method named head_object which does a read but not a download and metadata is retrievable in the json response structure.

Answer 3

It's true that the AWS CLI uses boto, but the cli is not a thin wrapper, as you might expect. When it comes to copying a tree of S3 data (which includes the multipart chunks behind a single large file), it is quite a lot of logic to make a wrapper that is as thorough and fast, and that does things like seamlessly pick up where a partial download has left off, or efficiently sync down only the changed data on the server.

The implementation in the awscli code is more thorough than anything in the Python or Java SDKs, as far as I have seen. I've seen several developers who were too proud to call the CLI from their code, but zero thus far all such attempts I have seen have failed to measure up. Love to see a counter example, though.