IncompleteReadError when reading file from S3 on AWS Lambda

Question

When reading a file from S3 on AWS Lambda I get IncompleteReadError . When I try it locally it works just fine. This is only happening on Python3.6 and works fine on Python3.7 - I need to use Python3.6 however. I also tried to use resource instead of client but got the same error

Traceback (most recent call last):
  File "/var/task/function.py", line 141, in handler
    i = d.read()
  File "/var/runtime/botocore/response.py", line 82, in read
    self._verify_content_length()
  File "/var/runtime/botocore/response.py", line 134, in _verify_content_length
    expected_bytes=int(self._content_length))
botocore.exceptions.IncompleteReadError: 0 read, but total bytes expected is 36678.

The area of code where it fails is here:

client = boto3.client('s3')        
get_json_file = client.get_object(
    Bucket=os.environ['S3_BUCKET'],
    Key="{0}".format(file_name),
)

d = get_json_file.get('Body')
i = d.read()
data = json.loads(i)

Answer 1

I faced the same problem, I guess the problem arises because of the large size of the stream. I used _raw_stream to solve this

Try doing this:

client = boto3.client('s3')        
object = client.get_object(
    Bucket=os.environ['S3_BUCKET'],
    Key="{0}".format(file_name),
)

raw_data = object['Body']._raw_stream.data.decode("utf-8")
data = json.loads(raw_data)