在AWS Glus pyspark作业中从s3加载JSON

Question

I am trying to retrieve a JSON file from an s3 bucket inside a glue pyspark script.

I am running this function in the job inside aws glue:

def run(spark):
    s3_bucket_path = 's3://bucket/data/file.gz'

    df = spark.read.json(s3_bucket_path)
    df.show()

After this I am getting: AnalysisException: u'Path does not exist: s3://bucket/data/file.gz;'

I searched for this issue and did not find anything that would be similar enough to infer where is the issue. I think there might be permission issues accessing the bucket, but then the error message should be different.

Answer 1

Here You can Try This :

    s3 = boto3.client("s3", region_name="us-west-2", aws_access_key_id=" 
        ", aws_secret_access_key="")
    jsonFile = s3.get_object(Bucket=bucket, Key=key)
    jsonObject = json.load(jsonFile["Body"])

where Key = full path to your file in bucket

and use this jsonObject in spark.read.json(jsonObject)