TypeError：“ JavaPackage”对象在PySpark，AWS Glue上不可调用

Question

I've learned Spark in Scala but I'm very new to pySpark and AWS Glue,
so I followed this official tutorial by AWS.
https://docs.aws.amazon.com/ja_jp/glue/latest/dg/aws-glue-programming-python-samples-legislators.html

I successfully created development endpoint,
connected to pyspark REPL via ssh and typed in these commands:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

glueContext = GlueContext(SparkContext.getOrCreate())

But on the last line, I got

>>> glueContext = GlueContext(SparkContext.getOrCreate())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/share/aws/glue/etl/python/PyGlue.zip/awsglue/context.py", line 44, in __init__
  File "/usr/share/aws/glue/etl/python/PyGlue.zip/awsglue/context.py", line 64, in _get_glue_scala_context
TypeError: 'JavaPackage' object is not callable

I also tried importing py4j manually, but it just didn't work.

How can I fix this?
Any little help will be appreciated.

Answer 1

Finally I solved by myself.
Looks like it was Glue/AWS specific issue, not spark or python.

After several trials, I got an error message that says "ListObject" operation has failed when starting Spark(pyspark) REPL.
ListObject is obviously the name of boto3's API call to access contents on S3.

So I checked its IAM role which had AWSGlueConsoleFullAccess with some S3Access included in it already, attached AmazonS3FullAccess policy to it, and the error disappeared.
Also, I made another glue-development-endpoint cluster and also there was no error on the new cluster either, even without S3FullAccess.

Maybe every time I wake up Spark on a glue cluster, the cluster automatically tries to fetch some update from some designated S3 bucket, and sometimes it got in trouble when the cluster was built just before some update release.