您能否在 AWS Glue 中使用 PySpark 而不是 Glue PySpark？

Question

I find Glue PySpark has it's own little twist to everything like is 'select' is 'select_fields' in Glue PySpark. How can I use spark instead of the Glue version?

Answer 1

You can just use the SparkSession directly instead of the GlueContext wrapper:

from pyspark.context import SparkContext
from awsglue.context import GlueContext

sc = SparkContext()
gc = GlueContext(sc.getOrCreate())
spark = gc.spark_session

df = spark.read.format(...).load(...)

df.select("*").show()