在AWS Glue中读取镶木地板文件

Question

I'm a AWS Glue newbie that is trying to read some parquet objects that I have in S3 but I fail by ClassNotFoundException. This is my attempt so far based on the minimal documentation of Glue:

import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.JsonOptions
import org.apache.spark.sql.SparkSession

val gc: GlueContext = new GlueContext(sc)

val spark_session : SparkSession = gc.getSparkSession

val source = gc.getSource("s3", JsonOptions(Map("paths" -> Set("s3://path-to-parquet"))))

val parquetSource = source.withFormat("parquet")

parquetSource.getDynamicFrame().show(1)

And the exception:

   18/06/11 13:39:11 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 266, ip-172-31-8-179.eu-west-1.compute.internal, executor 16): java.lang.ClassNotFoundException: Failed to load format with name parquet
    at com.amazonaws.services.glue.util.ClassUtils$.loadByFullName(ClassUtils.scala:28)
    at com.amazonaws.services.glue.util.ClassUtils$.getClassByName(ClassUtils.scala:43)
    at com.amazonaws.services.glue.util.ClassUtils$.newInstanceByName(ClassUtils.scala:54)
    at com.amazonaws.services.glue.readers.DynamicRecordStreamReader$.apply(DynamicRecordReader.scala:187)
    ...
Caused by: java.lang.ClassNotFoundException: parquet
    at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:82)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.amazonaws.services.glue.util.ClassUtils$$anonfun$1.apply(ClassUtils.scala:25)
    at com.amazonaws.services.glue.util.ClassUtils$$anonfun$1.apply(ClassUtils.scala:25)
    at scala.util.Try$.apply(Try.scala:192)
    at com.amazonaws.services.glue.util.ClassUtils$.loadByFullName(ClassUtils.scala:25)
    ... 28 more

Answer 1

I solved the issue. I had specified the wrong connectionType for 'getSource': it should be "parquet" and not "s3":

import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.JsonOptions
import org.apache.spark.sql.SparkSession

val gc: GlueContext = new GlueContext(sc)

val spark_session : SparkSession = gc.getSparkSession

val source = gc.getSource("parquet", JsonOptions(Map("paths" -> Set("s3://path-to-parquet"))))

source.getDynamicFrame().show(1)

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-parquet

Hopefully this helps somebody!