如何在 AWS Glue 中导入 Spark 包？

Question

I would like to use the GrameFrames package, if I were to run pyspark locally I would use the command:

~/hadoop/spark-2.3.1-bin-hadoop2.7/bin/pyspark --packages graphframes:graphframes:0.6.0-spark2.3-s_2.11

But how would I run a AWS Glue script with this package? I found nothing in the documentation...

Answer 1

You can provide a path to extra libraries packaged into zip archives located in s3.

Please check out this doc for more details

Answer 2

It's possible to using graphframes as follows:

Download the graphframes python library package file eg from here . Unzip the .tar.gz and then re-archive to a .zip . Put somewhere in s3 that your glue job has access to

When setting up your glue job:

• Make sure that your Python Library Path references the zip file
• For job parameters, you need {"--conf": "spark.jars.packages=graphframes:graphframes:0.6.0-spark2.3-s_2.11"}

Answer 3

Every one looking for an answer please read this comment..

In order to use an external package in AWS Glue pySpark or Python-shell:

1) Clone the repo from follwing url.. https://github.com/bhavintandel/py-packager/tree/master

git clone git@github.com:bhavintandel/py-packager.git

cd py-packager

2) Add your required package under requirements.txt. For ex.,

pygeohash

Update the version and project name under setup.py. For ex.,

VERSION = "0.1.0"

PACKAGE_NAME = "dependencies"

3) Run the follwing "command1" to create .zip package for pyspark OR "command2" to create egg files for python-shell..

command1:

sudo make build_zip

Command2:

sudo make bdist_egg

Above commands will generate packae in dist folder.

4) Finally upload this pakcage from dist directory to S3 bucket. Then goto AWS Glue Job Console, edit job, find script libraries option, click on folder icon of "python library path" .. then select your s3 path.

finally use in your glue script:

import pygeohash as pgh

Done!