Accessing delta lake through Pyspark on EMR notebooks
Question
I have a query with respect to using external libraries like delta-core over AWS EMR notebooks. Currently there isn't any mechanism of installing the delta-core libraries through pypi packages. The available options include.
- Launching out pyspark kernel with --packages option
- The other option is to change the packages option in the python script through os configuration, but I don't see that it is able to download the packages and I still get import error on import delta.tables library.
- Third option is to download the JARs manually but it appears that there isn't any option on EMR notebooks.
Has anyone tried this out before?
1 answers
solution1
0 2020-06-26 00:38:46
- You can download the jars while creating EMR using bootstrap scripts.
- You can place the jars in s3 and pass it to pyspark with --jars option
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Add Delta Lake packages to AWS EMR Notebook
Error downloading Jars related to Delta Lake from pyspark packages
running pyspark script on EMR
Can we delete latest version of delta table in the delta lake?
IAM Role does not exist for EMR Notebooks
Running pySpark in Jupyter notebooks - Windows
loop through tables in databricks warehouse and extract certain values into another delta table with pyspark
Jupyter Notebook running through vscode jupyter server getting ModuleNotFoundError: No module named from pyspark on Amazon EMR
Amazon EMR Pyspark Module not found
pyspark submit command on AWS EMR
Related Tags