AWS EMR 上的 python 3 和 tensorflow

Question

For EMR AWS has tensorflow 1.9 as part of the software stack for release 5.17. I have my own bootstrap script to install python 3.6 and tensorflow 1.9, I took out the tensorflow installation - but it didn't work -- I get on the master node, run python3 , I get into my new python 3.6 but there is no tensorflow in my installation. I must have installed a new python, I guess my question is how do I use the native python3 installation with tensorflow on AWS EMR, with spark - and am I lucky enough for that python3 to by 3.6?

This is my bootstrap script:

#!/usr/bin/env bash

sudo yum -y upgrade
sudo yum -y install git autoconf automake libevent-devel python36.x86_64 python36-pip.noarch python36-devel.x86_64

sudo python36 -m pip install --upgrade pip

sudo python36 -m pip install --upgrade wheel cython

sudo python36 -m pip install py4j jupyter ipython pandas scipy pyyaml scikit-learn ipykernel matplotlib seaborn h5py configobj ujson

echo -e "\n\n" >> ~/.bashrc
echo 'export PYSPARK_PYTHON=/usr/bin/python36' >> ~/.bashrc

I guess I'll try it without a bootstrap script, maybe it will just work?

Answer 1

From AWS :

Amazon EMR release versions 4.6.0-5.19.0: Python 3.4 is installed on the cluster instances. Python 2.7 is the system default.

Amazon EMR release versions 5.20.0 and later: Python 3.6 is installed on the cluster instances. Python 2.7 is the system default.

You can install python3.6 using the following bootstrap action. Also, I recommend using virtualenv to run your python scripts and keeping the required libraries info on s3 using requirements.txt file.

#!/bin/bash -xe
sudo yum install -y python36 python36-devel postgresql-devel unixODBC-devel # For pyodbc, psycopg2
virtualenv --system-site-packages /home/hadoop/workspace -p /usr/bin/python3.6 # Install virualenv
source /home/hadoop/workspace/bin/activate
aws s3 cp s3://<bucket>/requirements.txt /home/hadoop/ # Keep your required pip freeze info (tensorflow, etc...) on s3
pip install -r /home/hadoop/requirements.txt # Install your packages
# Run your scripts during main execcution using /home/hadoop/workspace/bin/python3

Answer 2

It looks like right now (sept 2018) AWS EMR is at python 3.4. Without a bootstrap script, I can run python3 and get their tensorflow.