Great expectations installation to AWS EMR

Question

I tried to use great expectations for data quality purpose

I am running my jobs in AWS EMR cluster and I am trying to launch great expectations job on AWS EMR as well

I have bootstrap script for installation dependencies on a cluster. It looks like this

#!/bin/bash
sudo yes | sudo yum install python3-devel
sudo python3 -m pip install --upgrade pip
sudo python3 -m pip install cython

sudo python3 -m pip install boto3==1.26.37
sudo python3 -m pip install great-expectations==0.15.36

I saw that all dependencies was installed correctly based on log outputs, but then job started I got the following error

ImportError: this version of pandas is incompatible with numpy < 1.17.3
your numpy version is 1.16.5.
Please upgrade numpy to >= 1.17.3 to use this pandas version

I tried to uninstall numpy and install it manually via pip in bootstrap script like this but it didn't help

sudo python3 -m pip uninstall --yes numpy

I don't understand why it happens

Answer 1

sudo python3 -m pip install numpy==1.17.3

Answer 2

Usage of EMR of newer version solved problem.