Import error on EMR after bootstrap action

Question

I'm running a bootstrap action on an EMR cluster and I'm checking the numpy version after I run through requirements text file with:

sudo python3 -c "try:
    import numpy
    print(\"numpy_version\", numpy.version.version)
except:
    pass"

I'm getting the correct version specified in the requirements text file (1.15.4); however, after the job starts running, i get:

ImportError: Matplotlib requires numpy>=1.15; you have 1.14.5

What exactly is happening in between the bootstrap action and the job steps that is reverting the version of numpy?

Answer 1

This indicates that the application provisioning downgrades the numpy's version

A workaround for this issue is by using a bootstrap action which launches another background process.

The background process should upgrades numpy's version and should run right after the application provisioning.