在 EC2 实例 (AWS) 上使用 Bash 脚本导入错误加载模块

Question

I am trying to run a bash script through the user_data argument of the spot_fleet_request method. The script executes all the installations and connects to the filesystem, but when it runs the python file I need to run it gives me the following error:

Traceback (most recent call last):
  File "/home/ec2-user/efs/dir/create_lvl_output.py", line 8, in <module>
    from modeling import generators
ImportError: No module named modeling

Where modeling is a folder with a script I am importing "modeling/generators.py". I have attempted to assign the directory to the PYTHONPATH environmental variable as suggested in this post , here is my code:

#!/bin/bash
export PYTHONPATH=/home/ec2-user/efs/Day-Trader/day_trader/
echo $PYTHONPATH

...mount efs
...installs

python /home/ec2-user/efs/dir/create_lvl_output.py > levelset.txt

but it continues to fail with the same error. I tried this solution with each of the following variations and they all failed to solve the import issue.

echo "export PYTHONPATH=/home/ec2-user/efs/dir/" >> ~/.bashrc
. ~/.bashrc      #--- failed to import, set PYTHONPATH variable correctly     

echo "export PYTHONPATH=/home/ec2-user/efs/dir/" >> /home/ec2-user/.bashrc
. /home/ec2-user/.bashrc   #--- failed to import, set PYTHONPATH variable correctly but got following error as well: /home/ec2-user/.bashrc: line 2: /root/.dlamirc: No such file or directory

I also made sure that the script actually worked. If I copy and paste the script directly into the linux terminal when I've logged in a ec2-user (ie $HOME=/home/ec2-user/) and run perfectly and produced all the right output. So I thought that if I changed the user at the top of the bash script as suggested in this post it might work but the following commands still ran everything that followed as root:

su -l "ec2-user" , su - ec2-user

and I tried running the python script using

su ec2-user -c 'python /home/ec2-user/efs/dir/create_lvl_output.py'

but it didn't work either. I'm out of ideas. Please help.

Answer 1

I think the recommended method is of course to try to package your code as a proper module and install it prior with pip, either at the user/system level or a pythonenv style environment. This of course, can be highly difficult and time consuming depending on how much code you have to change to accommodate this, and its recommended that you try to do this at the beginning of making projects to avoid this headache.

Short of the doing the nontrivial amount of work that would be though, I actually have encountered this specific error you're having on Amazon's EC2 LTS ubuntu. No idea what causes it, but its like python just side steps the normal environment settings, and doesn't work as you expect. In my case, I just went for the dirty fix of:

import os

os.environ['PYTHONPATH'] = '{}:{}'.format('my/addition/to/pythonpath', os.environ.get('PYTHONPATH', ''))
print(os.environ['PYTHONPATH'])
# prints my/addition/to/pythonpath:existing/things/on/pythonpath

so for your usecase, try:

import os

os.environ['PYTHONPATH'] = '{}:{}'.format('/home/ec2-user/efs/Day-Trader/day_trader/', os.environ.get('PYTHONPATH', ''))
print(os.environ['PYTHONPATH'])
# prints /home/ec2-user/efs/Day-Trader/day_trader/:

you can of course do some pretty nasty things with this, such as:

python -c "import os; os.environ['PYTHONPATH'] = '/my/addition/to/pythonpath'; import mymodule; mymodule.doThing()'

if for some reason you dislike everyone who has the displeasure of working with this. Jokes aside, I really did use things like this in production, if you every figure out what the hell causes these env issues, I'd be interested to know even though I no longer work on the machines that had these issues.