bundling/executing python script + modules to a remote machine

Question

I have looked into other python module distribution questions. My need is a bit different (I think!, I am python newbie+)

I have a bunch of python scripts that I need to execute on remote machines. Here is what the target environment looks like;

• The machines will have base python run time installed
• I will have a SSH account; I can login or execute commands remotely using ssh
• i can copy files (scp) into my home dir
• I am NOT allowed to install any thing on the machine; the machines may not even have access to Internet
• my scripts may use some 'exotic' python modules -- most likely they won't be present in the target machine
• after the audit, my home directory will be nuked from the machine (leave no trace)

So what I like to do is:

• copy a directory structure of python scripts + modules to remote machine (say in /home/audituser/scripts). And modules can be copied into /home/audituser/scripts/pythhon_lib)
• then execute a script (say /home/audituser/scripts/myscript.py). This script will need to resolve all modules used from 'python_lib' sub directory.

is this possible? or is there a better way of doing this? I guess what I am looking is to 'relocate' the 3rd party modules into the scripts dir.

thanks in advance!

Answer 1

Are the remote machines the same as each other? And, if so, can you set up a development machine that's effectively the same as the remote machines?

If so, virtualenv makes this almost trivial. Create a virtualenv on your dev machine, use the virtualenv copy of pip to install any third-party modules into it, build your script within it, then just copy that entire environment to each remote machine.

There are three things that make it potentially non-trivial:

• If the remote machines don't (and can't) have virtualenv installed, you need to do one of the following:
  • In many cases, just copying a --relocatable environment over just works. See the documentation section on "Making Environments Relocatable".
  • You can always bundle virtualenv itself, and pip install --user virtualenv (and, if they don't even have pip , a few steps before that) on each machine. This will leave the user account in a permanently-changed state. (But fortunately, your user account is going to be nuked, so who cares?)
  • You can write your own manual bootstrapping. See the section on "Creating Your Own Bootstrap Scripts".
  • By default, you get a lot more than you need—the Python executable, the standard library, etc.
  • If the machines aren't identical, this may not work, or at least might not be as efficient.
  • Even if they are, you're still often making your bundle orders of magnitude bigger.
  • See the documentation sections on Using Virtualenv without bin/python, --system-site-packages, and possibly bootstrapping.
• If any of the Python modules you're installing also need C libraries (eg, libxml2 for lxml ), virtualenv doesn't help with that. In fact, you will need the C libraries to be almost exactly the same (same path, compatible version).

Three other alternatives:

• If your needs are simple enough (or the least-simple parts involve things that virtualenv doesn't help with, like installing libxml2 ), it may be easier to just bundle the .egg/.tgz/whatever files for third-party modules, and write a script that does a pip install --user and so on for each one, and then you're done.
• Just because you don't need a full app-distribution system doesn't mean you can't use one. py2app , py2exe , cx_freeze , etc. aren't all that complicated, especially in simple cases, and having a click-and-go executable to copy around is even easier than having an explicit environment.
• zc.buildout is an amazingly flexible and manageable tool that can do the equivalent of any of the three alternatives. The main downside is that there's a much, much steeper learning curve.

Answer 2

You can use virtualenv to create a self-contained environment for your project. This can house your own script, as well as any dependency libraries. Then you can make the env relocatable ( --relocatable ), and sync it over to the target machine, activate it, and run your scripts.

If these machines do have network access (not internet, but just local network), you can also place the virtualenv on a shared location and activate from there.

It looks something like this:

virtualenv --no-site-packages portable_proj
cd portable_proj/
source bin/activate
# install some deps
pip install xyz
virtualenv --relocatable .

Now portable_proj can be disted to other machines.