How to automatically install required packages from a python script as necessary?

Question

Is there anything in python or linux what basically instructs the system to "install whatever is necessary". Basically I find it annoying to install python packages for each new script/system/server that I work on. Each time I end up doing a sudo pip or an apt-get or dnf anyway. Why not automate that within the script itself. Whereever a 'no package found' error crops up, pass the library name to the install statement. Is this there ?

PS: I know docker exists, but am talking at a python/script level or a direct system level for purely execution purposes.

Thanks

Answer 1

How to automatically install required packages from a python script as necessary?

Let's assume that your Python script is example.py :

import os
import time
import sys
import fnmatch
import requests
import urllib.request
from bs4 import BeautifulSoup
from multiprocessing.dummy import Pool as ThreadPool 
print('test')

You can use pipreqs to automatically generate a requirements.txt file based on the import statements that the Python script(s) contain. To use pipreqs , assuming that you are in the directory where example.py is located:

pip install pipreqs
pipreqs .

It will generate the following requirements.txt file:

requests==2.23.0
beautifulsoup4==4.9.1

which you can install with:

pip install -r requirements.txt

Answer 2

The best way I know is, create a file requirements.txt list out all the packages name in it and install it using pip

pip install -r requirements.txt

Example requirements.txt :

BeautifulSoup==3.2.0
Django==1.3
Fabric==1.2.0
Jinja2==2.5.5
PyYAML==3.09
Pygments==1.4
SQLAlchemy==0.7.1
South==0.7.3
amqplib==0.6.1
anyjson==0.3
...

Answer 3

You can use setuptools to install dependencies automatically when you install your custom project on a new machine. Requirements file works just fine if all you want to do is to install a few PyPI packages.

Here is a nice comparison between the two. From the same link you can see that if your project has two dependent packages A and B , all you have to include in your setp.py file is a line

install_requires=[
   'A',
   'B'
] 

Of course, setuptools can do much more. You can include setups for external libraries (say C files), non PyPI dependencies, etc. The documentation gives a detailed overview on installing dependencies. There is also a really good tutorial on getting started with python packaging.

From their example, a typical setup.py file would look like this.

from setuptools import setup

setup(name='funniest',
      version='0.1',
      description='The funniest joke in the world',
      url='http://github.com/storborg/funniest',
      author='Flying Circus',
      author_email='flyingcircus@example.com',
      license='MIT',
      packages=['funniest'],
      install_requires=[
          'markdown',
      ],
      zip_safe=False)

In conclusion, it is so simple to get started with setuptools . This package can make it fairly easy to migrate your code to a new machine.

Answer 4

pip3 install pipreqs

pip3 install -U --user pipreqs

mv requirements.txt  requirements.txt.orig
pipreqs --force ./

and

cat requirements.txt

Keras==2.2.4 
scipy==1.3.0 
numba==0.44.1 
imgaug==0.2.9 
opencv_python==4.1.0.25

BUT, some modules needs binary compilation (where you don't have libraries) and some could be installed by system package manager with different names.

Answer 5

Automatic requirements.txt updating approach

I'm not really sure about auto installing what is necessary, but it you stop on using requirements.txt , there are 3 approaches:

1. Generate requirements.txt after development, when we want to deploy it. It is performed by pip freeze > requirements.txt or pipreqs for less messy result.
2. Add every module to requirements.txt manually after each install.
3. Install manager that will handle requirements.txt updates for us.

There are many answers for the 1-st option on stackoverflow, the 2-d option is self-explanatory, so I would like to describe the 3-d approach. There is a library called to-requirements.txt . To install it type this:

pip install to-requirements.txt  # Pip install to requirements.txt

If you read the whole command at once you would see, what it does. After installing you should setup it. Run:

requirements-txt setup

It overrides the pip scripts so that each pip install or pip uninstall updates the requirements.txt file of your project automatically with required versions of packages. The overriding is made safely, so that after uninstalling this package the pip will behave ordinary.

And you could customize the way it works. For example, disable it globally and activate it only for the required directories, activate it only for git repositories, or allow / disallow to create requirements.txt file if it does not exist.

Links:

1. Documentation - https://requirements-txt.readthedocs.io/en/latest/
2. GitHub - https://github.com/VoIlAlex/requirements-txt
3. PyPI - https://pypi.org/project/to-requirements.txt/