Organizing external dependencies in AWS lambda

Question

I am relatively new to python and even newer to Lambda. I have created a lambda function that requires some external dependencies ( elasticsearch and elasticsearch-curator ).

My root folder is called index_curator , in which I have one single python file main.py . I installed the dependencies via pip as per Amazon's instructions eg

pip install elasticsearch elasticsearch-curator -t /path/to/index_curator

There are now many other files in this root directory and many child directories, which is not a surprise as these dependencies are quite large. For someone else looking at this package it would be difficult to differentiate between files I wrote and external dependencies. For example:

index_curator/
    dateutil/
    click/
    idna/
    main.py    <-- the only file I wrote
    README
    LICENSE
    six.py
    ...

Is there any way of shifting all these external dependencies into a sub-folder, eg

index_curator/
    /external/
        dateutil/
        click/
        idna/
        README
        LICENSE
        six.py
    main.py    <-- the only file I wrote

For completeness the imports in main.py are:

from elasticsearch import Elasticsearch, RequestsHttpConnection
import curator

Any pointers would be appreciated.

Answer 1

Separating external dependencies from your code is definitely best practice. There are a number of ways you can achieve this in python.

By default, python searches for modules in the locations specified here . To specify an additional location (ie your external dependencies folder), this new location must be added to the python path. You can do this in your main.py as follows:

import sys, os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), 'external')))

Then you will be able to import all your dependencies as usual, since the interpreter will check the additional folder during runtime:

from elasticsearch import Elasticsearch, RequestsHttpConnection
import curator

For more detail, check the answers here

Since your code will be a lambda function, you will always have one handler. But for a more generalised case, or if you start to write multiple files, and need to manage external dependencies in those as well - you can choose to maintain a context.py file, which sets the dependency path and all imports like so:

import sys, os
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), 'external')))
import elasticsearch
import context

Then in your files, you can call them with:

from context.elasticsearch import Elasticsearch, RequestsHttpConnection
import context.curator