AWS Lambda connecting to SQL with Python and pyodbc

Question

I have an AWS Lambda that I want to connect to my on prem SQL server to read and write data from\\to. I am using Python and pyodbc. I have got pyodbc installed (compiled zip file in an S3 bucket added to the lambda through a layer), but when I try and run this code I get an odd error:

import boto3
import pyodbc

s3 = boto3.client('s3')

def lambda_handler(event, context):
    # print(help(pyodbc))
    server = "Server"
    database = "Database"
    username = "AWS-Lamdba-RO"
    password = "Password"
    cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
    cursor = cnxn.cursor()

This is the error:

[ERROR] AttributeError: module 'pyodbc' has no attribute 'connect' Traceback (most recent call last):   File "/var/task/lambda_function.py", line 13, in lambda_handler     cnxn = pyodbc.connect('DRIVER={ODBC Driver 13 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)

All I'm finding online is people who are unable to get the pyodbc library installed in the first place, so having got past that sticking point I thought I was free and clear. Can anyone explain what I've run into now?

I got pyodbc from here: https://github.com/Miserlou/lambda-packages/tree/master/lambda_packages/pyodbc

AWS didn't recognise .tar.gz files, so I changed it to a zip file and also added in the folder structure which another googled site told me was necessary: \\python\\lib\\python3.7\\site-packages\\pyodbc that folder contains: libodbc.so.2 pyodbc.so

I uploaded this Zip file to an S3 bucket and pointed a Lambda layer at it.

Have I done something silly with this?

Answer 1

From your description, I believe you may have gotten your folder structure wrong.

If you were to look in your zip file, you should see the following structure:

layerZip.zip
└ python
  └ lib
    └ python3.7
      └ site-packages
        └ pyodbc

But I believe you may actually have

layerZip.zip
└ \
  └ python
    └ lib
      └ python3.7
        └ site-packages
          └ pyodbc

But honestly, just use this structure:

layerZip.zip
└ python
  └ pyodbc

It'll work just as well, it's just setting the module up as global instead of per user.

Answer 2

I had the same issue. You must use correct version of python when building the package to upload to lambda.

I did a 'pip install pyodbc -t .'

It placed the following:

drwxr-xr-x 2 root root   4096 Sep 20 21:29 pyodbc-4.0.27.dist-info
-rwxr-xr-x 1 root root 658704 Sep 20 21:29 pyodbc.cpython-36m-x86_64-linux-gnu.so

The lib is very specific to the version of python. In the file name,'-36m-', so it will work the lambda python 3.6 environment.

My initial issue was I was using lambci/lambda:build-python3.7 for my docker environment. So pyodbc installed the python 3.7 version of the library, '-37m-'.

Since lambda was looking for the 3.6 version it did not see the 3.7 version.

I switched to using lambci/lambda:build-python3.6 and all was better.

I followed this article for getting everything working:

https://medium.com/faun/aws-lambda-microsoft-sql-server-how-to-66c5f9d275ed

Answer 3

Notable Points

• The environment in which you build the layer should be the same as your Lambda function runtime environment. ie. if you build the package in a python3.7 environment then lambda should be launched with the python 3.7 runtime.

Guidance from AWS Support

pyodbc is distributed in a shared lib format (which can be a little challenging to compile for lambda). It also requires other shared libs such as unixODBC (connector which the wrapper works around) and also the database drivers (in this case we will be using msodbcsql17).

The folder structure for this layer should look like this:

|-- pyodbc-layer
    |-- bin
    |-- include
    |-- lib
    |-- odbc.ini
    |-- ODBCDataSources
    |-- odbcinst.ini
    |-- python
        |-- pyodbc-4.0.26.dist-info
        |-- pyodbc.cpython-37m-x86_64-linux-gnu.so
    |-- share

In order to generate this layer, you need to execute the following steps:

1. create an EC2 Ubuntu 18.04 LTS instance (t2.micro is fine) and SSH into it.

2. Install docker using snap with the following command: sudo snap install docker

3. Run the following command to create a container based on amazon linux with python3.7 as it's environment. Keep in mind that you can change to python 3.6 just by changing build-python3.7 to build-python3.6.

sudo docker run -it --rm -iv${PWD}:/host-volume --entrypoint bash -e ODBCINI=/opt/odbc.ini -e ODBCSYSINI=/opt/ lambci/lambda:build-python3.7

4. When you fist run this command, docker will download the amazon linux image from dockerhub (it may take 30-60 seconds to download and unpack it). After you download it, you will enter the docker's image bash (In case you weren't redirected to the dockers bash, just run the command again)

5. After you are in the docker's bash, copy and paste the following commands to your docker:

curl ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.7.tar.gz  -O
tar xzvf unixODBC-2.3.7.tar.gz
cd unixODBC-2.3.7

./configure --sysconfdir=/opt --disable-gui --disable-drivers --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE --prefix=/opt
make
make install

cd ..
rm -rf unixODBC-2.3.7 unixODBC-2.3.7.tar.gz

curl https://packages.microsoft.com/config/rhel/6/prod.repo  > /etc/yum.repos.d/mssql-release.repo
yum install e2fsprogs.x86_64 0:1.43.5-2.43.amzn1 fuse-libs.x86_64 0:2.9.4-1.18.amzn1 libss.x86_64 0:1.43.5-2.43.amzn1
ACCEPT_EULA=Y yum install msodbcsql17 --disablerepo=amzn*
export CFLAGS="-I/opt/include"
export LDFLAGS="-L/opt/lib"

cd /opt
cp -r /opt/microsoft/msodbcsql17/ .
rm -rf /opt/microsoft/

mkdir /opt/python/
cd /opt/python/
pip install pyodbc -t .

cd /opt
cat <<EOF > odbcinst.ini
[ODBC Driver 17 for SQL Server]
Description=Microsoft ODBC Driver 17 for SQL Server
Driver=/opt/msodbcsql17/lib64/libmsodbcsql-17.7.so.2.1
UsageCount=1
EOF

cat <<EOF > odbc.ini
[ODBC Driver 17 for SQL Server]
Driver = ODBC Driver 17 for SQL Server
Description = My ODBC Driver 17 for SQL Server
Trace = No
EOF

cd /opt
zip -r9 ~/pyodbc-layer.zip .

(In case you get errors related to "read-only file system" just exit the docker shell by using the command "exit" and try the steps again)

6. Now make sure you exit your container using the command:

exit

7. Now the file will be available in the folder "/home/ubuntu". You can upload it using AWS CLI (if you have it configured) or retrieve via Sftp. In case you are going to use cyberduck/winscp/fillezilla or even sftp shell, you will need to change permissions on the file in order to download it.

The fastest way to give the file permission to be downloaded is using the following line:

sudo chmod o+rw pyodbc-layer.zip

8. Now you can retrieve it using sftp or any sftp-compatible client.