Azure Functions - 无法在被调用脚本中导入其他 Python 模块

Question

I have created a simple HTTP trigger-based azure function in python which is calling another python script to create a sample file in azure data lake gen 1. My solution structure is given below: -

Requirements.txt contains the following imports: -

• azure-functions
• azure-mgmt-resource
• azure-mgmt-datalake-store
• azure-datalake-store

init.py

import logging, os, sys

import azure.functions as func
import json

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')
    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:         
            name = req_body.get('name')

    if name:
        full_path_to_script = os.path.join(os.path.dirname( __file__ ) + '/Test.py')
        logging.info(f"Path: - {full_path_to_script}")
        os.system(f"python {full_path_to_script}")
        return func.HttpResponse(f"Hello {name}!")
    else:
        return func.HttpResponse(
             "Please pass a name on the query string or in the request body",
             status_code=400
        )

Test.py

import json
from azure.datalake.store import core, lib, multithread

directoryId = ''
applicationKey = ''
applicationId = ''
adlsCredentials = lib.auth(tenant_id = directoryId, client_secret = applicationKey, client_id = applicationId)
adlsClient = core.AzureDLFileSystem(adlsCredentials, store_name = '')

with adlsClient.open('stage1/largeFiles/TestFile.json', 'rb') as input_file:
    data = json.load(input_file)
    with adlsClient.open('stage1/largeFiles/Result.json', 'wb') as responseFile:
        responseFile.write(data)

Test.py is failing with an error that no module found azure.datalake.store Why other required modules are not working for Test.py since it is inside the same directory?

pip freeze output: -

adal==1.2.2
azure-common==1.1.23
azure-datalake-store==0.0.48
azure-functions==1.0.4
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==6.0.0
azure-nspkg==3.0.2
certifi==2019.9.11
cffi==1.13.2
chardet==3.0.4
cryptography==2.8
idna==2.8
isodate==0.6.0
msrest==0.6.10
msrestazure==0.6.2
oauthlib==3.1.0
pycparser==2.19
PyJWT==1.7.1
python-dateutil==2.8.1
requests==2.22.0
requests-oauthlib==1.3.0
six==1.13.0
urllib3==1.25.6

Answer 1

Problem

os.system(f"python {full_path_to_script}") from your functions project is causing the issue.

Azure Functions Runtime sets up the environment, along with modifying process level variables like os.path so that your function can load any dependencies you may have. When you create a sub-process like that, not all information will flow through. Additionally, you will face issues with logging -- logs from test.py would not show up properly unless explicitly handled.

Importing works locally because you have all your requirements.txt modules installed and available to test.py . This is not the case in Azure. After remotely building as part of publish, your modules are included as part of your code package published. It's not "installed" globally in the Azure environment per se.

Solution

You shouldn't have to run your script like that. In the example above, you could import your test.py from your __init__.py file, and that should behave like it was called python test.py (at least in the case above). Is there a reason you'd want to do python test.py in a sub-process over importing it?

Here's the official guide on how you'd want to structure your app to import shared code -- https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python#folder-structure

Side-Note

I think once you get through the import issue, you may also face problems with adlsClient.open('stage1/largeFiles/TestFile.json', 'rb') . We recommend following the developer guide above to structure your project and using __file__ to get the absolute path ( reference ).

For example --

import pathlib
with open(pathlib.Path(__file__).parent / 'stage1' / 'largeFiles' /' TestFile.json'):
....

---

Now, if you really want to make os.system(f"python {full_path_to_script}") work, we have workarounds to the import issue. But, I'd rather not recommend such approach unless you have a really compelling need for it. :)