Run a python script on Azure Batch

Question

Iam trying to execute a python script on azure batch which is a linux dsvm so that the script can install python packages and then execute the python script.

Below is the code i used:

try:
   from pip import main as pipmain
except ImportError:
   from pip._internal import main as pipmain

try:
    import pandas as pd
except:

   pipmain(['install', 'pandas',"])

import pandas

When i run the python script on azure Batch command line, the pool task errors out at the last statement(import pandas) eventhough i can see in the stdout log file that the pandas, numpy etc packages are installed.

It seems that the packages are installed at some other location and while trying to import it is trying to import from some other location. It gives the error ImportError: No module named pandas in the stderr.txt file on the azure batch pool tasks.

The reason why iam trying to install python packages and importing it the same script is because the azure batch command line doesnt allow me to write 2 commands, something like

pip install pandas
python test.py

where it first install the packages and then invokes script where it just does the import of pandas library.

I have also used the command in the pip install pandas and pip install --install-option="--prefix=$AZ_BATCH_TASK_WORKING_DIR" pandas at the start task of the batch pool. AZ_BATCH_TASK_WORKING_DIR as per my understanding is working directory to which the task and script has access when the task batch runs

Is there a way i run the python script successfully on Azure Batch. At the momemt iam only running one command: import pandas

Answer 1

You need to provide a inline shell script to run your multiple commands and take advantage of shell expansion. Please see this doc . You'll want to run your two commands like:

/bin/bash -c "pip install pandas && python test.py"

Additionally, tasks are run under context-specific directories (ie, a start task runs in the start task directory versus a normal task will run in a different directory, although $AZ_BATCH_TASK_WORKING_DIR is named the same) and user identities can also modify the user context for which a task is run.

Answer 2

Although fpark's answer put me on the right track, I was not successful. However, I run into this post which worked for me.

The start task command would be (for python3):

/bin/bash -c "sudo apt-get -y update && sudo dpkg --configure -a && sudo apt-get install -y python3-pip && pip3 install --upgrade pip && sudo pip3 install pandas"