如何打包gcloud ml的子文件夹？

Question

I am trying to upload my project to google cloud ml-engine for training. I have followed the "getting started" guide, replacing in relevant places with my own files.

I manage to train locally using

gcloud ml-engine local train --module-name="my-model.task" --package-path=my-model/ -- ./my_model/model_params_google.json

Yes, I have dashes in the module name :(. I also made a symbolic link my_module -> my-module so that I can use the name with underscore instead of dash. In any case, I don't think this is the problem, since the above command works well locally.

My folder structure doesn't follow the recommended one, since I had the project before thinking about ml-engine. It looks like this:

my-model/
    ├── __init__.py
    ├── setup.py
    ├── task.py
    ├── model_params_google.json
    ├── src
    │   ├── __init__.py
    │   ├── data_handler.py
    │   ├── elastic_helpers.py
    │   ├── model.py

The problem is that the src folder is not packaged/uploaded with the code, so in the cloud, when I say from .src.model import model_fn in task.py , it fails.

The command I use for packaging is (in folder my-model/../ ):

gcloud ml-engine jobs submit training my_model_$(date +"%Y%m%d_%H%M%S") \
    --staging-bucket gs://model-data \
    --job-dir $OUTPUT_PATH \
    --module-name="my_model.task" \
    --package-path=my_model/ \
    --region=$REGION \
    --config config.yaml --runtime-version 1.8 \
    -- \
    tf_crnn/model_params_google.json --verbosity DEBUG

It packages my-model.0.0.0.tar.gz without the contents of my-model/src . I cannot figure out why. I'm using the example setup.py :

from setuptools import find_packages
from setuptools import setup

REQUIRED_PACKAGES = ['tensorflow>=1.8']

setup(
    name='my_model',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='my first model'
)

So, the question is, why does gcloud not pack the src folder ?

Answer 1

You need to put the setup.py in the directory above my-model .

You can check your results by invoking:

python setup.py sdist

Then un-taring the tarball in the dist directory. As is, you'll see that task.py is not included in the tarball.

By moving setup.py one directory higher and repeating, you'll see that task.py is included, as is everything in src.