Snapcraft python 脚本与模块，暂存问题

Question

I am building a snap to test integration of a python script and a python SDK with snapcraft and there appears to be a conflict when two python 'parts' are built in the same snap.

What is the best way to build a snap with multiple python modules?

I have a simple script which imports the SDK and then prints some information. I also have the python SDK library ( https://help.iotconnect.io/documentation/sdk-reference/device-sdks-flavors/download-python-sdk/ ) in a different folder.

I have defined the two parts, and each one can be built stand alone (snapcraft build PARTNAME), however it seems the python internals are conflicting at the next step of 'staging' them together.

tree output of structure

tree
.
├── basictest
│   ├── basictest.py
│   ├── __init__.py
│   └── setup.py
├── iotconnect-sdk-3.0.1
│   ├── iotconnect
│   │   ├── assets
│   │   │   ├── config.json
│   │   │   └── crt.txt
│   │   ├── client
│   │   │   ├── httpclient.py
│   │   │   ├── __init__.py
│   │   │   ├── mqttclient.py
│   │   │   └── offlineclient.py
│   │   ├── common
│   │   │   ├── data_evaluation.py
│   │   │   ├── infinite_timer.py
│   │   │   ├── __init__.py
│   │   │   └── rule_evaluation.py
│   │   ├── __init__.py
│   │   ├── IoTConnectSDKException.py
│   │   ├── IoTConnectSDK.py
│   │   └── __pycache__
│   │       ├── __init__.cpython-38.pyc
│   │       └── IoTConnectSDK.cpython-38.pyc
│   ├── iotconnect_sdk.egg-info
│   │   ├── dependency_links.txt
│   │   ├── not-zip-safe
│   │   ├── PKG-INFO
│   │   ├── requires.txt
│   │   ├── SOURCES.txt
│   │   └── top_level.txt
│   ├── PKG-INFO
│   ├── README.md
│   ├── setup.cfg
│   └── setup.py
└── snap
    └── snapcraft.yaml

9 directories, 30 files

snapcraft.yaml

name: basictest
base: core20
version: '0.1'
summary: Test snap to verifiy integration with python SDK
description: |
  Test snap to verifiy integration with python SDK

grade: devel
confinement: devmode

apps: 
  basictest:
    command: bin/basictest

parts:
  lib-basictest:
    plugin: python
    source: ./basictest/
    after: [lib-pythonsdk]
    disable-parallel: true
    
  lib-pythonsdk:
    plugin: python
    source: ./iotconnect-sdk-3.0.1/

Running 'snapcraft' shows tons of errors which look like conflicts between the two 'parts' related to the internals of python.

snapcraft output

snapcraft
Launching a VM.
Skipping pull lib-pythonsdk (already ran)
Skipping pull lib-basictest (already ran)
Skipping build lib-pythonsdk (already ran)
Skipping build lib-basictest (already ran)
Failed to stage: Parts 'lib-pythonsdk' and 'lib-basictest' have the following files, but with different contents:
    bin/activate
    bin/activate.csh
    bin/activate.fish
    lib/python3.8/site-packages/_distutils_hack/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/_distutils_hack/__pycache__/override.cpython-38.pyc
    lib/python3.8/site-packages/pip-21.3.1.dist-info/RECORD
    lib/python3.8/site-packages/pip/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/pip/__pycache__/__main__.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/build_env.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/cache.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/configuration.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/exceptions.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/main.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/pyproject.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/self_outdated_check.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/__pycache__/wheel_builder.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/autocompletion.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/base_command.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/cmdoptions.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/command_context.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/main.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/main_parser.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/parser.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/progress_bars.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/req_command.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/spinners.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/cli/__pycache__/status_codes.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/cache.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/check.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/completion.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/configuration.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/debug.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/download.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/freeze.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/hash.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/help.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/index.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/install.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/list.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/search.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/show.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/uninstall.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/commands/__pycache__/wheel.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/distributions/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/distributions/__pycache__/base.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/distributions/__pycache__/installed.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/distributions/__pycache__/sdist.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/distributions/__pycache__/wheel.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/index/__pycache__/__init__.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/index/__pycache__/collector.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/index/__pycache__/package_finder.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/index/__pycache__/sources.cpython-38.pyc
    lib/python3.8/site-packages/pip/_internal/locations/__pycache__/__init__.cpython-38.pyc
... Tons more removed

Snapcraft offers some capabilities to solve this by use of the following keywords:
    - `filesets`
    - `stage`
    - `snap`
    - `organize`

To learn more about these part keywords, run `snapcraft help plugins`.
Run the same command again with --debug to shell into the environment if you wish to introspect this failure.

Main question What is the best way to build a snap with multiple python modules?

Answer 1

It looks like the best solution is to remove the offending build files from being included by the library. The 'lib-basictest' part is the main executing script, the files generated there should be included over the SDK library versions

Here is the updated lib-pythonsdk part

  lib-pythonsdk:
    plugin: python
    source: ./iotconnect-sdk-3.0.1/
    stage:
      - -lib/python3.8/site-packages/pip/
      - -lib/python3.8/site-packages/setuptools/
      - -lib/python3.8/site-packages/wheel/
      - -lib/python3.8/site-packages/_distutils_hack
      - -lib/python3.8/site-packages/pip-21.3.1.dist-info
      - -lib/python3.8/site-packages/pkg_resources
      - -lib/python3.8/site-packages/wheel-0.37.1.dist-info
      - -bin/activate
      - -bin/activate.csh
      - -bin/activate.fish
      - -pyvenv.cfg

This is what I will go with, unless there is some 'overwrite' parameter I am missing?

Answer 2

To make the solution more portable and handle different version numbers, the sub folders can use a wildcard

lib-pythonsdk:
    plugin: python
    source: ./iotconnect-sdk-3.0.1/
    stage:
      - -lib/python3.8/site-packages/pip*
      - -lib/python3.8/site-packages/wheel*
      - -lib/python3.8/site-packages/setuptools/
      - -lib/python3.8/site-packages/_distutils_hack
      - -lib/python3.8/site-packages/pkg_resources
      - -bin/activate
      - -bin/activate.csh
      - -bin/activate.fish
      - -pyvenv.cfg