poetry install different package version based on extras

Question

Using python-poetry, I would like to install different package versions based on the extras that I pass during the installation. Eg I would like to

# when extra == 'a', install numpy == 1.20.0
$ poetry install -E a
# when extra == 'b', install numpy == 1.19.0
$ poetry install -E b

I tried it with the following toml file

[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["tenticon"]

[tool.poetry.dependencies]
python = "^3.8"
numpy = [
    { version = "1.20.0", markers = "extra == 'a'", optional = true},
    { version = "1.19.0", markers = "extra == 'b'", optional = true}
]

[tool.poetry.extras]
a = [ "numpy" ]
b = [ "numpy" ]

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

but when I do $ poetry install -E a I get

  SolverProblemError

  Because demo-poetry depends on both numpy (1.20.0) and numpy (1.19.0), version solving failed.

My poetry version is 1.1.6

Answer 1

I think there is some confusion between the concept of extras in poetry and pip. Poetry extras are sets of packages (eg math = ["numpy", "scipy"] ) that can be (optionally) installed together ( poetry install -E math ). They are similar to the concept of groups which are introduced in the pre-release 1.2.0a2 . However, pip extras are related only to one package, ie they are dependencies that you can install in addition to the regular dependencies of a package, if you ask for them explicitly (eg pip install ludwig[text] ).

It is indeed possible to use environment markers as install conditions for dependencies. However, the extras defined in the PEP-0508 definition of environment markers relate to pip extras, and not poetry extras. Thus, poetry extras can not be used as environment markers for constraining install conditions.

Neither using extras nor groups (from the pre-release) was sufficient in order to achieve the expected results. In this regard, I think the comment from @finswimmer is correct: extras are not mutually exclusive, and poetry checks that the dependency resolution works in every case. The same happens with groups.

The closest I could get to an acceptable solution was by defining conditions based on the python version, or the platform. For instance, if you have the following pyproject.toml :

[tool.poetry]
name = "demo-poetry"
version = "0.1.0"
description = ""
authors = ["vreyespue"]

[tool.poetry.dependencies]
numpy = [
    { version = "1.19.0", python = "~3.7"},
    { version = "1.20.0", python = "~3.9"}
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

You could install different versions of numpy in different environments:

$ poetry env use 3.7
  Using virtualenv: /***/demo-poetry-***-py3.7

$ poetry install
  Installing numpy (1.19.0)

$ poetry env use 3.9
  Using virtualenv: /***/demo-poetry-***-py3.9

$ poetry install
  Installing numpy (1.20.0)

Answer 2

This bothered me for internal development, and while it was extremely frustrating since neither poetry nor cleo offer any documentation on their SDK, I finally managed to create a plugin for this ( pypi , git ).

The idea is that with this plugin, you can have non-mutual exclusive definitions in groups, and when using the --without or --only flags, the relevant groups are dropped from dependency parsing. The plugin needs to be installed prior to any such group definition.

Installation via poetry add poet-plugin (if starting from an empty, clean project, without such dependencies), or then via:

poetry shell
pip install poet-plugin

I have not battle-tested this, but it works nicely for our use case with eg

poetry install --only prod
poetry install --only dev
poetry install --without dev
poetry install --without prod

Feedback and updates etc are welcome, hope this momentarily resolves this issue for some of us out there.