tox-poetry-installer/README.md

tox-poetry-installer

A plugin for Tox that lets you install test environment dependencies from the Poetry lockfile.

⚠️ This project is beta software and is under active development ⚠️

Documentation

• Feature Overview
• Using the Plugin
  • Installing
  • Quick Start
  • References
    • Config Options
    • Runtime Options
    • Errors
  • Other Notes
    • Unsupported Tox config options
    • Updating locked dependencies in a testenv
    • Installing unsafe dependencies
    • Using with an unmanaged Poetry installation
• Developing the Plugin
• Road Map

See the Changelog for release history.

See also: official Tox plugins, Poetry-Dev-Dependencies Tox plugin, Poetry Tox plugin

Feature Overview

• Manage package versions in exactly one place and with exactly one tool: Poetry.
• Ensure CI/CD and other automation tools are using the same package versions that you are in your local development environment.
• Add only the packages you need to a Tox test environment, instead of everything in your lockfile.
• Directly integrate with Poetry, re-using your existing package indexes and credentials with no additional configuration.
• Wherever possible, built-in Tox config options are always respected and their behavior kept consistent.
• Extremely configurable. Every feature can be disabled or enabled for any given Tox test environment.
• Friendly to other Tox plugins and supports a wide range of environments.

User Documentation

This section is for users looking to integrate the plugin with their project or CI system. For information on contributing to the plugin please see the Developer Docs

Installing

The recommended way to install the plugin is to add it to a project using Poetry:

poetry add tox-poetry-installer[poetry] --dev

Note: Always install the plugin with the [poetry] extra, unless you are managing the Poetry installation yourself.

Alternatively, it can be installed directly to a virtual environment using Pip, though this is not recommended:

source somevenv/bin/activate
pip install tox-poetry-installer

Alternatively alternatively, it can be installed using the Tox requires option by adding the below to tox.ini, though this is also not recommended:

requires =
    tox-poetry-installer[poetry] == 0.7.0

After installing, check that Tox recognizes the plugin by running poetry run tox --version. The command should give output similar to below:

3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py
registered plugins:
    tox-poetry-installer-0.7.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py

Quick Start

Congratulations! 🎉 Just by installing the plugin your Tox config is already using locked dependencies: when Tox builds and installs your project package to a test environment, your project package's dependencies will be installed from the lockfile.

Now lets update an example tox.ini to install the other test environment dependencies from the lockfile.

A testenv from the example tox.ini we're starting with is below:

[testenv]
description = Some very cool tests
deps =
    black == 20.8b1
    pylint >=2.4.4,<2.7.0
    mypy <0.800
commands = ...

To update the config so that the testenv dependencies are installed from the lockfile, we can replace the built-in deps option with the locked_deps option provided by the plugin, and then remove the inline version specifiers. With these changes the three testenv dependencies (as well as all of their dependencies) will be installed from the lockfile when the test environment is recreated:

[testenv]
description = Some very cool tests
locked_deps =
    black
    pylint
    mypy
commands = ...

We can also add the require_locked_deps option to the test environment. This will both block any other install tools (another plugin or Tox itself) from installing dependencies to the Tox environment and also cause Tox to fail if the test environment also uses the built-in deps option:

[testenv]
description = Some very cool tests
require_locked_deps = true
locked_deps =
    black
    pylint
    mypy
commands = ...

Note: Settings configured on the main testenv environment are inherited by child test environments (for example, testenv:foo). To override this, specify the setting in the child environment with a different value.

Alternatively, we can skip specifying all of our dependencies for a test environment in the Tox config and just install all of our Poetry dev-dependencies automatically:

[testenv]
description = Some very cool tests
require_locked_deps = true
install_dev_deps = true
commands = ...

Note: Setting install_dev_deps = true on an environment that also installs the project package is functionally equivalent to running poetry install.

Finally, we can also install an unlocked dependency (a dependency which doesn't take its version from the Poetry lockfile) into the test environment alongside the locked ones. We need to remove the require_locked_deps = true option, otherwise the environment will error, and then we can add the unlocked dependency using the built-in deps option:

[testenv]
description = Some very cool tests
deps =
    pytest >= 5.6.0,<6.0.0
locked_deps =
    black
    pylint
    mypy
commands = ...

References

Configuration Options

All options listed below are Tox environment options and can be applied to one or more environment sections of the tox.ini file. They cannot be applied to the global Tox configuration section.

Note: Settings configured on the main testenv environment are inherited by child test environments (for example, testenv:foo). To override this, specify the setting in the child environment with a different value.

Option	Type	Default	Description
locked_deps	List	[]	Names of packages to install to the test environment from the Poetry lockfile. Transient dependencies (packages required by these dependencies) are automatically included.
require_locked_deps	Boolean	False	Whether the plugin should block attempts to install unlocked dependencies to the test environment. If enabled, then the tox_testenv_install_deps plugin hook will be intercepted and an error will be raised if the test environment has the deps option configured.
install_dev_deps	Boolean	False	Whether all of the Poetry dev-dependencies should be installed to the test environment.
require_poetry	Boolean	False	Whether Tox should be forced to fail if the plugin cannot import Poetry locally. If False then the plugin will be skipped for the test environment if Poetry cannot be imported. If True then the plugin will force the environment to error and the Tox run to fail.

Runtime Options

All arguments listed below can be passed to the tox command to modify runtime behavior of the plugin.

Argument	Type	Default	Description
--parallelize-locked-install	Integer	0	Number of worker threads to use to install dependencies in parallel. Installing in parallel with more threads can greatly speed up the install process, but can cause race conditions during install. The default, 0, disables the parallel install so that dependencies are installed sequentially.

Note: The --require-poetry runtime option is deprecated and will be removed in version 1.0.0. Please set require_poetry = true in tox.ini for environments that should fail if Poetry is not available.

Errors

There are several errors that the plugin can encounter for a test environment when Tox is run. If an error is encountered then the status of the test environment that caused the error will be set to one of the "Status" values below to indicate what the error was.

Status/Name	Cause
ExtraNotFoundError	Indicates that the extras config option specified an extra that is not configured by Poetry in pyproject.toml.
LockedDepVersionConflictError	Indicates that an item in the locked_deps config option includes a PEP-508 version specifier (ex: pytest >=6.0, <6.1).
LockedDepNotFoundError	Indicates that an item specified in the locked_deps config option does not match the name of a package in the Poetry lockfile.
LockedDepsRequiredError	Indicates that a test environment with the require_locked_deps config option set to true also specified unlocked dependencies using the deps config option.
PoetryNotInstalledError	Indicates that the poetry module could not be imported under the current runtime environment, and the --require-poetry flag was provided.

Note: One or more of these errors can be caused by the pyproject.toml being out of sync with the Poetry lockfile. If this is the case, than a warning will be logged when Tox is run.

Other Notes

Unsupported Tox config options

Below are the built-in Tox config options that are not respected by this plugin. All of these options are made obsolete by the Poetry lockfile: either they aren't needed or their equivalent functionality is instead taken directly from the package details Poetry stores in its lockfile.

Note: The unsupported Tox config options will still apply to unlocked dependencies being installed with the default Tox installation backend.

• install_command
• pip_pre
• download
• indexserver
• usedevelop

Updating locked dependencies in a testenv

When Poetry updates the version of a package in the lockfile (using either poetry lock or poetry update) then the plugin will automatically use this new version to install the package to a test environment; there is no need to manually update tox.ini after updating the Poetry lockfile.

However, the plugin cannot determine when the lockfile is updated. If a Tox test environment has already been created then it will need to be recreated (using Tox's built-in --recreate option) for the new version to be found and installed.

Note: To force Tox to always recreate a test environment the recreate config option can be set.

Installing unsafe dependencies

There are several packages that cannot be installed from the lockfile because they are excluded by Poetry itself. As a result these packages cannot be installed by this plugin either as test environment dependencies passed directly to locked_deps or as a transient dependency. When one of these packages is encountered by the plugin a warning will be logged to the console and the unsafe package will not be installed to the test environment.

This list can be found in the Poetry source code here. As of Poetry 1.1.6 there are four packages classified as "unsafe" by Poetry and excluded from the lockfile:

• setuptools
• distribute
• pip
• wheel

Using with an unmanaged Poetry installation

In CI/CD systems, automation environments, or other Python environments where the loaded site packages are not managed by Poetry, it can be useful to manage the local installation of Poetry externally. This also helps to avoid problems that can be caused by the --no-root, --no-dev, or --remove-untracked arguments to the poetry install command which, in some situations, can cause Poetry to uninstall itself if Poetry is specified as a dependency of one of the packages it is managing (like this plugin). To support these use cases, this plugin specifies the poetry package as an optional dependency that can be installed using a setuptools extra also named poetry.

Critical Warning: This plugin requires Poetry to function. If the plugin is installed without the poetry setuptools extra then Poetry must be installed independently for the plugin to function properly.

To skip installing the poetry package as a dependency of tox-poetry-installer, do not specify the poetry extra when adding the plugin:

# Adding the package without the "[poetry]" extra specifier so that
# Poetry is not added as a transient dev-dependency:
poetry add tox-poetry-installer --dev

# Adding the package with the "[poetry]" extra specifier, so the Poetry
# package will be added to the environment and tracked in the lockfile:
poetry add tox-poetry-installer[poetry] --dev

Once the plugin is installed- either with or without the Poetry extra- you can validate that the plugin will run correctly with the following command. This command checks that all three required components (Tox, Poetry, and the plugin itself) are available in the current Python environment:

python -c '\
  import tox;\
  import tox_poetry_installer;\
  from poetry.poetry import Poetry;\
'

Note: To force Tox to fail if Poetry is not installed, run the tox command with the --require-poetry option. See the Runtime Options for more information.

Developer Documentation

All project contributors and participants are expected to adhere to the Contributor Covenant Code of Conduct, v2 (external link).

The devel branch has the latest (and potentially unstable) changes. The stable releases are tracked on Github, PyPi, and in the Changelog.

• To report a bug, request a feature, or ask for assistance, please open an issue on the Github repository.
• To report a security concern or code of conduct violation, please contact the project author directly at ‌me [at‌] enp dot‎ ‌one.
• To submit an update, please fork the repository and open a pull request.

Developing this project requires at least Python 3.6 and at least Poetry 1.0. GNU Make can optionally be used to quickly setup a local development environment, but this is not required.

To setup a local development environment:

# Clone the repository...
# ...over HTTPS
git clone https://github.com/enpaul/tox-poetry-installer.git
# ...over SSH
git clone git@github.com:enpaul/tox-poetry-installer.git

cd tox-poetry-installer/

# Create and configure the local development environment...
# ...with make:
make dev
# ...manually:
poetry install -E poetry --remove-untracked
poetry run pre-commit install

# Run tests and CI locally...
# ...with make:
make test
# ...manually:
poetry run tox --recreate

# See additional make targets
make help

Note: The pre-commit hooks require dependencies in the Poetry environment to run. To make a commit with the pre-commit hooks, you will need to run poetry run git commit or, alternatively, launch an environment shell.

Road Map

This project is under active development and is classified as beta software, ready for production environments on a provisional basis only.

• Beta classification was assigned with v0.6.0
• Stable classification will be assigned when the test suite covers an acceptable number of use cases

Path to Beta

• Verify that primary package dependencies (from the .package env) are installed correctly using the Poetry backend.
• Support the extras Tox configuration option (#4)
• Add per-environment Tox configuration option to fall back to default installation backend.
• ~Add warnings when an unsupported Tox configuration option is detected while using the Poetry backend.~ (#5)
• Add trivial tests to ensure the project metadata is consistent between the pyproject.toml and the module constants.
• Update to use poetry-core and improve robustness of the Tox and Poetry module imports to avoid potentially breaking API changes in upstream packages. (#2)
• ~Find and implement a way to mitigate the UNSAFE_DEPENDENCIES issue in Poetry.~ (#6)
• Fix logging to make proper use of Tox's logging reporter infrastructure (#3)
• Add configuration option for installing all dev-dependencies to a testenv (#14)

Path to Stable

Everything in Beta plus...

• Fully replace dependency on poetry with dependency on poetry-core (#2)
• Add comprehensive unit tests
• Add tests for each feature version of Tox between 3.8 and 3.20
• Add tests for Python-3.6, 3.7, 3.8, and 3.9
• Add Github Actions based CI