From 03f46d34f353b2585f612e5f5cc667a892616f5a Mon Sep 17 00:00:00 2001 From: Ethan Paul <24588726+enpaul@users.noreply.github.com> Date: Sat, 5 Dec 2020 15:41:34 -0500 Subject: [PATCH 1/6] Rewrite usage reference documentation to improve clarity Add more details to error documentation Update error and option documentation to be referenceable via slugs Add docs for missing --require-poetry option Add more crosslinks to assit with navigation --- README.md | 299 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 207 insertions(+), 92 deletions(-) diff --git a/README.md b/README.md index f4c7da6..f0aa0dc 100644 --- a/README.md +++ b/README.md @@ -15,11 +15,12 @@ dependencies to be installed using [Poetry](https://python-poetry.org/) from its * [Installation](#installation) * [Quick Start](#quick-start) -* [Reference and Usage](#reference-and-usage) - * [Config Option Reference](#config-option-reference) - * [Error Reference](#error-reference) - * [Example Config](#example-config) -* [Known Drawbacks and Problems](#known-drawbacks-and-problems) +* [Plugin Usage](#reference-and-usage) +* [Reference](#reference) + * [Configuration Options](#configuration-options) + * [Command-line Arguments](#command-line-arguments) + * [Errors](#errors) + * [Advanced Usage](#advanced-usage) * [Why would I use this?](#why-would-i-use-this) (What problems does this solve?) * [Developing](#developing) * [Contributing](#contributing) @@ -108,9 +109,9 @@ Alternatively, to quickly install all Poetry dev-dependencies to a Tox environme one Tox is testing) will always be installed from the lockfile. -## Reference and Usage +## Reference -### Config Option Reference +### Configuration Options All options listed below are Tox environment options and can be applied to one or more environment sections of the `tox.ini` file. They cannot be applied to the global Tox @@ -120,104 +121,218 @@ configuration section. inherited by child environments (i.e. `testenv:foo`) unless they are explicitly overridden by the child environment's configuration. -| Option | Type | Default | Usage | -|:----------------------|:----------------|:--------|:-----------------------------------------------| -| `locked_deps` | Multi-line list | `[]` | Names of packages in the Poetry lockfile to install to the Tox environment. All dependencies specified here (and their dependencies) will be installed to the Tox environment using the version the Poetry lockfile specifies for them. | -| `require_locked_deps` | Bool | `false` | Indicates whether the environment should allow unlocked dependencies (dependencies not in the Poetry lockfile) to be installed alongside locked dependencies. If `true` then installation of unlocked dependencies will be blocked and an error will be raised if the `deps` option specifies any values. | -| `install_dev_deps` | Bool | `false` | Indicates whether all Poetry development dependencies should be installed to the environment. Provides a quick and easy way to install all dev-dependencies without needing to specify them individually. | +#### `locked_deps` -### Error Reference +* **Type:** multi-line list +* **Default:** `[]` -* `LockedDepVersionConflictError` - Indicates that a locked dependency included a PEP-508 version - specifier (i.e. `pytest >=6.0, <6.1`). Locked dependencies always take their version from the - Poetry lockfile so specifying a specific version for a locked dependency is not supported. -* `LockedDepNotFoundError` - Indicates that a locked dependency could not be found in the Poetry - lockfile. This can be solved by [adding the dependency using Poetry](https://python-poetry.org/docs/cli/#add). -* `ExtraNotFoundError` - Indicates that the Tox `extras` option specified a project extra that - Poetry does not know about. This may be due to a misconfigured `pyproject.toml` or out of date - lockfile. -* `LockedDepsRequiredError` - Indicates that an environment with `require_locked_deps = true` also - specified unlocked dependencies using Tox's `deps` option. This can be solved by either setting - `require_locked_deps = false` (the default) or removing the `deps` option from the environment - configuration. +Names of packages in the Poetry lockfile to install to the Tox environment. All +dependencies specified here will be installed to the Tox environment using the details +given by the Poetry lockfile. -### Example Config - -```ini -[tox] -envlist = py, foo, bar, baz -isolated_build = true - -# The base testenv will always use locked dependencies and only ever installs the project package -# (and its dependencies) and the two pytest dependencies listed below -[testenv] -description = Some very cool tests -require_locked_deps = true -locked_deps = - pytest - pytest-cov -commands = ... - -# This environment also requires locked dependencies, but the "skip_install" setting means that -# the project dependencies will not be installed to the environment from the lockfile -[testenv:foo] -description = FOObarbaz -skip_install = true -require_locked_deps = true -locked_deps = - requests - toml - ruamel.yaml -commands = ... - -# This environment allows unlocked dependencies to be installed ad-hoc. Below, the "mypy" and -# "pylint" dependencies (and their dependencies) will be installed from the Poetry lockfile but the -# "black" dependency will be installed using the default Tox backend. Note, this environment does -# not specify "require_locked_deps = true" to allow the unlocked "black" dependency without raising -# an error. -[testenv:bar] -description = fooBARbaz -locked_deps = - mypy - pylint -deps = - black -commands = ... - -# This environment requires locked dependencies but does not specify any. Instead it specifies the -# "install_dev_deps = true" option which will cause all of the Poetry dev-dependencies to be -# installed from the lockfile. -[testenv:baz] -description = foobarBAZ -install_dev_deps = true -require_locked_deps = true -commands = ... -``` +#### `require_locked_deps` -## Known Drawbacks and Problems +* **Type:** boolean +* **Default:** `false` + +Whether the environment should allow unlocked dependencies (dependencies not in the +Poetry lockfile) to be installed alongside locked dependencies. If `true` then an error +will be raised if the environment specifies unlocked dependencies to install and the +plugin will block any other plugins from using the +[`tox_testenv_install_deps`](https://tox.readthedocs.io/en/latest/plugins.html#tox.hookspecs.tox_testenv_install_deps) +hook. + +#### `install_dev_deps` + +* **Type:** boolean +* **Default:** `false` + +Whether all Poetry dev-dependencies should be installed to the environment. If `true` +then all dependencies specified in the +[`dev-dependencies` section](https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies) +of `pyproject.toml` will be installed automatically. + +### Command-line Arguments + +All arguments listed below can be passed to the `tox` command to modify runtime behavior +of the plugin. + +#### `--require-poetry` + +Indicates that Poetry is expected to be available to Tox and, if it is not, then the Tox +run should fail. If provided and the `poetry` package is not installed to the same +environment as the `tox` package then Tox will fail. + +**NOTE:** See [Advanced Usage](installing-alongside-an-existing-poetry-installation) +for more information. + +### Errors + +If the plugin encounters an error while processing a Tox environment then it will mark +the environment as failed and set the environment status to one of the values below: + +**NOTE:** In addition to the reasons noted below, the plugin can encounter errors if the +Poetry lockfile is not up-to-date with `pyproject.toml`. To resynchronize the +lockfile with the `pyproject.toml` run one of +[`poetry update`](https://python-poetry.org/docs/cli/#update) or +[`poetry lock`](https://python-poetry.org/docs/cli/#lock) + +#### Poetry Not Installed Error + +* **Status value:** `PoetryNotInstalledError` +* **Cause:** Indicates that the `poetry` module could not be imported from the same + environment as the running `tox` module and the runtime flags specified + [`--require-poetry`](#--require-poetry). +* **Resolution options:** + * Install Poetry: ensure that `poetry` is installed to the same environment as `tox`. + * Skip running the plugin: remove the `--require-poetry` flag from the runtime options. + +**NOTE:** See [Advanced Usage](installing-alongside-an-existing-poetry-installation) +for more information. + +#### Locked Dependency Version Conflict Error + +* **Status value:** `LockedDepVersionConflictError` +* **Cause:** Indicates that a dependency specified in the [`locked_deps`](#locked_deps) + configuration option in `tox.ini` includes a + [PEP-508 version specifier](https://www.python.org/dev/peps/pep-0508/#grammar) + (i.e. `pytest >=6.0, <6.1`). +* **Resolution options:** + * Use the dependency version from the lockfile: remove any/all version specifiers + from the item in the `locked_deps` list in `tox.ini`. + * Do not install the dependency: remove the item from the `locked_deps` list in + `tox.ini`. + +#### Locked Dependency Not Found Error + +* **Status value:** `LockedDepNotFoundError` +* **Cause:** Indicates that a dependency specified in the [`locked_deps`](#locked_deps) + configuration option in `tox.ini` could not be found in the Poetry lockfile. +* **Resolution options:** + * Add the dependency to the lockfile: run `poetry add `; + see [the Poetry documentation](https://python-poetry.org/docs/cli/#add) for more + information. + * Do not install the dependency: remove the item from the `locked_deps` list in + `tox.ini`. + +#### Extra Not Found Error + +* **Status value:** `ExtraNotFoundError` +* **Cause:** Indicates that the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras) + configuration option specified a setuptools extra that is not configured by Poetry in + `pyproject.toml` +* **Resolution options:** + * Configure the extra: add a section for the named extra to the + [`extras` section of `pyproject.toml`](https://python-poetry.org/docs/pyproject/#extras) + and optionally assign dependencies to the named extra using the + [`--optional` dependency setting](https://python-poetry.org/docs/cli/#options_3). + * Remove the extra: remove the item from the `extras` list in `tox.ini`. + +#### Locked Dependencies Required Error + +* **Status value:** `LockedDepsRequiredError` +* **Cause:** Indicates that an environment with the [`require_locked_deps`](#require_locked_deps) + configuration option also specified unlocked dependencies using + [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) option in + `tox.ini`. +* **Resolution options:** + * Remove all unlocked dependencies: remove the `deps` configuration option in + `tox.ini`. + * Allow unlocked dependencies: remove the `require_locked_deps` configuration option + in `tox.ini` or explicitly set `require_locked_deps = false`. + +### Advanced Usage + +#### Unsupported Tox configuration options + +The `tox.ini` configuration options listed below have no effect on the dependencies +installed by this plugin the Poetry lockfile. Note that these settings will still be +applied by the default Tox installation backend when installing unlocked dependencies +using the built-in `deps` option. -* The following `tox.ini` configuration options have no effect on the dependencies installed from - the Poetry lockfile (note that they will still affect unlocked dependencies): * [`install_command`](https://tox.readthedocs.io/en/latest/config.html#conf-install_command) * [`pip_pre`](https://tox.readthedocs.io/en/latest/config.html#conf-pip_pre) - * [`downloadcache`](https://tox.readthedocs.io/en/latest/config.html#conf-downloadcache) (deprecated) * [`download`](https://tox.readthedocs.io/en/latest/config.html#conf-download) * [`indexserver`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver) * [`usedevelop`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver) -* Tox will not automatically detect changes to the locked dependencies and so - environments will not be automatically rebuilt when locked dependencies are changed. - When changing the locked dependencies (or their versions) the environments will need to - be manually rebuilt using either the `-r`/`--recreate` CLI option or the - `recreate = true` option in `tox.ini`. +All of these options are obsoleted by using the Poetry backend. If a given package +installs successfully using Poetry (using either `poetry add ` or +`poetry install`) then the required configuration options are already properly set in +the Poetry configuration and the plugin will automatically use the same settings when +installing the package. -* There are a handful of packages that cannot be installed from the lockfile, whether as specific - dependencies or as transient dependencies (dependencies of dependencies). This is due to - [an ongoing discussion in the Poetry project](https://github.com/python-poetry/poetry/issues/1584); - the list of dependencies that cannot be installed from the lockfile can be found - [here](https://github.com/python-poetry/poetry/blob/cc8f59a31567f806be868aba880ae0642d49b74e/poetry/puzzle/provider.py#L55). - This plugin will skip these dependencies entirely, but log a warning when they are encountered. +#### Reinstalling locked dependencies to a Tox environment + +Updating the `poetry.lock` file will not automatically cause Tox to install the updated +lockfile specifications to the Tox environments that specify them. + +The Tox environment(s) with updated locked dependencies must be deleted and recreated +using the [`--recreate`](https://tox.readthedocs.io/en/latest/config.html#cmdoption-tox-r) +runtime flag. Alternatively Tox can be configured to always recreate an environment by +setting the [`recreate`](https://tox.readthedocs.io/en/latest/config.html#conf-recreate) +option in `tox.ini`. + +#### Installing Poetry's unsafe dependencies + +There are several packages that cannot be installed from the lockfile because they are +excluded by Poetry itself. As a result these packages cannot be installed by this plugin +either as environment dependencies (passed directly to [`locked_deps`](#locked_deps)) or +as transient dependencies (a dependency of a locked dependency). + +As of [Poetry-1.1.4](https://github.com/python-poetry/poetry/releases/tag/1.1.4) there +are four packages classified as "unsafe" by Poetry and excluded from the lockfile: + +* `setuptools` +* `distribute` +* `pip` +* `wheel` + +When one of these packages is encountered by the plugin a warning will be logged and +_**the package will not be installed to the environment**_. If the unsafe package +is required for the environment then it will need to be specified as an unlocked +dependency using the [`deps`](https://github.com/python-poetry/poetry/releases/tag/1.1.4) +configuration option in `tox.ini`, ideally with an exact pinned version. + +* The set of packages excluded from the Poetry lockfile can be found at + [`poetry.puzzle.provider.Provider.UNSAFE_DEPENDENCIES`](https://github.com/python-poetry/poetry/blob/master/poetry/puzzle/provider.py) +* There is an ongoing discussion of Poetry's handling of these packages at + [python-poetry/poetry#1584](https://github.com/python-poetry/poetry/issues/1584) + +#### Installing alongside an existing Poetry installation + +The plugin specifies the `poetry` package as an optional dependency to support an +externally managed Poetry installation such as in a container or CI environment. This +gives greater flexibility when using Poetry arguments like `--no-root`, `--no-dev`, or +`--remove-untracked` which can cause Poetry to uninstall itself if Poetry is specified +as a dependency of one of the packages it is managing (like this plugin). + +To have the plugin use the externally-managed Poetry package simply do not install the +`poetry` extra when installing this plugin: + +```bash +# Installing Poetry as a dependency with the plugin +poetry add tox-poetry-installer[poetry] + +# Relying on an externally managed Poetry installation +poetry add tox-poetry-installer +``` + +Note that Poetry is an optional dependency to support this use case _only_: Poetry must +be installed to the same environment as Tox for the plugin to function. To check that +the local environment has all of the required modules in scope run the below command: + +```bash +python -c '\ + import tox;\ + import tox_poetry_installer;\ + from poetry.poetry import Poetry;\ +' +``` + +**NOTE:** To force Tox to fail if Poetry is not installed, run the `tox` command with +the [`--require-poetry`](#--require-poetry) option. ## Why would I use this? From ea518d1201f9c88a2385ed0bc23cb6c71d1b936c Mon Sep 17 00:00:00 2001 From: Ethan Paul <24588726+enpaul@users.noreply.github.com> Date: Sat, 5 Dec 2020 17:00:55 -0500 Subject: [PATCH 2/6] Reorder TOC Add garbage to email to (hopefully) avoid scraping --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f0aa0dc..1fd6934 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,11 @@ dependencies to be installed using [Poetry](https://python-poetry.org/) from its Related resources: * [Poetry Python Project Manager](https://python-poetry.org/) * [Tox Automation Project](https://tox.readthedocs.io/en/latest/) +* [Other Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html) + +Similar projects: * [Poetry Dev-Dependencies Tox Plugin](https://github.com/sinoroc/tox-poetry-dev-dependencies) * [Poetry Tox Plugin](https://github.com/tkukushkin/tox-poetry) -* [Other Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html) ## Installation @@ -491,7 +493,7 @@ releases on PyPI. * To report a bug, request a feature, or ask for assistance, please [open an issue on the Github repository](https://github.com/enpaul/tox-poetry-installer/issues/new). * To report a security concern or code of conduct violation, please contact the project author - directly at **ethan dot paul at enp dot one**. + directly at **‌me [at‌] enp dot‎ ‌one**. * To submit an update, please [fork the repository](https://docs.github.com/en/enterprise/2.20/user/github/getting-started-with-github/fork-a-repo) and From 0ad5fb721988c2ee05fc1f8a31bb196f68aed10d Mon Sep 17 00:00:00 2001 From: Ethan Paul <24588726+enpaul@users.noreply.github.com> Date: Sat, 5 Dec 2020 17:49:06 -0500 Subject: [PATCH 3/6] Overhaul basic usage documentation to improve clarity Add intra document links Update to document new/updated features Add badge for downloads per month Update badge order --- README.md | 330 +++++++++++++++++++++++++++++------------------------- 1 file changed, 175 insertions(+), 155 deletions(-) diff --git a/README.md b/README.md index 1fd6934..f4e0c29 100644 --- a/README.md +++ b/README.md @@ -6,22 +6,23 @@ dependencies to be installed using [Poetry](https://python-poetry.org/) from its ⚠️ **This project is alpha software and should not be used in production environments** ⚠️ [![ci-status](https://github.com/enpaul/tox-poetry-installer/workflows/CI/badge.svg?event=push)](https://github.com/enpaul/tox-poetry-installer/actions) -[![license](https://img.shields.io/pypi/l/tox-poetry-installer)](https://opensource.org/licenses/MIT) [![pypi-version](https://img.shields.io/pypi/v/tox-poetry-installer)](https://pypi.org/project/tox-poetry-installer/) +[![pypi-downloads](https://img.shields.io/pypi/dm/tox-poetry-installer)](https://libraries.io/pypi/tox-poetry-installer) +[![license](https://img.shields.io/pypi/l/tox-poetry-installer)](https://opensource.org/licenses/MIT) [![python-versions](https://img.shields.io/pypi/pyversions/tox-poetry-installer)](https://www.python.org) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) **Documentation** -* [Installation](#installation) -* [Quick Start](#quick-start) -* [Plugin Usage](#reference-and-usage) +* [Introduction](#introduction) + * [Install](#install) + * [Quick Start](#quick-start) + * [Why would I use this?](#why-would-i-use-this) (What problems does this solve?) * [Reference](#reference) * [Configuration Options](#configuration-options) * [Command-line Arguments](#command-line-arguments) * [Errors](#errors) * [Advanced Usage](#advanced-usage) -* [Why would I use this?](#why-would-i-use-this) (What problems does this solve?) * [Developing](#developing) * [Contributing](#contributing) * [Roadmap](#roadmap) @@ -38,44 +39,88 @@ Similar projects: * [Poetry Tox Plugin](https://github.com/tkukushkin/tox-poetry) -## Installation +## Introduction -Add the plugin as a development dependency of a Poetry project: +This is a plugin to unify two great projects in the Python ecosystem: the +[Tox](https://tox.readthedocs.io/en/latest/) automation project and the +[Poetry](https://python-poetry.org) project/dependency manager. Specifically it allows +the repeatable dependency resolution and installation tools that Poetry uses to benefit +the isolated environments that Tox uses to run automated tests. The motivation to write +this plugin came from a need for a single source of truth for the versions of all +packages that should be installed to an environment. -``` -~ $: poetry add tox-poetry-installer --dev +When in use this plugin will allow a Tox environment to install its required +dependencies using the versions specified in the Poetry lockfile. This eliminates +needing to specify package versions in multiple places as well as ensures that the Tox +environment has the exact same versions of a given package as the Poetry environment. +This reduces (or hopefully eliminates) hard to debug problems caused by subtle +differences in the dependency graph of the active development environment (the one managed +by Poetry) and the automated test environment(s) created by Tox. + +To learn more about the problems this plugin aims to solve jump ahead to +[What problems does this solve?](#what-problems-does-this-solve). +Otherwise keep reading to get started. + +### Install + +The recommended way to install the plugin is to add it to a project's `pyproject.toml` +and lockfile using Poetry: + +```bash +poetry add tox-poetry-installer[poetry] --dev ``` -Confirm that the plugin is installed, and Tox recognizes it, by checking the Tox version: +**WARNING:** The below installation methods are vulnerable to the +[transient dependency issues this plugin aims to avoid](#why-would-i-use-this). It is +always recommended to install dependencies using Poetry whenever possible. + +The plugin can also be installed with pip directly, though it is recommended to always +install to a virtual environment and pin to a specific version: + +```bash +source my-venv/bi/activate +pip install tox-poetry-installer[poetry] == 0.6.0 +``` + +The plugin can also be installed using the Tox +[`requires`]((https://tox.readthedocs.io/en/latest/config.html#conf-requires)) +configuration option. Note however that dependencies installed via the `requires` option +are not handled by the plugin and will be installed the same way as a `pip install ...` +above. For this reason it is also recommended to always pin to a specific version when +using this installation method: + +```ini +# tox.ini +[tox] +requires + tox-poetry-installer[poetry] == 0.6.0 +``` + +Check that the plugin is registered by checking the Tox version: ``` ~ $: poetry run tox --version 3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py registered plugins: - tox-poetry-installer-0.5.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py + tox-poetry-installer-0.6.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py ``` -If using Pip, ensure that the plugin is installed to the same environment as Tox: +**NOTE:** Installing the `tox-poetry-installer[poetry]` extra will add the `poetry` +package as a managed environment dependency which can cause problems when the Poetry +installation is externally managed (such as in a CI or container environment). See +[Advanced Usage](#installing-alongside-an-existing-poetry-installation) for more +information on this use case. -``` -# Calling the virtualenv's 'pip' binary directly will cause pip to install to that virtualenv -~ $: /path/to/my/automation/virtualenv/bin/pip install tox -~ $: /path/to/my/automation/virtualenv/bin/pip install tox-poetry-installer -``` +### Quick Start -**Note:** While it is possible to install this plugin using Tox's -[`requires`](https://tox.readthedocs.io/en/latest/config.html#conf-requires) -configuration option, it is not recommended. Dependencies from the `requires` option are -installed using the default Tox installation backend which opens up the -[possibility of transient dependency problems](#why-would-i-use-this) in your automation -environment. +Before making any changes to `tox.ini` the project is already benefiting from having +the plugin installed: all dependencies of the root project package are installed using +the Poetry backend to all Tox environments that install the root package without any +configuration changes. - -## Quick Start - -To add dependencies from the lockfile to a Tox environment, add the option `locked_deps` -to the environment configuration and list names of dependencies (with no version -specifier) under it: +To add dependencies from the lockfile to a Tox environment, add the option +[`locked_deps`](#locked_deps) to the environment configuration and list names of +dependencies (with no version specifier) under it: ```ini [testenv] @@ -87,9 +132,9 @@ locked_deps = commands = ... ``` -The standard `deps` option can be used in parallel with the `locked_deps` option to -install unlocked dependencies (dependencies not in the lockfile) alongside locked -dependencies: +The standard [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) option +can be used in parallel with the `locked_deps` option to install unlocked dependencies +(dependencies not in the lockfile) alongside locked dependencies: ```ini [testenv] @@ -105,10 +150,105 @@ commands = ... ``` Alternatively, to quickly install all Poetry dev-dependencies to a Tox environment, add the -`install_dev_deps = true` option to the environment configuration. +[`install_dev_deps`](#install_dev_deps) option to the environment configuration: -**Note:** Regardless of the settings outlined above, all dependencies of the project package (the -one Tox is testing) will always be installed from the lockfile. +```ini +[testenv] +description = Some very cool tests +install_dev_deps = true +``` + +See the [Plugin Usage](#plugin-usage) section for more details on available +configuration options and the [Advanced Usage](#advanced-usage) section for some +unusual use cases. + +### Why would I use this? + +**The Problem** + +By default Tox uses [Pip](https://docs.python.org/3/tutorial/venv.html) to install the +[PEP-508](https://www.python.org/dev/peps/pep-0508/) compliant dependencies to a test +environment. This plugin extends the default Tox dependency installation behavior to +support installing dependencies using a Poetry-based installation method that makes use +of the dependency metadata from Poetry's lockfile. + +Environment dependencies for a Tox environment are usually specified in PEP-508 format, like +the below example: + +```ini +[testenv] +description = Some very cool tests +deps = + foo == 1.2.3 + bar >=1.3,<2.0 + baz +``` + +Let's assume these dependencies are also useful during development, so they can be added to the +Poetry environment using this command: + + ``` + poetry add --dev \ + foo==1.2.3 \ + bar>=1.3,<2.0 \ + baz + ``` + + However there is a potential problem that could arise from each of these environment + dependencies that would _only_ appear in the Tox environment and not in the Poetry + environment in use by a developer: + + * **The `foo` dependency is pinned to a specific version:** let's imagine a security + vulnerability is discovered in `foo` and the maintainers release version `1.2.4` to fix + it. A developer can run `poetry remove foo` and then `poetry add foo^1.2` to get the new + version, but the Tox environment is left unchanged. The development environment, as defined by + the lockfile, is now patched against the vulnerability but the Tox environment is not. + +* **The `bar` dependency specifies a dynamic range:** a dynamic range allows a range of + versions to be installed, but the lockfile will have an exact version specified so that + the Poetry environment is reproducible; this allows versions to be updated with + `poetry update` rather than with the `remove` and `add` commands used above. If the + maintainers of `bar` release version `1.6.0` then the Tox environment will install it + because it is valid for the specified version range. Meanwhile the Poetry environment will + continue to install the version from the lockfile until `poetry update bar` explicitly + updates it. The development environment is now has a different version of `bar` than the Tox + environment. + +* **The `baz` dependency is unpinned:** unpinned dependencies are + [generally a bad idea](https://python-poetry.org/docs/faq/#why-are-unbound-version-constraints-a-bad-idea), + but here it can cause real problems. Poetry will interpret an unbound dependency using + [the carrot requirement](https://python-poetry.org/docs/dependency-specification/#caret-requirements) + but Pip (via Tox) will interpret it as a wildcard. If the latest version of `baz` is `1.0.0` + then `poetry add baz` will result in a constraint of `baz>=1.0.0,<2.0.0` while the Tox + environment will have a constraint of `baz==*`. The Tox environment can now install an + incompatible version of `baz` and any errors that causes cannot be replicated using `poetry update`. + +All of these problems can apply not only to the dependencies specified for a Tox environment, +but also to the dependencies of those dependencies, those dependencies' dependencies, and so on. + +**The Solution** + +This plugin allows dependencies specified in Tox environment take their version directly from +the Poetry lockfile without needing an independent version to be specified in the Tox +environment configuration. The modified version of the example environment given below appears +less stable than the one presented above because it does not specify any versions for its +dependencies: + +```ini +[testenv] +description = Some very cool tests +require_locked_deps = true +locked_deps = + foo + bar + baz +``` + +However with the `tox-poetry-installer` plugin installed Tox will install these +dependencies from the Poetry lockfile so that the version installed to the Tox +environment exactly matches the version Poetry is managing. When `poetry update` updates +the lockfile with new versions of these dependencies, Tox will automatically install +these new versions without needing any changes to the configuration. ## Reference @@ -337,126 +477,6 @@ python -c '\ the [`--require-poetry`](#--require-poetry) option. -## Why would I use this? - -**Introduction** - -The lockfile is a file generated by a package manager for a project that records what -dependencies are installed, the versions of those dependencies, and any additional metadata that -the package manager needs to recreate the local project environment. This allows developers -to have confidence that a bug they are encountering that may be caused by one of their -dependencies will be reproducible on another device. In addition, installing a project -environment from a lockfile gives confidence that automated systems running tests or performing -builds are using the same environment as a developer. - -[Poetry](https://python-poetry.org/) is a project dependency manager for Python projects, and -so it creates and manages a lockfile so that its users can benefit from all the features -described above. [Tox](https://tox.readthedocs.io/en/latest/#what-is-tox) is an automation tool -that allows Python developers to run tests suites, perform builds, and automate tasks within -self-contained [Python virtual environments](https://docs.python.org/3/tutorial/venv.html). -To make these environments useful Tox supports installing dependencies in each environment. -However, since these environments are created on the fly and Tox does not maintain a lockfile, -there can be subtle differences between the dependencies a developer is using and the -dependencies Tox uses. - -This is where this plugin comes into play. - -By default Tox uses [Pip](https://docs.python.org/3/tutorial/venv.html) to install the -PEP-508 compliant dependencies to a test environment. This plugin extends the default -Tox dependency installation behavior to support installing dependencies using a Poetry-based -installation method that makes use of the dependency metadata from Poetry's lockfile. - -**The Problem** - -Environment dependencies for a Tox environment are usually specified in PEP-508 format, like -the below example: - -```ini -# from tox.ini -... - -[testenv] -description = Some very cool tests -deps = - foo == 1.2.3 - bar >=1.3,<2.0 - baz - -... -``` - -Let's assume these dependencies are also useful during development, so they can be added to the -Poetry environment using this command: - - ``` - poetry add --dev \ - foo==1.2.3 \ - bar>=1.3,<2.0 \ - baz - ``` - - However there is a potential problem that could arise from each of these environment - dependencies that would _only_ appear in the Tox environment and not in the Poetry - environment in use by a developer: - - * **The `foo` dependency is pinned to a specific version:** let's imagine a security - vulnerability is discovered in `foo` and the maintainers release version `1.2.4` to fix - it. A developer can run `poetry remove foo` and then `poetry add foo^1.2` to get the new - version, but the Tox environment is left unchanged. The development environment, as defined by - the lockfile, is now patched against the vulnerability but the Tox environment is not. - -* **The `bar` dependency specifies a dynamic range:** a dynamic range allows a range of - versions to be installed, but the lockfile will have an exact version specified so that - the Poetry environment is reproducible; this allows versions to be updated with - `poetry update` rather than with the `remove` and `add` commands used above. If the - maintainers of `bar` release version `1.6.0` then the Tox environment will install it - because it is valid for the specified version range. Meanwhile the Poetry environment will - continue to install the version from the lockfile until `poetry update bar` explicitly - updates it. The development environment is now has a different version of `bar` than the Tox - environment. - -* **The `baz` dependency is unpinned:** unpinned dependencies are - [generally a bad idea](https://python-poetry.org/docs/faq/#why-are-unbound-version-constraints-a-bad-idea), - but here it can cause real problems. Poetry will interpret an unbound dependency using - [the carrot requirement](https://python-poetry.org/docs/dependency-specification/#caret-requirements) - but Pip (via Tox) will interpret it as a wildcard. If the latest version of `baz` is `1.0.0` - then `poetry add baz` will result in a constraint of `baz>=1.0.0,<2.0.0` while the Tox - environment will have a constraint of `baz==*`. The Tox environment can now install an - incompatible version of `baz` and any errors that causes cannot be replicated using `poetry update`. - -All of these problems can apply not only to the dependencies specified for a Tox environment, -but also to the dependencies of those dependencies, those dependencies' dependencies, and so on. - -**The Solution** - -This plugin allows dependencies specified in Tox environment take their version directly from -the Poetry lockfile without needing an independent version to be specified in the Tox -environment configuration. The modified version of the example environment given below appears -less stable than the one presented above because it does not specify any versions for its -dependencies: - -```ini -# from tox.ini -... - -[testenv] -description = Some very cool tests -require_locked_deps = true -locked_deps = - foo - bar - baz - -... -``` - -However with the `tox-poetry-installer` plugin installed the `require_locked_deps = true` -setting means that Tox will install these dependencies from the Poetry lockfile so that the -version installed to the Tox environment exactly matches the version Poetry is managing. When -`poetry update` updates the lockfile with new versions of these dependencies, Tox will -automatically install these new versions without needing any changes to the configuration. - - ## Developing This project requires a developer to have Poetry version 1.0+ installed on their workstation, see From 1941a103d30d6dc4d92860ba9715a6b1d9f3cf7a Mon Sep 17 00:00:00 2001 From: Ethan Paul <24588726+enpaul@users.noreply.github.com> Date: Sat, 5 Dec 2020 18:04:14 -0500 Subject: [PATCH 4/6] Update contributor, devel, and roadmap documentation --- README.md | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index f4e0c29..32c5ade 100644 --- a/README.md +++ b/README.md @@ -479,9 +479,11 @@ the [`--require-poetry`](#--require-poetry) option. ## Developing -This project requires a developer to have Poetry version 1.0+ installed on their workstation, see +This project requires Poetry version 1.0+ on the development workstation, see the [installation instructions here](https://python-poetry.org/docs/#installation). +Local environment setup instructions: + ```bash # Clone the repository... # ...over HTTPS @@ -491,7 +493,7 @@ git clone git@github.com:enpaul/tox-poetry-installer.git # Create a the local project virtual environment and install dependencies cd tox-poetry-installer -poetry install +poetry install -E poetry # Install pre-commit hooks poetry run pre-commit install @@ -500,11 +502,18 @@ poetry run pre-commit install poetry run tox ``` +**NOTE:** Because the pre-commit hooks require dependencies in the Poetry environment it +is recommend to [launch an environment shell](https://python-poetry.org/docs/cli/#shell) +when developing the project. Alternatively, many `git` commands will need to be run from +outside of the environment shell by prefacing the command with +[`poetry run`](https://python-poetry.org/docs/cli/#run). + ## Contributing All project contributors and participants are expected to adhere to the -[Contributor Covenant Code of Conduct, Version 2](CODE_OF_CONDUCT.md). +[Contributor Covenant Code of Conduct, v2](CODE_OF_CONDUCT.md) +([external link](https://www.contributor-covenant.org/version/2/0/code_of_conduct/)). The `devel` branch has the latest (potentially unstable) changes. The [tagged versions](https://github.com/enpaul/tox-poetry-installer/releases) correspond to the @@ -537,24 +546,25 @@ for usage in production environments. Tox configuration option ([#4](https://github.com/enpaul/tox-poetry-installer/issues/4)) - [X] Add per-environment Tox configuration option to fall back to default installation backend. -- [ ] Add warnings when an unsupported Tox configuration option is detected while using the - Poetry backend. ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5)) +- [ ] ~Add warnings when an unsupported Tox configuration option is detected while using the + Poetry backend. ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5))~ - [X] Add trivial tests to ensure the project metadata is consistent between the pyproject.toml and the module constants. - [X] Update to use [poetry-core](https://github.com/python-poetry/poetry-core) and improve robustness of the Tox and Poetry module imports to avoid potentially breaking API changes in upstream packages. ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2)) -- [ ] Find and implement a way to mitigate the [UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in Poetry. - ([#6](https://github.com/enpaul/tox-poetry-installer/issues/6)) -- [ ] Fix logging to make proper use of Tox's logging reporter infrastructure ([#3](https://github.com/enpaul/tox-poetry-installer/issues/3)) +- [ ] ~Find and implement a way to mitigate the [UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in Poetry. + ([#6](https://github.com/enpaul/tox-poetry-installer/issues/6))~ +- [X] Fix logging to make proper use of Tox's logging reporter infrastructure ([#3](https://github.com/enpaul/tox-poetry-installer/issues/3)) - [X] Add configuration option for installing all dev-dependencies to a testenv ([#14](https://github.com/enpaul/tox-poetry-installer/issues/14)) ### Path to Stable Everything in Beta plus... -- [ ] Add tests for each feature version of Tox between 2.3 and 3.20 -- [ ] Add tests for Python-3.6, 3.7, and 3.8 +- [ ] Add comprehensive unit tests +- [ ] Add tests for each feature version of Tox between 3.0 and 3.20 +- [ ] Add tests for Python-3.6, 3.7, 3.8, and 3.9 - [X] Add Github Actions based CI - [ ] Add CI for CPython, PyPy, and Conda - [ ] Add CI for Linux and Windows From 8356d52c4f97fe7bae0edeb222f870c3d8365e50 Mon Sep 17 00:00:00 2001 From: Ethan Paul <24588726+enpaul@users.noreply.github.com> Date: Sat, 5 Dec 2020 18:09:01 -0500 Subject: [PATCH 5/6] =?UTF-8?q?Update=20project=20project=20status=20from?= =?UTF-8?q?=20alpha=20to=20beta=20=F0=9F=8E=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 8 ++++---- pyproject.toml | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 32c5ade..de0ba4d 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ A plugin for [Tox](https://tox.readthedocs.io/en/latest/) that allows test environment dependencies to be installed using [Poetry](https://python-poetry.org/) from its lockfile. -⚠️ **This project is alpha software and should not be used in production environments** ⚠️ +⚠️ **This project is beta software and is under active development** ⚠️ [![ci-status](https://github.com/enpaul/tox-poetry-installer/workflows/CI/badge.svg?event=push)](https://github.com/enpaul/tox-poetry-installer/actions) [![pypi-version](https://img.shields.io/pypi/v/tox-poetry-installer)](https://pypi.org/project/tox-poetry-installer/) @@ -531,10 +531,10 @@ releases on PyPI. ## Roadmap -This project is under active development and is classified as alpha software, not yet ready -for usage in production environments. +This project is under active development and is classified as beta software, ready for +production environments on a provisional basis only. -* Beta classification will be assigned when the initial feature set is finalized +* Beta classification was assigned with [v0.6.0](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.0) * Stable classification will be assigned when the test suite covers an acceptable number of use cases diff --git a/pyproject.toml b/pyproject.toml index b4ad101..06a5188 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -15,7 +15,7 @@ include = [ keywords = ["tox", "poetry", "plugin"] readme = "README.md" classifiers = [ - "Development Status :: 3 - Alpha", + "Development Status :: 4 - Beta", "Environment :: Plugins", "Framework :: tox", "Intended Audience :: Developers", From a7d9b25b621f709bd52888407cd0bbef65603b4f Mon Sep 17 00:00:00 2001 From: Ethan Paul <24588726+enpaul@users.noreply.github.com> Date: Sat, 5 Dec 2020 18:18:20 -0500 Subject: [PATCH 6/6] Fix broken links in readme --- README.md | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index de0ba4d..ee5b95c 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ differences in the dependency graph of the active development environment (the o by Poetry) and the automated test environment(s) created by Tox. To learn more about the problems this plugin aims to solve jump ahead to -[What problems does this solve?](#what-problems-does-this-solve). +[What problems does this solve?](#why-would-i-use-this). Otherwise keep reading to get started. ### Install @@ -158,7 +158,7 @@ description = Some very cool tests install_dev_deps = true ``` -See the [Plugin Usage](#plugin-usage) section for more details on available +See the [Reference](#reference) section for more details on available configuration options and the [Advanced Usage](#advanced-usage) section for some unusual use cases. @@ -166,11 +166,10 @@ unusual use cases. **The Problem** -By default Tox uses [Pip](https://docs.python.org/3/tutorial/venv.html) to install the -[PEP-508](https://www.python.org/dev/peps/pep-0508/) compliant dependencies to a test -environment. This plugin extends the default Tox dependency installation behavior to -support installing dependencies using a Poetry-based installation method that makes use -of the dependency metadata from Poetry's lockfile. +By default Tox uses Pip to install the [PEP-508](https://www.python.org/dev/peps/pep-0508/) +compliant dependencies to a test environment. This plugin extends the default Tox +dependency installation behavior to support installing dependencies using a Poetry-based +installation method that makes use of the dependency metadata from Poetry's lockfile. Environment dependencies for a Tox environment are usually specified in PEP-508 format, like the below example: @@ -292,8 +291,8 @@ hook. Whether all Poetry dev-dependencies should be installed to the environment. If `true` then all dependencies specified in the -[`dev-dependencies` section](https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies) -of `pyproject.toml` will be installed automatically. +[`dev-dependencies`](https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies) +section of `pyproject.toml` will be installed automatically. ### Command-line Arguments @@ -306,7 +305,7 @@ Indicates that Poetry is expected to be available to Tox and, if it is not, then run should fail. If provided and the `poetry` package is not installed to the same environment as the `tox` package then Tox will fail. -**NOTE:** See [Advanced Usage](installing-alongside-an-existing-poetry-installation) +**NOTE:** See [Advanced Usage](#installing-alongside-an-existing-poetry-installation) for more information. ### Errors @@ -330,7 +329,7 @@ lockfile with the `pyproject.toml` run one of * Install Poetry: ensure that `poetry` is installed to the same environment as `tox`. * Skip running the plugin: remove the `--require-poetry` flag from the runtime options. -**NOTE:** See [Advanced Usage](installing-alongside-an-existing-poetry-installation) +**NOTE:** See [Advanced Usage](#installing-alongside-an-existing-poetry-installation) for more information. #### Locked Dependency Version Conflict Error @@ -352,9 +351,8 @@ for more information. * **Cause:** Indicates that a dependency specified in the [`locked_deps`](#locked_deps) configuration option in `tox.ini` could not be found in the Poetry lockfile. * **Resolution options:** - * Add the dependency to the lockfile: run `poetry add `; - see [the Poetry documentation](https://python-poetry.org/docs/cli/#add) for more - information. + * Add the dependency to the lockfile: run + [`poetry add `](https://python-poetry.org/docs/cli/#add). * Do not install the dependency: remove the item from the `locked_deps` list in `tox.ini`. @@ -366,9 +364,9 @@ for more information. `pyproject.toml` * **Resolution options:** * Configure the extra: add a section for the named extra to the - [`extras` section of `pyproject.toml`](https://python-poetry.org/docs/pyproject/#extras) - and optionally assign dependencies to the named extra using the - [`--optional` dependency setting](https://python-poetry.org/docs/cli/#options_3). + [`extras`](https://python-poetry.org/docs/pyproject/#extras) section of + `pyproject.toml` and optionally assign dependencies to the named extra using the + [`--optional`](https://python-poetry.org/docs/cli/#options_3) dependency setting. * Remove the extra: remove the item from the `extras` list in `tox.ini`. #### Locked Dependencies Required Error @@ -434,10 +432,10 @@ are four packages classified as "unsafe" by Poetry and excluded from the lockfil When one of these packages is encountered by the plugin a warning will be logged and _**the package will not be installed to the environment**_. If the unsafe package is required for the environment then it will need to be specified as an unlocked -dependency using the [`deps`](https://github.com/python-poetry/poetry/releases/tag/1.1.4) +dependency using the [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) configuration option in `tox.ini`, ideally with an exact pinned version. -* The set of packages excluded from the Poetry lockfile can be found at +* The set of packages excluded from the Poetry lockfile can be found in [`poetry.puzzle.provider.Provider.UNSAFE_DEPENDENCIES`](https://github.com/python-poetry/poetry/blob/master/poetry/puzzle/provider.py) * There is an ongoing discussion of Poetry's handling of these packages at [python-poetry/poetry#1584](https://github.com/python-poetry/poetry/issues/1584)