From 60551366f12f0cab2623f1faf0e695513f488552 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 28 Jan 2025 12:28:48 +0000 Subject: [PATCH 01/19] Added specific examples of packages in the motivation section --- peps/pep-0771.rst | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index ee5e42413b9..aca0776882f 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -41,7 +41,14 @@ control can use ``package``. However, in practice, many users are unaware of the ``[recommended]`` syntax, placing the burden on them to know this for a typical installation. Having a way to have recommended dependencies be installed by default while providing a way for users to request a more minimal installation -would satisfy this use case. The present PEP will describe a solution for this use case. +would satisfy this use case, and the present PEP will describe a solution to this. + +Real-world examples of packages that encourage users to include extra +dependencies by default include for example: + +* `astropy `_, a core package for astronomy, which recommends that users install ``astropy[recommended]`` +* `fastapi `_, a web framework, which recommends that users install ``fastapi[standard]`` +* `tensorflow `_, a machine learning platform, which recommends that users install ``tensorflow[and-cuda]`` Packages supporting multiple backends or frontends -------------------------------------------------- @@ -70,6 +77,31 @@ or backend package. There is currently no mechanism in Python packaging infrastructure to disallow conflicting or incompatible extras to be installed, and this PEP does not change that. +Real-world examples of packages that allow users to select different backends or +frontends with extras include: + +* `kivy `_, an app development framework, which supports + various backends for e.g. text, image, video, and needs at least one backend + to be installed in order to be functional. Currently the solution is to + recommend that users install ``kivy[base]`` + +* `napari `_, an interactive viewer for multi-dimensional images + which can work with either `PyQt5 `_, `PyQt6 + `_, `PySide2 + `_, or `PySide6 + `_ for the frontend, and will not work if none of them are + installed. Users are instructed to install ``napari[all]`` for an easy installation + which will include one of the frontends, and have the option to explicitly + install a different frontend by doing e.g. ``napari[pyside2]``. + +* `glueviz `_, a data visualization application which + can also use different Qt frontends as for napari, and will not work if none + of them are installed. Users are instructed to install ``glueviz[qt]`` to + include one of the Qt backends by default. + +In all three cases, installing the package without any extras results in a +broken installation. + Rationale ========= From 5898fa95cbd5ddddeca093baab9b1e050355ecf2 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 28 Jan 2025 12:29:24 +0000 Subject: [PATCH 02/19] Rename default-optional-dependencies to default-optional-dependency-keys --- peps/pep-0771.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index aca0776882f..dbffca06655 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -149,12 +149,12 @@ New key in ``[project]`` metadata table A new key will be added to the ``[project]`` table in project metadata as originally defined in :pep:`621` and now defined in the `PyPA specifications `_. This key will be named -``default-optional-dependencies`` with the following description: +``default-optional-dependency-keys`` with the following description: * `TOML `_ type: Array of strings * Corresponding core metadata field: ``Default-Extra`` -Each string in ``default-optional-dependencies`` must be the name of an extra +Each string in ``default-optional-dependency-keys`` must be the name of an extra defined in `optional-dependencies `_, and each extra in this array will be converted to a matching ``Default-Extra`` @@ -164,7 +164,7 @@ produce the example ``Default-Extra`` entries presented in the previous section .. code-block:: toml [project] - default-optional-dependencies = [ + default-optional-dependency-keys = [ "recommended", ] @@ -173,7 +173,7 @@ and: .. code-block:: toml [project] - default-optional-dependencies = [ + default-optional-dependency-keys = [ "backend1", "backend2", "backend3" @@ -258,7 +258,7 @@ How to teach this The rule above regarding only installing default extras when no extras are explicitly specified, combined with the introduction of the -``Default-Extra:`` keyword and ``default-optional-dependencies`` metadata key +``Default-Extra:`` keyword and ``default-optional-dependency-keys`` metadata key allows us to address several different use cases. Below we take a look at the two specific use cases raised in the `Motivation`_ section and how package maintainers should be taught to address these. @@ -277,7 +277,7 @@ would choose to have this be included as a default extra: .. code-block:: toml [project] - default-optional-dependencies = [ + default-optional-dependency-keys = [ "recommended" ] @@ -326,7 +326,7 @@ for each backend or frontend, and provide a default, e.g.: .. code-block:: toml [project] - default-optional-dependencies = [ + default-optional-dependency-keys = [ "backend1" ] @@ -364,7 +364,7 @@ This could be achieved with e.g: .. code-block:: toml [project] - default-optional-dependencies = [ + default-optional-dependency-keys = [ "recommended" ] From 14f8560ee0b818a30ee1f655149aaf1c23354789 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 28 Jan 2025 13:06:02 +0000 Subject: [PATCH 03/19] Rename 'How to teach this' to 'Examples of Usage' and add concrete examples in each sub-section --- peps/pep-0771.rst | 130 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 122 insertions(+), 8 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index dbffca06655..f3c947147e0 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -46,9 +46,21 @@ would satisfy this use case, and the present PEP will describe a solution to thi Real-world examples of packages that encourage users to include extra dependencies by default include for example: -* `astropy `_, a core package for astronomy, which recommends that users install ``astropy[recommended]`` -* `fastapi `_, a web framework, which recommends that users install ``fastapi[standard]`` -* `tensorflow `_, a machine learning platform, which recommends that users install ``tensorflow[and-cuda]`` +* `astropy `_, a core package for astronomy, which + recommends that users install ``astropy[recommended]``. Some of the + dependencies in ``recommended``, such as `scipy `_, are + used in a number of places in astropy for common + functionality and scipy should ideally be a required dependency, but there has + been reluctance to so as it is not a lightweight dependency and advanced users + who want minimal installs want the ability to opt out of it (however, as + described above, this places a burden on the majority of regular uses to + satisfy a use case for advanced users). +* `fastapi `_, a web framework, which recommends + that users install ``fastapi[standard]`` in order to include commonly needed + dependencies. +* `tensorflow `_, a machine learning platform, which + recommends that users install ``tensorflow[and-cuda]`` since CUDA support is + commonly needed by users. Packages supporting multiple backends or frontends -------------------------------------------------- @@ -253,15 +265,15 @@ Security Implications There are no known security implications for this PEP. -How to teach this +Examples of usage ================= -The rule above regarding only installing default extras when no extras -are explicitly specified, combined with the introduction of the +The rule in `Overriding default extras`_ regarding only installing default extras +when no extras are explicitly specified, combined with the introduction of the ``Default-Extra:`` keyword and ``default-optional-dependency-keys`` metadata key allows us to address several different use cases. Below we take a look at the -two specific use cases raised in the `Motivation`_ section and how package -maintainers should be taught to address these. +two specific use cases raised in the `Motivation`_ section and how these +can be addressed as a result of the changes described in `Specification`_. Recommended dependencies and minimal installations -------------------------------------------------- @@ -309,6 +321,34 @@ Maintainers would have the choice as to whether to offer the capability to do a minimal installation or not - in some cases, such as highlighted in the next section, this might not be desirable. +To take one of the concrete examples from the `Motivation`_ section, the +`astropy`_ package could declare the existing ``recommended`` +extra as a default extra and provide a ``minimal`` extra: + +.. code-block:: toml + + [project] + default-optional-dependency-keys = [ + "recommended" + ] + + [project.optional-dependencies] + minimal = [] + recommended = [ + "scipy", + "..." + ] + +meaning that installing:: + + pip install astropy + +would then get optional but important optional dependencies such as `scipy`_ +would get installed. Advanced users who want a minimal install could then use:: + + pip install astropy[minimal] + + Packages requiring at least one backend or frontend --------------------------------------------------- @@ -347,6 +387,43 @@ If packages can support e.g. multiple backends at the same time, and some of the backends should always be installed, then the dependencies for these must be given as required dependencies rather than using the default extras mechanism. +To take one of the concrete examples mentioned in `Motivation`_, the `napari`_ package +could define a configuration such as: + +.. code-block:: toml + + [project] + default-optional-dependency-keys = [ + "pyqt5" + ] + + [project.optional-dependencies] + pyqt5 = [ + "PyQt5", + "..." + ] + pyside2 = [ + "PySide2", + "..." + ] + pyqt6 = [ + "PyQt6", + "..." + ] + pyside6 = [ + "PySide6", + "..." + ] + +meaning that:: + + pip install napari + +would work out-of-the-box, but there would be a mechanism for users to explicitly +specify a frontend, e.g.:: + + pip install napari[pyside6] + Supporting minimal installations while not always removing default extras ------------------------------------------------------------------------- @@ -382,6 +459,43 @@ This could be achieved with e.g: The ability for a package to reference itself in the extras is supported by existing Python packaging tools. +Once again taking a concrete example from `Motivation`_, astropy defines several +other extras, including for example ``jupyter``, which adds packages that +enhance the user experience inside `Jupyter `_-based +environments. However, it is likely that users opting in to this extra would not +want recommended dependencies to *not* be installed. The following configuration +would solve this case: + +.. code-block:: toml + + [project] + default-optional-dependency-keys = [ + "recommended" + ] + + [project.optional-dependencies] + minimal = [] + recommended = [ + "scipy", + "..." + ] + jupyter = [ + "astropy[recommended]", + "ipywidgets", + "..." + ] + +Users installing:: + + pip install astropy[jupyter] + +would then get the same as:: + + pip install astropy[recommended, jupyter] + +How to teach this +================= + Teaching package authors ------------------------ From fa12de71f9d50e84c2f5f67624d37e67c03eccc4 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 29 Jan 2025 11:31:04 +0000 Subject: [PATCH 04/19] Expand backward compatibility and how to teach this --- peps/pep-0771.rst | 318 ++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 276 insertions(+), 42 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index f3c947147e0..a0017d17980 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -112,7 +112,8 @@ frontends with extras include: include one of the Qt backends by default. In all three cases, installing the package without any extras results in a -broken installation. +broken installation (and this is a commonly reported issue for some of these +packages). Rationale ========= @@ -123,9 +124,9 @@ the community for several years, including in `this DPO thread as well as in numerous issues and pull requests. The solution that will be presented below: -* does not break backward-compatibility of existing packaging infrastructure * is an opt-in solution which means that package maintainers can choose whether or not to use it -* is flexible enough to accommodate both of the major use cases described in `Motivation`_. +* is flexible enough to accommodate both of the major use cases described in `Motivation`_ +* does not introduce new syntax compared to :pep:`508` (which would have serious backward-compatibility implications) It is the only solution out of all those discussed that meets all three criteria. @@ -200,21 +201,27 @@ extras are not installed. Otherwise, the default extras are used. For example, if a package defines an ``extra1`` default extra as well as a non-default ``extra2`` -extra, then if a user were to install the package with:: +extra, then if a user were to install the package with: - pip install package +.. code-block:: console -the ``extra1`` dependency would be included. If the user instead uses:: + $ pip install package - pip install package[extra2] +the ``extra1`` dependency would be included. If the user instead uses: + +.. code-block:: console + + $ pip install package[extra2] then the ``extra1`` extra would not be installed. If the same package is specified multiple times in an installation command or dependency tree, the default extras must be installed if any of the instances of -the package are specified without extras. For instance:: +the package are specified without extras. For instance: + +.. code-block:: console - pip install package package[extra2] + $ pip install package package[extra2] should install the default extras. @@ -246,19 +253,69 @@ extras. Backward Compatibility ====================== -All package specification cases valid under :pep:`508` will remain valid. -Therefore, this proposal is fully backward-compatible with existing :pep:`508` -usage. +There are several different aspects related to backward compatibility that need +to be considered. + +Packages not using default extras +--------------------------------- + +Once support for this PEP is added to tools in the packaging ecosystem, packages +that do not make use of default extras will continue to work as-is and there +should be no break in compatibility. + +Packages using default extras +----------------------------- Once packages start defining default extras, those defaults will only be honored with recent versions of packaging tools which implement this PEP, but those -packages will remain fully backward-compatible with older packaging tools - with -the only difference that the default extras will not be installed automatically -when older packaging tools are used. +packages will remain installable with older packaging tools -- with the main +difference being that the default extras will not be installed automatically +when older packaging tools are used. As described in `How to teach this`_, +package authors need to carefully evaluate when and how they adopt +the default extra feature depending on their user base, as some actions (such as +moving a required dependency to a default extra) will likely result in breakage +for users if a significant fraction of them are still using older package +installers that do not support default extras. In this sense, package authors +should be aware that this feature, if used in certain ways, can cause +backward-compatibility issues for users, and they are thus responsible for +ensuring that they minimize the impact to users. + +Packaging-related tools +----------------------- + +The most significant backward-compatibility aspect is related to assumptions +pakaging tools make about extras -- specifically, this PEP changes the +assumption that extras are no longer exclusively additive in terms of adding +dependencies to the dependency tree, and specifying some extras can result in +fewer dependencies being installed. + +A specific example of change in behavior can be seen with `pip`_: consider a +package ``package`` which has a required dependency of ``numpy``, a (default) +extra called ``recommended`` which includes ``scipy``, and a ``minimal`` extra +which does not contain any dependencies. If a user installs ``package[minimal]``, +only ``package`` and ``numpy`` will be installed. If a user then does: + +.. code-block:: console + + $ pip freeze > requirements.txt + +then ``requirements.txt`` will contain e.g.:: + + package==1.0.2 + numpy==2.1.0 + +However, if the user then installs the requirements from this file using: + +.. code-block:: console -The only conceptual backward-compatibility issue to consider is the fact that -this PEP changes extras to no longer be strictly additive, in that specifying -an extra such as ``minimal`` could result in fewer packages being installed. + $ pip install -r requirements.txt + +then pip will install ``package`` (which will include the default extra) as well +as ``numpy``, so the final environment will contain ``scipy``. A solution in this +specific case is for the user to pass ``--no-deps`` to ``pip install`` to avoid +resolving the dependency tree, but the point here is to illustrate that there +may be changes in behavior in packaging tools due to the change in the +assumption about what impact an extra can have. Security Implications ===================== @@ -272,7 +329,7 @@ The rule in `Overriding default extras`_ regarding only installing default extra when no extras are explicitly specified, combined with the introduction of the ``Default-Extra:`` keyword and ``default-optional-dependency-keys`` metadata key allows us to address several different use cases. Below we take a look at the -two specific use cases raised in the `Motivation`_ section and how these +specific use cases raised in the `Motivation`_ section and how these can be addressed as a result of the changes described in `Specification`_. Recommended dependencies and minimal installations @@ -339,14 +396,18 @@ extra as a default extra and provide a ``minimal`` extra: "..." ] -meaning that installing:: +meaning that installing: + +.. code-block:: console - pip install astropy + $ pip install astropy would then get optional but important optional dependencies such as `scipy`_ -would get installed. Advanced users who want a minimal install could then use:: +would get installed. Advanced users who want a minimal install could then use: + +.. code-block:: console - pip install astropy[minimal] + $ pip install astropy[minimal] Packages requiring at least one backend or frontend @@ -415,14 +476,18 @@ could define a configuration such as: "..." ] -meaning that:: +meaning that: - pip install napari +.. code-block:: console + + $ pip install napari would work out-of-the-box, but there would be a mechanism for users to explicitly -specify a frontend, e.g.:: +specify a frontend, e.g.: + +.. code-block:: console - pip install napari[pyside6] + $ pip install napari[pyside6] Supporting minimal installations while not always removing default extras ------------------------------------------------------------------------- @@ -462,8 +527,8 @@ existing Python packaging tools. Once again taking a concrete example from `Motivation`_, astropy defines several other extras, including for example ``jupyter``, which adds packages that enhance the user experience inside `Jupyter `_-based -environments. However, it is likely that users opting in to this extra would not -want recommended dependencies to *not* be installed. The following configuration +environments. However, it is likely that users opting in to this extra would still +want recommended dependencies to be installed. The following configuration would solve this case: .. code-block:: toml @@ -485,22 +550,162 @@ would solve this case: "..." ] -Users installing:: +Users installing: - pip install astropy[jupyter] +.. code-block:: console -would then get the same as:: + $ pip install astropy[jupyter] - pip install astropy[recommended, jupyter] +would then get the same as: + +.. code-block:: console + + $ pip install astropy[recommended, jupyter] How to teach this ================= -Teaching package authors ------------------------- - -Packages making use of any of the approaches above must ensure that they -properly document the options available to users in terms of installation. +Package authors +--------------- + +While the mechanism used to define extras and the associated rule about when to +use it are simple, package authors need to carefully consider several points +before adopting this capability in their packages, as this has implications for +backward-compatibility. + +Supporting older versions of package installers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Package installers such as `pip`_ or `uv `_ will not +necessarily implement support for default extras at the same time, and once they +do it is likely that package authors will want to keep supporting users who do +not have the most recent version of a package installer. In this case, the +following recommendations would apply: + +* Moving a package from being a required dependency to a default extra would be + a breaking change, because older versions of package installers would not + recognise the concept of default extras, and would then install the package + with fewer dependencies, which could affect users that would have been relying + on these. Therefore, changing dependencies from required to a default extra in + an established package should only be done in future once the developers only + want to support users with installers that implement this PEP. + +* Making an existing extra become a default should be safer, such as making + ``recommended`` in `astropy`_ be a default extra, but in order to support users + with older versions of package installers, the documentation should still mention + the extra explicitly as long as possible (until it is clear that most/all users + are using package installers that implement this PEP). There is no downside to + keeping the extra be explicitly mentioned, but this will ensure that users with + modern tooling who do not read documentation (which may be a non-negligeable + fraction of the user community) will start getting the recommended + dependencies by default. + +* Adding a new extra, whether it be ``minimal`` or another new extra that is to + be the default, comes with the same caveats that it does prior to this PEP, which + is that users will only be able to use this extra for releases that define + this extra. This might seem obvious, but consider a package that has a version + 1.0 prior to using default extras. Suppose that package now defines + ``minimal`` in 2.0, then downstream users and packages that want to depend on + a minimal version of the package cannot declare the following dependency:: + + packaage[minimal]>=1.0 + + because ``package[minimal]==1.0`` does not exist (in practice, pip ignores + unknown extras, so it might be possible to do this, but there is no guarantee + that other tools won't error on an unrecognized extra). + + The easiest solution to this problem is for package authors to define a no-op + ``minimal`` extra as soon as possible, even if only planning to adopt default + extras further down the road, as it will allow ``package[minimal]`` to work + for versions prior to when defaults were adopted. + +Avoiding adding too many default dependencies +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One temptation for authors might be to include many dependencies by default since +they can provide a way to opt out from these. We recommend however that authors +carefully consider what is included by default to avoid unecessarily bloating +installations and complicating dependency trees. Using default extras does not +mean that all extras need to be defaults, and there is still scope for users to +explicitly opt in to non-default extras. + +Inheriting from default extras +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If package authors choose to make an extra be installed by default, it is important +that they are aware that if users explicitly specify another extra, the default may +not be installed, unless they use the approach described in `Supporting minimal +installations while not always removing default extras`_. + +There are cases, such as the interchangeable backends, or the ``minimal`` extras, +where ignoring the default if an extra is explicitly specified is the right +thing to do. However, for other cases, such as using default extras to include +recommended dependencies while still providing a way to do minimal installs, it +may be that many of the other extras *should* explicitly 'inherit' the default +extra(s), so package authors should carefully consider in which cases they want +the default extras to be installed. + +Incompatible extras +^^^^^^^^^^^^^^^^^^^ + +In some cases, it may be that packages have extras that are mutually +incompatible. In this case, we recommend against making using the default extra +feature for any extra that contains a dependency that could be incompatible with +another. + +Consider a package that has extras ``package[A]`` and ``package[B]``. Users +could already currently try and install ``package[A]`` and ``package[B]`` or +``package[A,B]`` which would result in a broken installation, however it would +at least be explicit that both extras were being installed. Making ``A`` into a +default extra however could lead to unintuitive issues. A user could do: + +.. code-block:: console + + $ pip install package + $ pip install package[B] + +and end up with a broken installation, even though A and B were never explicitly +both installed. For this reason, we recommend against using default extras +for dependencies where this is likely to be an issue. + +Package users +------------- + +Package users should be provided with clear installation instructions that show +what extras are available for packages and how they behave, for example +explaining that by default some recommended dependencies or a given frontend or +backend will be installed, and how to opt out of this or override defaults, +depending what is available. + +Repackagers of Python libraries +------------------------------- + +The impact on individuals who repackage Python libraries for different +distributions, such as `conda `_, `Homebrew +`_, linux package installers (such as ``apt`` and ``yum``) and +so on, needs to be considered. Not all package distributions have mechanisms +that would line up with the approach described. In fact, some distributions such +as conda, do not even have the concept of extras. + +There are two cases to consider here: + +* In cases where the repackaging is done by hand, such as for a number of conda-forge + recipes, and especially where there is no equivalent to extras, the + introduction of default extras should not have a large impact since manual + decisions already have to be made as to which dependencies to include (for + example, the conda-forge recipe for the `astropy`_ package mentioned in the + `Motivation`_ includes all the ``recommended`` dependencies by default since + there is no way for users to explicitly request them otherwise). + +* In cases where the repackaging is done in an automated, way, distribution maintainers + will need to carefully consider how to treat default extras, and this may + imply a non-negligible amount of work and discussion. + +It is impossible for a PEP such as this to exhaustively consider each of the +different package distributions. However, ultimately, default extras should be +understood as how package authors would like their package to be installed for +the majority of users, and this should inform decisions about how default extras +should be handled, whether manually or automatically. Reference Implementation ======================== @@ -616,14 +821,18 @@ Relying on tooling to deselect any default extras ------------------------------------------------- Another option to unselect extras would be to implement this at the -level of packaging tools. For instance, pip could include an option such as:: +level of packaging tools. For instance, pip could include an option such as: + +.. code-block:: console - pip install package --no-default-extras + $ pip install package --no-default-extras This option could apply to all or specific packages, similar to -the ``--no-binary`` option, e.g.,:: +the ``--no-binary`` option, e.g.,: - pip install package --no-default-extras :all: +.. code-block:: console + + $ pip install package --no-default-extras :all: The advantage of this approach is that tools supporting default extras could also support unselecting them. This approach would be similar to the ``--no-install-recommends`` @@ -648,6 +857,31 @@ highlighted in `How to teach this`_, there may also be cases where package maintainers do not actually want to support an installation without any extras, for example in cases where at least one backend or frontend must be installed. +Open issues +=========== + +Should ``package[]`` disable default extras? +-------------------------------------------- + +Currently, the PEP as written above does not allow ``package[]`` to be +equivalent to installing the package with no extras, but there would be some +benefits to allowing this: + +* It would avoid different packages using different names for a 'no default' + extras (e.g. ``minimal``, ``no-default``, ``no-defaults``) and reduce the + burden for people who don’t want to have to scan through source code or + documentation to figure out whether there is the equivalent of a ``minimal`` + extra. + +* It would allow people who want to use existing packages as-is and future + versions of those packages with no default extras to use ``package[]`` because + that syntax works right now, so it would provide a consistent way over time to + get a minimal install. + +On the other hand, it is not clear at this point whether any tools are currently +relying on ``package[]`` being identical to ``package`` in a way that would +break compatibility if this was done, so this needs to be investigated. + Copyright ========= From c93dac55cc91beac5d8730fb44765f2128c483d2 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 29 Jan 2025 11:34:40 +0000 Subject: [PATCH 05/19] Add note about unrecognized extras --- peps/pep-0771.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index a0017d17980..e49e219707f 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -229,6 +229,13 @@ Note that ``package[]`` would continue to be equivalent to ``package`` and would not be provided as a way to install without default extras (see the `Rejected Ideas`_ section for the rationale). +We also note that some tools such as `pip`_ currently ignore unrecognized +extras, and emit a warning to the user to indicate that the extra has not been +recognized. In this case, if no extras are recognized as being valid in a +dependency specification, we recommend that this case be treated as if the user +had not passed any explicit extras, and therefore any default extras should be +installed. + Installing without default extras --------------------------------- From da37547aa2ccf340c1e31ea0a568e3bbb7cca5ee Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 29 Jan 2025 11:45:11 +0000 Subject: [PATCH 06/19] Tweak link --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index e49e219707f..e13b60cd5a9 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -241,7 +241,7 @@ Installing without default extras In some cases, package maintainers may want to facilitate installing packages without any default extras. In this case, as will be shown in more detail in -`How to teach this`_, the best approach is to define an extra which could be +`Examples of usage`_, the best approach is to define an extra which could be called e.g. ``minimal`` or ``nodefault`` (the naming would be up to the package maintainer) which would be an empty set of dependencies. If this extra is specified, no default extras will be included, so that e.g. ``package[minimal]`` From eed0a9842ead77e4d102c371c100afaacfea3c1d Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 30 Jan 2025 10:21:53 +0000 Subject: [PATCH 07/19] Apply suggestions from code review Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0771.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index e13b60cd5a9..65ecd84da63 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -41,10 +41,10 @@ control can use ``package``. However, in practice, many users are unaware of the ``[recommended]`` syntax, placing the burden on them to know this for a typical installation. Having a way to have recommended dependencies be installed by default while providing a way for users to request a more minimal installation -would satisfy this use case, and the present PEP will describe a solution to this. +would satisfy this use case, and this PEP describes a solution to this. Real-world examples of packages that encourage users to include extra -dependencies by default include for example: +dependencies by default include: * `astropy `_, a core package for astronomy, which recommends that users install ``astropy[recommended]``. Some of the @@ -121,12 +121,12 @@ Rationale A number of possible solutions have been extensively and vigorously discussed by the community for several years, including in `this DPO thread `__ -as well as in numerous issues and pull requests. The solution that will be +as well as in numerous issues and pull requests. The solution that is presented below: * is an opt-in solution which means that package maintainers can choose whether or not to use it * is flexible enough to accommodate both of the major use cases described in `Motivation`_ -* does not introduce new syntax compared to :pep:`508` (which would have serious backward-compatibility implications) +* re-uses the syntax from :pep:`508` It is the only solution out of all those discussed that meets all three criteria. @@ -260,8 +260,6 @@ extras. Backward Compatibility ====================== -There are several different aspects related to backward compatibility that need -to be considered. Packages not using default extras --------------------------------- From 600da1cde2537f254581573ae32aaa070374fb74 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 30 Jan 2025 10:25:33 +0000 Subject: [PATCH 08/19] Make examples into a sub-section of the specification --- peps/pep-0771.rst | 152 +++++++++++++++++++++++----------------------- 1 file changed, 76 insertions(+), 76 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 65ecd84da63..4a4efa87af9 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -257,88 +257,18 @@ be installed but will not work if none are provided, so a package maintainer may therefore not want to provide an option to install the package without any extras. -Backward Compatibility -====================== - - -Packages not using default extras ---------------------------------- - -Once support for this PEP is added to tools in the packaging ecosystem, packages -that do not make use of default extras will continue to work as-is and there -should be no break in compatibility. - -Packages using default extras ------------------------------ - -Once packages start defining default extras, those defaults will only be honored -with recent versions of packaging tools which implement this PEP, but those -packages will remain installable with older packaging tools -- with the main -difference being that the default extras will not be installed automatically -when older packaging tools are used. As described in `How to teach this`_, -package authors need to carefully evaluate when and how they adopt -the default extra feature depending on their user base, as some actions (such as -moving a required dependency to a default extra) will likely result in breakage -for users if a significant fraction of them are still using older package -installers that do not support default extras. In this sense, package authors -should be aware that this feature, if used in certain ways, can cause -backward-compatibility issues for users, and they are thus responsible for -ensuring that they minimize the impact to users. - -Packaging-related tools ------------------------ - -The most significant backward-compatibility aspect is related to assumptions -pakaging tools make about extras -- specifically, this PEP changes the -assumption that extras are no longer exclusively additive in terms of adding -dependencies to the dependency tree, and specifying some extras can result in -fewer dependencies being installed. - -A specific example of change in behavior can be seen with `pip`_: consider a -package ``package`` which has a required dependency of ``numpy``, a (default) -extra called ``recommended`` which includes ``scipy``, and a ``minimal`` extra -which does not contain any dependencies. If a user installs ``package[minimal]``, -only ``package`` and ``numpy`` will be installed. If a user then does: - -.. code-block:: console - - $ pip freeze > requirements.txt - -then ``requirements.txt`` will contain e.g.:: - - package==1.0.2 - numpy==2.1.0 - -However, if the user then installs the requirements from this file using: - -.. code-block:: console - - $ pip install -r requirements.txt - -then pip will install ``package`` (which will include the default extra) as well -as ``numpy``, so the final environment will contain ``scipy``. A solution in this -specific case is for the user to pass ``--no-deps`` to ``pip install`` to avoid -resolving the dependency tree, but the point here is to illustrate that there -may be changes in behavior in packaging tools due to the change in the -assumption about what impact an extra can have. - -Security Implications -===================== - -There are no known security implications for this PEP. - -Examples of usage -================= +Examples +-------- The rule in `Overriding default extras`_ regarding only installing default extras when no extras are explicitly specified, combined with the introduction of the ``Default-Extra:`` keyword and ``default-optional-dependency-keys`` metadata key allows us to address several different use cases. Below we take a look at the specific use cases raised in the `Motivation`_ section and how these -can be addressed as a result of the changes described in `Specification`_. +can now be addressed: Recommended dependencies and minimal installations --------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ First, we consider the case of packages that want recommended but not strictly required dependencies installed by default, while also @@ -416,7 +346,7 @@ would get installed. Advanced users who want a minimal install could then use: Packages requiring at least one backend or frontend ---------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As described in `Motivation`_, some packages may support multiple backends and/or frontends, and in some cases it may be desirable to ensure that there @@ -495,7 +425,7 @@ specify a frontend, e.g.: $ pip install napari[pyside6] Supporting minimal installations while not always removing default extras -------------------------------------------------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An additional case we consider here is where a package maintainer wants to support minimal installations without any extras, but also wants to support having users @@ -567,6 +497,76 @@ would then get the same as: $ pip install astropy[recommended, jupyter] + +Backward Compatibility +====================== + +Packages not using default extras +--------------------------------- + +Once support for this PEP is added to tools in the packaging ecosystem, packages +that do not make use of default extras will continue to work as-is and there +should be no break in compatibility. + +Packages using default extras +----------------------------- + +Once packages start defining default extras, those defaults will only be honored +with recent versions of packaging tools which implement this PEP, but those +packages will remain installable with older packaging tools -- with the main +difference being that the default extras will not be installed automatically +when older packaging tools are used. As described in `How to teach this`_, +package authors need to carefully evaluate when and how they adopt +the default extra feature depending on their user base, as some actions (such as +moving a required dependency to a default extra) will likely result in breakage +for users if a significant fraction of them are still using older package +installers that do not support default extras. In this sense, package authors +should be aware that this feature, if used in certain ways, can cause +backward-compatibility issues for users, and they are thus responsible for +ensuring that they minimize the impact to users. + +Packaging-related tools +----------------------- + +The most significant backward-compatibility aspect is related to assumptions +pakaging tools make about extras -- specifically, this PEP changes the +assumption that extras are no longer exclusively additive in terms of adding +dependencies to the dependency tree, and specifying some extras can result in +fewer dependencies being installed. + +A specific example of change in behavior can be seen with `pip`_: consider a +package ``package`` which has a required dependency of ``numpy``, a (default) +extra called ``recommended`` which includes ``scipy``, and a ``minimal`` extra +which does not contain any dependencies. If a user installs ``package[minimal]``, +only ``package`` and ``numpy`` will be installed. If a user then does: + +.. code-block:: console + + $ pip freeze > requirements.txt + +then ``requirements.txt`` will contain e.g.:: + + package==1.0.2 + numpy==2.1.0 + +However, if the user then installs the requirements from this file using: + +.. code-block:: console + + $ pip install -r requirements.txt + +then pip will install ``package`` (which will include the default extra) as well +as ``numpy``, so the final environment will contain ``scipy``. A solution in this +specific case is for the user to pass ``--no-deps`` to ``pip install`` to avoid +resolving the dependency tree, but the point here is to illustrate that there +may be changes in behavior in packaging tools due to the change in the +assumption about what impact an extra can have. + +Security Implications +===================== + +There are no known security implications for this PEP. + How to teach this ================= From f7cc0c8d5de3d53fe8ae8503d01bd8827a76df82 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 30 Jan 2025 10:37:58 +0000 Subject: [PATCH 09/19] Make examples more concise, and move examples into a sub-section of the specification --- peps/pep-0771.rst | 61 ++++++++++++++++------------------------------- 1 file changed, 21 insertions(+), 40 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 4a4efa87af9..4673d6d2ddb 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -6,6 +6,8 @@ Status: Draft Type: Standards Track Topic: Packaging Created: 13-Jan-2025 +Post-History: + `15-Jan-2025 `__, Abstract ======== @@ -43,24 +45,12 @@ typical installation. Having a way to have recommended dependencies be installed by default while providing a way for users to request a more minimal installation would satisfy this use case, and this PEP describes a solution to this. -Real-world examples of packages that encourage users to include extra -dependencies by default include: - -* `astropy `_, a core package for astronomy, which - recommends that users install ``astropy[recommended]``. Some of the - dependencies in ``recommended``, such as `scipy `_, are - used in a number of places in astropy for common - functionality and scipy should ideally be a required dependency, but there has - been reluctance to so as it is not a lightweight dependency and advanced users - who want minimal installs want the ability to opt out of it (however, as - described above, this places a burden on the majority of regular uses to - satisfy a use case for advanced users). -* `fastapi `_, a web framework, which recommends - that users install ``fastapi[standard]`` in order to include commonly needed - dependencies. -* `tensorflow `_, a machine learning platform, which - recommends that users install ``tensorflow[and-cuda]`` since CUDA support is - commonly needed by users. +Examples of packages that demonstrate this pattern by encouraging users to +include extra dependencies by default include: + +* `astropy `_: ``astropy[recommended]``. +* `fastapi `_: ``fastapi[standard]`` +* `tensorflow `_: ``tensorflow[and-cuda]`` Packages supporting multiple backends or frontends -------------------------------------------------- @@ -71,7 +61,10 @@ frontend. A package might need at least one backend or frontend to be installed in order to be functional, but may be flexible on which backend or frontend this is. Concrete examples of such frontends or backends include: -* The Qt frontend library, which requires either PySide or PyQt to be installed +* The Qt frontend library, which can be provided by `PyQt5 `_, `PyQt6 + `_, `PySide2 + `_, or `PySide6 + `_ * BLAS/LAPACK, which have different possible implementations (e.g. OpenBLAS, and MKL) * FFT libraries, which also have different implementations (e.g. ``scipy.fft`` and pyFFTW) @@ -89,27 +82,14 @@ or backend package. There is currently no mechanism in Python packaging infrastructure to disallow conflicting or incompatible extras to be installed, and this PEP does not change that. -Real-world examples of packages that allow users to select different backends or -frontends with extras include: +Examples of packages that require at least one backend or frontend to work and +recommend a default extra to install a backend or frontend include: -* `kivy `_, an app development framework, which supports - various backends for e.g. text, image, video, and needs at least one backend - to be installed in order to be functional. Currently the solution is to - recommend that users install ``kivy[base]`` +* `kivy `_: ``kivy[base]`` -* `napari `_, an interactive viewer for multi-dimensional images - which can work with either `PyQt5 `_, `PyQt6 - `_, `PySide2 - `_, or `PySide6 - `_ for the frontend, and will not work if none of them are - installed. Users are instructed to install ``napari[all]`` for an easy installation - which will include one of the frontends, and have the option to explicitly - install a different frontend by doing e.g. ``napari[pyside2]``. +* `napari `_: ``napari[all]`` -* `glueviz `_, a data visualization application which - can also use different Qt frontends as for napari, and will not work if none - of them are installed. Users are instructed to install ``glueviz[qt]`` to - include one of the Qt backends by default. +* `glueviz `_: ``glueviz[qt]``. In all three cases, installing the package without any extras results in a broken installation (and this is a commonly reported issue for some of these @@ -241,7 +221,7 @@ Installing without default extras In some cases, package maintainers may want to facilitate installing packages without any default extras. In this case, as will be shown in more detail in -`Examples of usage`_, the best approach is to define an extra which could be +`Examples`_, the best approach is to define an extra which could be called e.g. ``minimal`` or ``nodefault`` (the naming would be up to the package maintainer) which would be an empty set of dependencies. If this extra is specified, no default extras will be included, so that e.g. ``package[minimal]`` @@ -337,8 +317,9 @@ meaning that installing: $ pip install astropy -would then get optional but important optional dependencies such as `scipy`_ -would get installed. Advanced users who want a minimal install could then use: +would then get optional but important optional dependencies such as `scipy +`_ would get installed. Advanced users who want a minimal +install could then use: .. code-block:: console From f0bbf2f805ebd93a8054f2c093ee9a46cf5a6b04 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 30 Jan 2025 10:40:27 +0000 Subject: [PATCH 10/19] Fixed link --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 4673d6d2ddb..f775902ed1b 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -100,7 +100,7 @@ Rationale A number of possible solutions have been extensively and vigorously discussed by the community for several years, including in `this DPO thread -`__ +`__ as well as in numerous issues and pull requests. The solution that is presented below: From 39671c0c2b086e3c9554d06a5f10e9cd5273e747 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 3 Feb 2025 10:46:01 +0000 Subject: [PATCH 11/19] Apply suggestions from @willingc Co-authored-by: Carol Willing Co-authored-by: Pradyun Gedam --- peps/pep-0771.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index f775902ed1b..ec9afffe877 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -187,7 +187,7 @@ extra, then if a user were to install the package with: $ pip install package -the ``extra1`` dependency would be included. If the user instead uses: +the default ``extra1`` dependency would be included. If the user instead uses: .. code-block:: console @@ -510,7 +510,7 @@ Packaging-related tools ----------------------- The most significant backward-compatibility aspect is related to assumptions -pakaging tools make about extras -- specifically, this PEP changes the +packaging tools make about extras -- specifically, this PEP changes the assumption that extras are no longer exclusively additive in terms of adding dependencies to the dependency tree, and specifying some extras can result in fewer dependencies being installed. @@ -555,8 +555,8 @@ Package authors --------------- While the mechanism used to define extras and the associated rule about when to -use it are simple, package authors need to carefully consider several points -before adopting this capability in their packages, as this has implications for +use it are clear, package authors need to carefully consider several points +before adopting this capability in their packages, to avoid inadvertently breaking backward-compatibility. Supporting older versions of package installers @@ -594,7 +594,7 @@ following recommendations would apply: ``minimal`` in 2.0, then downstream users and packages that want to depend on a minimal version of the package cannot declare the following dependency:: - packaage[minimal]>=1.0 + package[minimal]>=1.0 because ``package[minimal]==1.0`` does not exist (in practice, pip ignores unknown extras, so it might be possible to do this, but there is no guarantee @@ -605,7 +605,7 @@ following recommendations would apply: extras further down the road, as it will allow ``package[minimal]`` to work for versions prior to when defaults were adopted. -Avoiding adding too many default dependencies +Avoiding the addition of many default dependencies ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ One temptation for authors might be to include many dependencies by default since @@ -635,7 +635,7 @@ Incompatible extras ^^^^^^^^^^^^^^^^^^^ In some cases, it may be that packages have extras that are mutually -incompatible. In this case, we recommend against making using the default extra +incompatible. In this case, we recommend against using the default extra feature for any extra that contains a dependency that could be incompatible with another. From 34d42d3dbcffd6bebe6415da88686541ef090bca Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 3 Feb 2025 11:42:14 +0000 Subject: [PATCH 12/19] Implement comments by @willingc --- peps/pep-0771.rst | 96 +++++++++++++++++++++++++++++------------------ 1 file changed, 59 insertions(+), 37 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index ec9afffe877..2a1b84e370c 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -177,33 +177,39 @@ Overriding default extras ------------------------- If extras are explicitly given in a dependency specification, the default -extras are not installed. Otherwise, the default extras are used. +extras are ignored. Otherwise, the default extras are installed. -For example, if a package -defines an ``extra1`` default extra as well as a non-default ``extra2`` -extra, then if a user were to install the package with: +For example, if a package defines an ``extra1`` default extra as well as a +non-default ``extra2`` extra, then if a user were to install the package with: .. code-block:: console $ pip install package -the default ``extra1`` dependency would be included. If the user instead uses: +the default ``extra1`` dependency would be included. If the user instead +installs the package with: .. code-block:: console $ pip install package[extra2] -then the ``extra1`` extra would not be installed. +then the ``extra2`` extra would be installed but the default ``extra1`` extra +would be ignored. If the same package is specified multiple times in an installation command or dependency tree, the default extras must be installed if any of the instances of -the package are specified without extras. For instance: +the package are specified without extras. For instance, if one installs a +package ``spam`` where ``package`` appears several times in the dependency +tree:: -.. code-block:: console - - $ pip install package package[extra2] + spam + ├── tomato + │ ├── package[extra2] + └── egg + └── package -should install the default extras. +then the default extra should be installed because ``package`` appears at least +once with no extras specified. Note that ``package[]`` would continue to be equivalent to ``package`` and would not be provided as a way to install without default extras (see the `Rejected @@ -211,10 +217,18 @@ Ideas`_ section for the rationale). We also note that some tools such as `pip`_ currently ignore unrecognized extras, and emit a warning to the user to indicate that the extra has not been -recognized. In this case, if no extras are recognized as being valid in a -dependency specification, we recommend that this case be treated as if the user -had not passed any explicit extras, and therefore any default extras should be -installed. +recognized, e.g: + +.. code-block:: console + + $ pip install package[non-existent-extra] + WARNING: package 3.0.0 does not provide the extra 'non-existent-extra' + ... + +For tools that behave like this (rather than raising an error), if an extra is +recognized as invalid in a dependency specification, it should be ignored and +treated as if the user has not passed an explicit extra. If none of the provided +extras are valid, default extras should be installed. Installing without default extras --------------------------------- @@ -240,12 +254,9 @@ extras. Examples -------- -The rule in `Overriding default extras`_ regarding only installing default extras -when no extras are explicitly specified, combined with the introduction of the -``Default-Extra:`` keyword and ``default-optional-dependency-keys`` metadata key -allows us to address several different use cases. Below we take a look at the -specific use cases raised in the `Motivation`_ section and how these -can now be addressed: +In this section we take a look at the use cases described in the `Motivation`_ +section and how these can now be addressed by using the specification outlined +above. Recommended dependencies and minimal installations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -293,9 +304,12 @@ Maintainers would have the choice as to whether to offer the capability to do a minimal installation or not - in some cases, such as highlighted in the next section, this might not be desirable. -To take one of the concrete examples from the `Motivation`_ section, the -`astropy`_ package could declare the existing ``recommended`` -extra as a default extra and provide a ``minimal`` extra: +To take a one of the concrete examples of package from the `Motivation`_ +section, the `astropy`_ package defines a ``recommended`` extra that users are +currently instructed to install in the default installation instructions. +With this PEP, the ``recommended`` extra could be declared as being a default +extra, and a new ``minimal`` extra could be provided for users wishing to +retain the ability to install only the strictly required dependencies: .. code-block:: toml @@ -317,8 +331,8 @@ meaning that installing: $ pip install astropy -would then get optional but important optional dependencies such as `scipy -`_ would get installed. Advanced users who want a minimal +would then also install optional but important optional dependencies such as `scipy +`_. Advanced users who want a minimal install could then use: .. code-block:: console @@ -365,7 +379,11 @@ the backends should always be installed, then the dependencies for these must be as required dependencies rather than using the default extras mechanism. To take one of the concrete examples mentioned in `Motivation`_, the `napari`_ package -could define a configuration such as: +can make use of one of `PyQt5`_, `PyQt6`_, `PySide2`_, or `PySide6`_, and users currently +need to explicitly specify ``napari[all]`` in order to have one of these be installed, +or e.g., ``napari[pyqt5]`` to explicitly specify one of the frontend packages. Installing +``napari`` with no extras results in a non-functional package. With this PEP, ``napari`` +could define the following configuration: .. code-block:: toml @@ -398,8 +416,8 @@ meaning that: $ pip install napari -would work out-of-the-box, but there would be a mechanism for users to explicitly -specify a frontend, e.g.: +would work out-of-the-box, but there would still be a mechanism for users to +explicitly specify a frontend, e.g.: .. code-block:: console @@ -440,12 +458,14 @@ This could be achieved with e.g: The ability for a package to reference itself in the extras is supported by existing Python packaging tools. -Once again taking a concrete example from `Motivation`_, astropy defines several -other extras, including for example ``jupyter``, which adds packages that +Once again considering a concrete example, `astropy`_ could with this PEP define a +``recommended`` extra (as described in `Recommended dependencies and minimal +installations`_). However, it also defines other extras, including for example +``jupyter``, which adds packages that enhance the user experience inside `Jupyter `_-based -environments. However, it is likely that users opting in to this extra would still -want recommended dependencies to be installed. The following configuration -would solve this case: +environments. It is possible that users opting in to this extra would still want +recommended dependencies to be installed. In this case, the following +configuration would solve this case: .. code-block:: toml @@ -496,7 +516,9 @@ Once packages start defining default extras, those defaults will only be honored with recent versions of packaging tools which implement this PEP, but those packages will remain installable with older packaging tools -- with the main difference being that the default extras will not be installed automatically -when older packaging tools are used. As described in `How to teach this`_, +when older packaging tools are used. + +As described in `How to teach this`_, package authors need to carefully evaluate when and how they adopt the default extra feature depending on their user base, as some actions (such as moving a required dependency to a default extra) will likely result in breakage @@ -606,7 +628,7 @@ following recommendations would apply: for versions prior to when defaults were adopted. Avoiding the addition of many default dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ One temptation for authors might be to include many dependencies by default since they can provide a way to opt out from these. We recommend however that authors @@ -647,7 +669,7 @@ default extra however could lead to unintuitive issues. A user could do: .. code-block:: console - $ pip install package + $ pip install package # this installs package[A] $ pip install package[B] and end up with a broken installation, even though A and B were never explicitly From 71d14b90415601125b316de69472d8f699c27d51 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Mon, 3 Feb 2025 12:01:27 +0000 Subject: [PATCH 13/19] Added a sub-section on circular dependencies --- peps/pep-0771.rst | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 2a1b84e370c..7ff46d85590 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -676,6 +676,37 @@ and end up with a broken installation, even though A and B were never explicitly both installed. For this reason, we recommend against using default extras for dependencies where this is likely to be an issue. +Circular dependencies +^^^^^^^^^^^^^^^^^^^^^ + +Authors need to take special care when circular dependencies are present. For instance, +consider the following dependency tree:: + + package1 + └── package2 + └── package1 + +If ``package1`` has a default extra named ``recommended`` and a ``minimal`` +extra which is empty, then: + +.. code-block:: console + + $ pip install package1[minimal] + +will still result in the ``recommended`` extra being installed if ``package2`` +continues to depend on ``package1`` (with no extras specified). If the dependency +tree was updated to instead be:: + + package1 + └── package2 + └── package1[minimal] + +Then ``package1`` would no longer be installable with tools that do not yet +implement this PEP (if those tools would fail on unrecognized extras). Authors +therefore need to carefully consider a migration plan, coordinating with the +authors of ``package2``. + + Package users ------------- From d0eacc3e189b846370a15015da546ae803abb5e1 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Tue, 4 Feb 2025 10:23:14 +0000 Subject: [PATCH 14/19] Added intro to 'How to teach this' --- peps/pep-0771.rst | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 7ff46d85590..d34a1760427 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -573,6 +573,26 @@ There are no known security implications for this PEP. How to teach this ================= +This section outlines information that should be made available to people in +different groups in the community in relation to the implementation of this PEP. +Some aspects described below will be relevant even before the PEP is fully +implemented in packaging tools as there are some preparations that can be done +in advance of this implementation to facilitate any potential transition later +on. The groups covered below are: + +- `Package end users`_ +- `Package authors`_ +- `Packaging repository maintainers`_ + +Package end users +----------------- + +Package users should be provided with clear installation instructions that show +what extras are available for packages and how they behave, for example +explaining that by default some recommended dependencies or a given frontend or +backend will be installed, and how to opt out of this or override defaults, +depending what is available. + Package authors --------------- @@ -707,17 +727,8 @@ therefore need to carefully consider a migration plan, coordinating with the authors of ``package2``. -Package users -------------- - -Package users should be provided with clear installation instructions that show -what extras are available for packages and how they behave, for example -explaining that by default some recommended dependencies or a given frontend or -backend will be installed, and how to opt out of this or override defaults, -depending what is available. - -Repackagers of Python libraries -------------------------------- +Packaging repository maintainers +-------------------------------- The impact on individuals who repackage Python libraries for different distributions, such as `conda `_, `Homebrew From b940c7b1f4371c0213491a75a84132ac238ab8b2 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 5 Feb 2025 14:13:47 +0000 Subject: [PATCH 15/19] Fix extraneous period Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index d34a1760427..231b0f845b2 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -48,7 +48,7 @@ would satisfy this use case, and this PEP describes a solution to this. Examples of packages that demonstrate this pattern by encouraging users to include extra dependencies by default include: -* `astropy `_: ``astropy[recommended]``. +* `astropy `_: ``astropy[recommended]`` * `fastapi `_: ``fastapi[standard]`` * `tensorflow `_: ``tensorflow[and-cuda]`` From 1484c244eca6353928b95bf3bd99f96880c97e60 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Wed, 5 Feb 2025 14:14:30 +0000 Subject: [PATCH 16/19] Fix extraneous period Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 231b0f845b2..357db39c513 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -89,7 +89,7 @@ recommend a default extra to install a backend or frontend include: * `napari `_: ``napari[all]`` -* `glueviz `_: ``glueviz[qt]``. +* `glueviz `_: ``glueviz[qt]`` In all three cases, installing the package without any extras results in a broken installation (and this is a commonly reported issue for some of these From 0b7521f9d08f96a172792142b1a6cb6192d4aa70 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 6 Feb 2025 16:57:22 +0000 Subject: [PATCH 17/19] Added Discussion-To --- peps/pep-0771.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 357db39c513..d3f26181e17 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -5,6 +5,7 @@ Sponsor: Pradyun Gedam Status: Draft Type: Standards Track Topic: Packaging +Discussions-To: https://discuss.python.org/t/pep-771-default-extras-for-python-software-packages/79706 Created: 13-Jan-2025 Post-History: `15-Jan-2025 `__, From 65d0057ddd878088284ceb7a276d4bbd194e31f5 Mon Sep 17 00:00:00 2001 From: Thomas Robitaille Date: Thu, 6 Feb 2025 16:59:01 +0000 Subject: [PATCH 18/19] Fix order --- peps/pep-0771.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index d3f26181e17..68c0ca9b8e7 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -2,10 +2,10 @@ PEP: 771 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam +Discussions-To: https://discuss.python.org/t/pep-771-default-extras-for-python-software-packages/79706 Status: Draft Type: Standards Track Topic: Packaging -Discussions-To: https://discuss.python.org/t/pep-771-default-extras-for-python-software-packages/79706 Created: 13-Jan-2025 Post-History: `15-Jan-2025 `__, From 36aafa4b10e5692c0d4a39687bf667e2ffeca17f Mon Sep 17 00:00:00 2001 From: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Date: Thu, 6 Feb 2025 17:00:15 +0000 Subject: [PATCH 19/19] Post-History --- peps/pep-0771.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0771.rst b/peps/pep-0771.rst index 68c0ca9b8e7..5755afc377c 100644 --- a/peps/pep-0771.rst +++ b/peps/pep-0771.rst @@ -2,13 +2,14 @@ PEP: 771 Title: Default Extras for Python Software Packages Author: Thomas Robitaille , Jonathan Dekhtiar Sponsor: Pradyun Gedam -Discussions-To: https://discuss.python.org/t/pep-771-default-extras-for-python-software-packages/79706 +Discussions-To: https://discuss.python.org/t/79706/ Status: Draft Type: Standards Track Topic: Packaging Created: 13-Jan-2025 Post-History: `15-Jan-2025 `__, + `06-Feb-2025 `__, Abstract ========