From ba28051c3f6bc335b200c9643a2187c4817aba9d Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 15:25:52 -0800 Subject: [PATCH 01/12] Clarify some things --- peps/pep-0751.rst | 99 ++++++++++++++++++++++++++++------------------- 1 file changed, 59 insertions(+), 40 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 713dfdca4ea..a18a88bb4ca 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -15,7 +15,7 @@ Replaces: 665 Abstract ======== -This PEP proposes a new file format for dependency specification +This PEP proposes a new file format for specifying dependencies to enable reproducible installation in a Python environment. The format is designed to be human-readable and machine-generated. Installers consuming the file should be able to calculate what to install without the need for dependency @@ -38,16 +38,16 @@ Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current environment while PDM, Poetry, and uv can/try to lock for multiple environments at once. There's also concerns around the lack of secure defaults in the face of -supply chain attacks (e.g., including hashes for files). +supply chain attacks (e.g. including hashes for files). The lack of a standard also has some drawbacks. For instance, any tooling that wants to work with lock files must choose which format to support, potentially -leaving users unsupported (e.g., Dependabot_ only supporting select tools, +leaving users unsupported (e.g. Dependabot_ only supporting select tools, same for cloud providers who can do dependency installations on your behalf, etc.). It also impacts portability between tools, which causes vendor lock-in. By not having compatibility and interoperability it fractures tooling around lock files where both users and tools have to choose what lock file format to -use upfront and making it costly to use/switch to other formats (e.g., tooling +use upfront and making it costly to use/switch to other formats (e.g. tooling around auditing a lock file). Rallying around a single format removes this cost/barrier. @@ -55,8 +55,8 @@ The closest the community has to a standard are pip's `requirements files`_ which all the aforementioned tools either use directly as their file format or export to (i.e. ``requirements.txt``). Unfortunately, the format is not a standard but is supported by convention. It's also designed very much for pip's -needs, limiting its flexibility and ease of use (e.g., it's a bespoke file -format). Lastly, it is not secure by default (e.g., file hash support is +needs, limiting its flexibility and ease of use (e.g. it's a bespoke file +format). Lastly, it is not secure by default (e.g. file hash support is entirely an opt-in feature, you have to tell pip to not look for other dependencies outside of what's in the requirements file, etc.). @@ -79,7 +79,7 @@ file. It should also lead to faster installs which are much more frequent than creating a lock file. The data in the file should be consumable by tools not written in Python. This -allows for e.g., cloud hosting providers to write their own tool to perform +allows for e.g. cloud hosting providers to write their own tool to perform installations in their preferred programming language. The file format should promote good security defaults. As the format is not @@ -87,7 +87,7 @@ meant to be human-writable, this means having tools provide security-related details is reasonable and not a costly burden. The contents of a lock file should be able to replace the vast majority of uses -of `requirements files`_ that are used as a lock file (e.g., what +of `requirements files`_ when used as a lock file (e.g. what pip-tools_ and ``pip freeze`` emit). This means the file format specified by this PEP can, at minimum, act as an export target for tools which have their own internal lock file format. @@ -102,7 +102,7 @@ File Name --------- A lock file MUST be named :file:`pylock.toml` or match the regular expression -``r"pylock\.(.+)\.toml"`` if a name for the lock file is desired or if multiple +``r"^pylock\.([^.]+)\.toml$"`` if a name for the lock file is desired or if multiple lock files exist. The use of the ``.toml`` file extension is to make syntax highlighting in editors easier and to reinforce the fact that the file format is meant to be human-readable. The prefix and suffix of a named file MUST be @@ -111,7 +111,7 @@ e.g.: .. code-block:: Python - if filename.startswith("pylock.") and filename.endswith(".toml"): + if len(filename) > 11 and filename.startswith("pylock.") and filename.endswith(".toml"): name = filename.removeprefix("pylock.").removesuffix(".toml") The lock file(s) SHOULD be located in the directory as appropriate for the scope @@ -133,8 +133,8 @@ recorded in a consistent order. As well, tools SHOULD sort arrays in consistent order. Usage of inline tables SHOULD also be kept consistent. -``metadata-version`` -==================== +``lock-version`` +================ - **Type**: string; value of ``"1.0"`` - **Required?**: yes @@ -257,7 +257,7 @@ order. Usage of inline tables SHOULD also be kept consistent. contains. - Tools MAY choose to not support version control systems, both from a locking and/or installation perspective. -- Tools SHOULD provide a way for users to opt out of using version control +- Tools SHOULD provide a way for users to opt in/out of using version control systems. @@ -311,7 +311,7 @@ order. Usage of inline tables SHOULD also be kept consistent. - **Required?**: yes - **Inspiration**: :ref:`packaging:direct-url-data-structure-vcs` - The exact commit/revision number that is to be installed. -- If the VCS supports commit-hash based revision identifiers, such commit-hash +- If the VCS supports commit-hash based revision identifiers, such a commit-hash MUST be used as the commit ID in order to reference an immutable version of the source code. @@ -324,7 +324,7 @@ order. Usage of inline tables SHOULD also be kept consistent. - **Inspiration**: :ref:`packaging:direct-url-data-structure-subdirectories` - The subdirectory within the :ref:`source tree ` where - the project root of the project is (e.g., the location of the + the project root of the project is (e.g. the location of the ``pyproject.toml`` file). - The path MUST be relative to the root of the source tree structure. @@ -341,7 +341,7 @@ order. Usage of inline tables SHOULD also be kept consistent. contains. - Tools MAY choose to not support local directories, both from a locking and/or installation perspective. -- Tools SHOULD provide a way for users to opt out of using local directories. +- Tools SHOULD provide a way for users to opt in/out of using local directories. ``packages.directory.path`` @@ -381,7 +381,7 @@ See ``packages.vcs.subdirectory``. :ref:`packaging:source-distribution-format-source-tree`. - Tools MAY choose to not support archive files, both from a locking and/or installation perspective. -- Tools SHOULD provide a way for users to opt out of using archive files. +- Tools SHOULD provide a way for users to opt in/out of using archive files. ``packages.archive.url`` @@ -400,9 +400,11 @@ See ``packages.vcs.path``. '''''''''''''''''''''''''' - **Type**: integer -- **Required?**: yes +- **Required?**: no - **Inspiration**: uv_, :ref:`packaging:simple-repository-api` - The size of the archive file. +- Tools SHOULD provide the file size when reasonably possible (e.g. the file + size is available via the `Content-Length` header from a HEAD_ HTTP request). ``[packages.archive.hashes]`` @@ -433,9 +435,12 @@ See ``packages.vcs.subdirectory``. - **Required?**: no - **Inspiration**: uv_ - The base URL for the package index from :ref:`packaging:simple-repository-api` - where the sdist and/or wheels were found (e.g., ``https://pypi.org/simple/``). + where the sdist and/or wheels were found (e.g. ``https://pypi.org/simple/``). - When possible, this SHOULD be specified to assist with generating - `software bill of materials`_ (aka SBOMs). + `software bill of materials`_ -- aka SBOMs -- and to assist in finding a file + if a URL ceases to be valid. +- Tools MAY support installing from an index if the URL recorded for a specific + file is no longer vaild (e.g. returns a 404 HTTP error code). ``[packages.sdist]`` @@ -449,18 +454,16 @@ See ``packages.vcs.subdirectory``. package. - Tools MAY choose to not support sdist files, both from a locking and/or installation perspective. -- Tools SHOULD provide a way for users to opt out of using sdist files. +- Tools SHOULD provide a way for users to opt in/out of using sdist files. ``packages.sdist.name`` ''''''''''''''''''''''' - **Type**: string -- **Required?**: no +- **Required?**: yes - **Inspiration**: PDM_, Poetry_, uv_ - The file name of the :ref:`packaging:source-distribution-format-sdist` file. -- The name SHOULD be recorded when it does not follow the standard outlined in - :ref:`packaging:source-distribution-format-sdist`. ``packages.sdist.upload-time`` @@ -560,9 +563,8 @@ See ``packages.archive.hashes``. - Similar usage as that of the ``[tool]`` table from the :ref:`packaging:pyproject-toml-spec`, but at the package version level instead of at the lock file level (which is also available via ``[tool]``). -- Useful for scoping package version/release details (e.g., recording signing - identities to then use to verify package integrity separately from where the - package is hosted, prototyping future extensions to this file format, etc.). +- Data recorded in the table MUST be disposable (i.e. it MUST NOT affect + installation). ``[tool]`` @@ -571,8 +573,7 @@ See ``packages.archive.hashes``. - **Type**: table - **Required?**: no - **Inspiration**: :ref:`packaging:pyproject-tool-table` -- Same usage as that of the equivalent ``[tool]`` table from the - :ref:`packaging:pyproject-toml-spec`. +- See ``packages.tool``. ------- @@ -629,13 +630,13 @@ The following outlines the steps to be taken to install from a lock file (while the requirements are prescriptive, the general steps and order are a suggestion): -#. Check if the metadata version specified by ``metadata-version`` is supported; +#. Check if the metadata version specified by ``lock-version`` is supported; an error or warning MUST be raised as appropriate. #. If ``requires-python`` is specified, check that the environment being installed for meets the requirement; an error MUST be raised if it is not met. #. If ``environments`` is specified, check that at least one of the environment - marker expressions is satisfied; error MUST be raised if no expression is + marker expressions is satisfied; an error MUST be raised if no expression is satisfied. #. For each package listed in ``[[packages]]``: @@ -666,6 +667,7 @@ a suggestion): - Else if ``archive`` is set: #. Get the file. + #. Validate the file size and hash. #. Build the package, respecting ``subdirectory``. #. Install. @@ -674,12 +676,27 @@ a suggestion): #. Look for the appropriate wheel file based on ``name``; if one is not found then move on to ``sdist`` or an error MUST be raised about a lack of source for the project. - #. Get the file. + #. Get the file: + + - If `path` is set, use it. + - If `url` is set, try to use it, optionally falling back to using + `packages.index` or some tool-specific mechanism to download the + selected wheel file (tools MUST NOT try to change what wheel file to + download based on what's available; what file to install should be + determined in an offline fashion for reproducibility). + + #. Validate the file size and hash. #. Install. - Else if no ``wheel`` file is found or ``sdist`` is solely set: #. Get the file. + + - If `path` is set, use it. + - If `url` is set, try to use it, optionally falling back to using + `packages.index` or some tool-specific mechanism to download the + file. + #. Validate the file size and hash. #. Build the package. #. Install. @@ -739,15 +756,15 @@ However, this PEP does not solve all potential security concerns. One potential concern is tampering with a lock file. If a lock file is not kept in source control and properly audited, a bad actor could change the file in nefarious ways (e.g., point to a malware version of a package). Tampering could -also occur in transit to e.g., a cloud provider who will perform an installation +also occur in transit to e.g. a cloud provider who will perform an installation on the user's behalf. Both could be mitigated by signing the lock file either within the file in a ``[tool]`` entry or via a side channel external to the lock file itself. This PEP does not do anything to prevent a user from installing incorrect packages. While including many details to help in auditing a package's inclusion, -there isn't any mechanism to stop e.g., name confusion attacks via -typosquatting. Tools may be able to provide some UX to help with this (e.g., by +there isn't any mechanism to stop e.g. name confusion attacks via +typosquatting. Tools may be able to provide some UX to help with this (e.g. by providing download counts for a package). @@ -758,7 +775,7 @@ How to Teach This Users should be informed that when they ask to install some package, the package may have its own dependencies, those dependencies may have dependencies, and so on. Without writing down what gets installed as part of installing the -package they requested, things could change from underneath them (e.g., package +package they requested, things could change from underneath them (e.g. package versions). Changes to the underlying dependencies can lead to accidental breakage of their code. Lock files help deal with that by providing a way to write down what was installed so you can install the exact same thing in the @@ -766,7 +783,7 @@ future. Having what to install written down also helps in collaborating with others. By agreeing to a lock file's contents, everyone ends up with the same packages -installed. This helps make sure no one relies on e.g., an API that's only +installed. This helps make sure no one relies on e.g. an API that's only available in a certain version that not everyone working on the project has installed. @@ -800,11 +817,11 @@ Recording the dependency graph for installation purposes A previous version of this PEP recorded the dependency graph of packages instead of a set of packages to install. The idea was that by recording the dependency graph you not only got more information, but it provided more flexibility by -supporting more features innately (e.g., platform-specific dependencies without +supporting more features innately (e.g. platform-specific dependencies without explicitly propagating markers). In the end, though, it was deemed to add complexity that wasn't worth the cost -(e.g., it impacted the ease of auditing for details which were not necessary +(e.g. it impacted the ease of auditing for details which were not necessary for this PEP to reach its goals). @@ -919,7 +936,7 @@ was that by specifying a single algorithm it would help with auditing the file when a specific hash algorithm was mandated for use. In the end there was some objection to this idea. Typically, it centered around -the cost of rehashing large wheel files (e.g., PyTorch). There was also concern +the cost of rehashing large wheel files (e.g. PyTorch). There was also concern about making hashing decisions upfront on the installer's behalf which they may disagree with. In the end it was deemed better to have flexibility and let people audit the lock file as they see fit. @@ -1299,7 +1316,9 @@ Copyright This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. +.. _Content-Length: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length .. _Dependabot: https://docs.github.com/en/code-security/dependabot +.. _HEAD: https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD .. _PDM: https://pypi.org/project/pdm/ .. _pip-tools: https://pypi.org/project/pip-tools/ .. _Poetry: https://python-poetry.org/ From fb3031951d460d8d1c15b804b4d9d037e6ec82e1 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 15:40:00 -0800 Subject: [PATCH 02/12] Add attestation identities --- peps/pep-0751.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index a18a88bb4ca..8afdfdd4731 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -553,6 +553,30 @@ See ``packages.archive.size``. See ``packages.archive.hashes``. +``[[packages.attestation-identities]]`` +--------------------------------------- + +- **Type**: array of tables +- **Required?**: no +- **Inspiration**: :ref:`packaging:provenance-object` +- A recording of the attestations for **any** file recorded for this package. +- If available, tools SHOULD include the attestation identities found. +- Publisher-specific keys are to be included in the table as-is + (i.e. top-level), following the spec at + :ref:`packaging:index-hosted-attestations`. + + +``packages.attestation-identites.kind`` +''''''''''''''''''''''''''''''''''''''' + +- **Type**: string +- **Required?**: yes +- **Inspiration**: :ref:`packaging:provenance-object` +- The unique identity of the Trusted Publisher. + + + + ``[packages.tool]`` ------------------- From d43dc4cf75c795d02ac6caa61e591f177c23a02a Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 16:37:48 -0800 Subject: [PATCH 03/12] Reject obvious open issues --- peps/pep-0751.rst | 155 +++++++++++++++++++--------------------------- 1 file changed, 65 insertions(+), 90 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 8afdfdd4731..2aab38adf73 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -1006,10 +1006,6 @@ decide it wasn't worth trying to do in this PEP upfront. Instead, a future PEP could propose a solution. -=========== -Open Issues -=========== - -------------- Simplification -------------- @@ -1017,13 +1013,12 @@ Simplification Drop recording the package version ================================== -As this is written, the package version is optional since it can only be -reliably recorded when an sdist of wheel file is used. And since both sources -record the version in file names it is technically redundant. +The package version is optional since it can only be reliably recorded when an +sdist of wheel file is used. And since both sources record the version in file +names it is technically redundant. -But having the version explicitly called out could be viewed as helping with -auditing by not having to find and parse file names (especially if an sdist -file name doesn't conform to :ref:`packaging:source-distribution-format-sdist`). +But in discussions it was decided the version number is useful for auditing +enough to still state it separately. Drop the requirement to specify the location of an sdist and/or wheels @@ -1032,13 +1027,11 @@ Drop the requirement to specify the location of an sdist and/or wheels At least one person has commented how their work has unstable URLs for all sdists and wheels. As such, they have to search for all files at install regardless of where the file was found previously. Dropping the requirement to -provide the URL or path to a file would help solve the issue of recording +provide the URL or path to a file would have helped solve the issue of recording known-bad information. -To support this, though, would require installation to support finding files via -a package index or some other mechanism specified outside of this PEP. The -former adds complexity (discussed as another open issue), while the latter means -this PEP cannot fully explain the installation process. +The decision to allow tools to look for a file in other ways beyond the URL +provided alleviated the need to make the URL optional. Drop requiring file size and hashes @@ -1050,12 +1043,8 @@ making the file size and hashes optional -- very likely through some opt-out mechanism -- then they could continue to produce lock files that meet this PEP's requirements. -As it weakens security by not making hashes and file sizes mandatory, it -somewhat dilutes the purpose of this PEP. It also only works with external -projects if the creator of the lock file is external to the company modifying -the files **and** chose to leave out hashes. It also is only beneficial if -the file modifications are not idempotent, thus causing random changes in -hashes and file size. +The decision was made that this weakens security too much. It also prevents +installing files from alternative locations. Drop recording the sdist file name @@ -1073,18 +1062,7 @@ when the recorded file location is no longer available (while sdist file names are now standardized thanks to :pep:`625`, that has only been true since 2020 and thus there are many older sdists with names that may not be guessable). - -Support installing files via a package index -============================================ - -With a package index URL and a file name, one can find the location of a file -at install-time. This not only allows recording the URL or path optional, it -could also act as a fallback if the original location is no longer valid. - -This does increase the burden on tools performing installation as they would -now have to support this fallback. It could be made as an optional feature, -although the chances are people will expect it to be implemented as it shouldn't -increase the complexity of an installer drastically. +The decision was made to require the sdist file name out of simplicity. Make ``packages.wheels`` a table @@ -1123,26 +1101,14 @@ For example: "numpy-2.0.1-cp312-cp312-win_amd64.whl" = {upload-time = 2024-07-21T13:40:17.532627Z, url = "https://files.pythonhosted.org/packages/b5/59/f6ad378ad85ed9c2785f271b39c3e5b6412c66e810d2c60934c9f/numpy-2.0.1-cp312-cp312-win_amd64.whl", size = 16255757, hashes = {sha256 = "bb2124fdc6e62baae159ebcfa368708867eb56806804d005860b6007388df171"} -It's entirely a structural change which some may (not) prefer. +In general, though, people did not prefer this over the approach this PEP has +taken. + ---------------- Self-Referential ---------------- -Record what tool created the lock file -====================================== - -Right now the PEP does not record any details about the tool that created a -file. That's out of simplicity reasons only. Which tool is used may be -implicitly recorded by a ``[tool]`` table. - -But one could record various amounts of details about the tool to help recreate -the file. Key details like tool name, the installation requirements when the -tool is hosted on PyPI (encoded as :ref:`packaging:dependency-specifiers`), and -the command used to create the file would allow another tool to re-run the tool. -It would also help discover what tool was used. - - Drop the ``[tool]`` table ========================= @@ -1150,25 +1116,14 @@ The ``[tool]`` table is included as it has been found to be very useful for ``pyproject.toml`` files. Providing similar flexibility to this PEP is done in hopes that similar benefits will materialize. -But some people are concerned that such a table will be too enticing to tools -and will lead to files that are tool-specific and unusable by other tools. This -could cause issues for tools trying to do installation, auditing, etc. as they -would not know what details in the ``[tool]`` table are somehow critical. - - -Restrict the ``[tool]`` table to data that is disposable -======================================================== - -The ``[tool]`` table is included as it has been found to be very useful for -``pyproject.toml`` files. Providing similar flexibility to this PEP is done in -hopes that similar benefits will materialize. +But some people have been concerned that such a table will be too enticing to +tools sand will lead to files that are tool-specific and unusable by other +tools. This could cause issues for tools trying to do installation, auditing, +etc. as they would not know what details in the ``[tool]`` table are somehow +critical. -But some people are concerned that such a table will be too enticing to tools -and will lead to files that are tool-specific and unusable by other tools. As -such, some have suggested only recording data that could be tossed at any time -and have no negative effect (e.g., caching info). That would allow another tool -to update a file and delete the ``[tool]`` tables without fear of impacting the -file adversely. +As a compromise, this PEP specifies that the details recorded in ``[tool]`` must +be disposable and not affect installation of packages. List the requirement inputs for the file @@ -1184,47 +1139,67 @@ But it may help in auditing and any recreation of the file if the original requirements were somehow recorded. This could be a single string or an array of strings if multiple requirements were used with the file. +In the end it was deemed too complicated to try and capture the inputs +that a tool used to construct the lock file in a generic fashion. + -------- Auditing -------- -Recording dependencies -====================== - -Recording the dependencies of a package is not necessary to install it. As such, -it has been left out of the PEP as it can be included via ``[tool]``. - -But knowing how costly a package is to include may be beneficial to users when -determining why a certain package was included in the lock file. A flexible -approach could be used to record the dependencies, e.g., as much detail as to -differentiate from any other entry for the same package in the file (inspired -by uv_). - - Recording dependents ==================== -Recording the dependencies of a package is not necessary to install it. As such, +Recording the dependents of a package is not necessary to install it. As such, it has been left out of the PEP as it can be included via ``[tool]``. But knowing how critical a package is to other packages may be beneficial. This information is included by `pip-tools`_ , so there's prior art in including it. -A flexible approach could be used to record the dependencies, e.g., as much +A flexible approach could be used to record the dependents, e.g. as much detail as to differentiate from any other entry for the same package in the file (inspired by uv_). +In the end, though, it was decided that recording the dependencies is a better +thing to record (if at all). -Including index-hosted attestations -=================================== -:ref:`packaging:index-hosted-attestations` specifies attestation details for -files uploaded to a package index like PyPI. Including some of those details may -help detect issues with packaging when auditing the file (e.g., the publisher -suddenly changing).The key reason this isn't included in the PEP is because the -specification is entirely focused on JSON. In order to bring it to this PEP -either how to translate JSON to TOML would need to be specified, embed the -JSON payload as a string, or re-specify some or all of the attestation spec. +=========== +Open Issues +=========== + +---------------- +Self-Referential +---------------- + +Record what tool created the lock file +====================================== + +Right now the PEP does not record any details about the tool that created a +file. That's out of simplicity reasons only. Which tool is used may be +implicitly recorded by a ``[tool]`` table. + +But one could record various amounts of details about the tool to help recreate +the file. Key details like tool name, the installation requirements when the +tool is hosted on PyPI (encoded as :ref:`packaging:dependency-specifiers`), and +the command used to create the file would allow another tool to re-run the tool. +It would also help discover what tool was used. + + +-------- +Auditing +-------- + +Recording dependencies +====================== + +Recording the dependencies of a package is not necessary to install it. As such, +it has been left out of the PEP as it can be included via ``[tool]``. + +But knowing how costly a package is to include may be beneficial to users when +determining why a certain package was included in the lock file. A flexible +approach could be used to record the dependencies, e.g., as much detail as to +differentiate from any other entry for the same package in the file (inspired +by uv_). ------------------------- From 8dd56bc4cf84d237867cd4ee77102b9dbcca5440 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 16:51:45 -0800 Subject: [PATCH 04/12] Record the name of the tool that created the lock file --- peps/pep-0751.rst | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 2aab38adf73..584741cd2b6 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -147,6 +147,20 @@ order. Usage of inline tables SHOULD also be kept consistent. - If an tool doesn't support a major version, it MUST raise an error. +``created-by`` +============== + +- **Type**: string +- **Required?**: yes +- **Inspiration**: Tools with their name in their lock file name +- Records the name of the tool used to create the lock file. +- Tools MAY use the ``[tool]`` table to record enough details that it can be + inferred what inputs were used to create the lock file. +- Tools SHOULD record the normalized name of the tool if it is available as a + Python package to facilitate finding the tool. + + + ``environments`` ================ @@ -1167,24 +1181,6 @@ thing to record (if at all). Open Issues =========== ----------------- -Self-Referential ----------------- - -Record what tool created the lock file -====================================== - -Right now the PEP does not record any details about the tool that created a -file. That's out of simplicity reasons only. Which tool is used may be -implicitly recorded by a ``[tool]`` table. - -But one could record various amounts of details about the tool to help recreate -the file. Key details like tool name, the installation requirements when the -tool is hosted on PyPI (encoded as :ref:`packaging:dependency-specifiers`), and -the command used to create the file would allow another tool to re-run the tool. -It would also help discover what tool was used. - - -------- Auditing -------- From 336e665f89660c0163d3f2cac54e404cdb0e0482 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 16:54:02 -0800 Subject: [PATCH 05/12] Clarify that not using `url` but some other mechanism is optional --- peps/pep-0751.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 584741cd2b6..dfc5cfcb23c 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -717,7 +717,7 @@ a suggestion): #. Get the file: - If `path` is set, use it. - - If `url` is set, try to use it, optionally falling back to using + - If `url` is set, try to use it; optionally tools MAY use `packages.index` or some tool-specific mechanism to download the selected wheel file (tools MUST NOT try to change what wheel file to download based on what's available; what file to install should be @@ -731,7 +731,7 @@ a suggestion): #. Get the file. - If `path` is set, use it. - - If `url` is set, try to use it, optionally falling back to using + - If `url` is set, try to use it; tools MAY use `packages.index` or some tool-specific mechanism to download the file. #. Validate the file size and hash. From c3c996363c6b477054bbc42a2296a5f75c315489 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 17:42:05 -0800 Subject: [PATCH 06/12] Add dependencies and drop recording extras and dependency groups --- peps/pep-0751.rst | 141 +++++++--------------------------------------- 1 file changed, 21 insertions(+), 120 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index dfc5cfcb23c..832459bc85c 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -129,8 +129,9 @@ The format of the file is TOML_. Tools SHOULD write their lock files in a consistent way to minimize noise in diff output. Keys in tables -- including the top-level table -- SHOULD be -recorded in a consistent order. As well, tools SHOULD sort arrays in consistent -order. Usage of inline tables SHOULD also be kept consistent. +recorded in a consistent order (if inspiration is desired, this PEP has tried to +write down keys in a logical order). As well, tools SHOULD sort arrays in +consistent order. Usage of inline tables SHOULD also be kept consistent. ``lock-version`` @@ -245,6 +246,24 @@ order. Usage of inline tables SHOULD also be kept consistent. for the package. +``[[packages.dependencies]]`` +----------------------------- + + **Type**: array of tables +- **Required?**: no +- **Inspiration**: PDM_, Poetry_, uv_ +- Records the other entries in ``[[packages]]`` which are direct dependencies of + this package. +- Each entry is a table which contains the minimum information required to tell + which other package entry it corresponds to where doing a key-by-key + comparison would find the appropriate package with no ambiguity (e.g. if there + are two entries for the ``spam`` package, then you can include the version + number like ``{name = "spam", version = "1.0.0"}``, or by source like + ``{name = "spam", vcs = { url = "..."}``). + + + + .. Installation ``packages.direct`` @@ -1177,124 +1196,6 @@ In the end, though, it was decided that recording the dependencies is a better thing to record (if at all). -=========== -Open Issues -=========== - --------- -Auditing --------- - -Recording dependencies -====================== - -Recording the dependencies of a package is not necessary to install it. As such, -it has been left out of the PEP as it can be included via ``[tool]``. - -But knowing how costly a package is to include may be beneficial to users when -determining why a certain package was included in the lock file. A flexible -approach could be used to record the dependencies, e.g., as much detail as to -differentiate from any other entry for the same package in the file (inspired -by uv_). - - -------------------------- -Expanding the feature set -------------------------- - -This PEP is currently oriented towards standardizing on something that can -replace a ``requirements.txt`` file that acts as a lock file (e.g., what -`pip-tools`_ produces). But with an expansion of features, the file format may be -able to replace the internal lock file format used by tools like PDM_ and -Poetry_, especially when a ``pyproject.toml`` file is viewed as the ideal input -for creating a lock file. - - -Record the requirements for extras of a package -=============================================== - -A project with a ``pyproject.toml`` file may define some extras which add -dependencies to install. In the simple case this would just be a matter of -marking an entry in ``[[packages]]`` as only applying when a specific extra is -requested. Unfortunately the simple case doesn't cover all cases. - -Consider the following example where the latest release of NumPy is 2.2.1 and -the last NumPy 1 release was 1.26.4: - -.. code-block:: TOML - - [project.optional-dependencies] - extra-1 = ["numpy"] - extra-2 = ["numpy~=1.0"] - -Individually those extras cause no issue. But extra-2 does "overpower" -extra-1 when it comes to what version of NumPy to install. That leads to the -issue of needing a way to record the fact that if extra-1 is requested on its -own then NumPy 2.2.1 should be recorded in the lock file, but if extra-2 is -specified (either on its own or in conjunction with extra-1), then NumPy 1.26.4 -should be recorded. - -There are two possible solutions to this. - - -A single version across all extras in a single lock file --------------------------------------------------------- - -One solution to the problem is to do what uv_ does and lock to a single version -for a package no matter what. That would mean any use of NumPy which could occur -in any scenario would use NumPy 1.26.4. Some argue that leads to consistency as -you won't be wondering what version of NumPy you will end up with based on what -extras you select. - -But this does mean that if you want the version of NumPy to vary across extras -you will need to create separate lock files for the various NumPy versions you -want. While not technically an issue, it is ergonomically a bit annoying when -this is necessary. But it's not known how frequently varying package versions -which depend on which extra(s) are chosen occur, and when they do occur do people -still want the variance or prefer the approach uv_ has taken. - -If this solution were to be taken, then very likely an ``extras`` key would be -added which would list the extras that the entry in ``[[package]]`` should be -used for. This works thanks to extras being additive, and thus only contributing -more packages. - - -Support Boolean logic for extra selection ------------------------------------------ - -Another solution to this problem is specifying the conditions under -which a package version applies. This would mean supporting Boolean logic -to fully express the conditions under which a package applies. - -But historically extras have not been expressed this way. The use of the -``extra`` clause in :ref:`packaging:core-metadata-requires-dist` is always -singular and with a ``==`` operator. This also means the operators on ``extra`` -have not been designed to treat the extras specified as a set, and so an -expression simultaneously using ``==`` and ``!=`` are not well-defined when it -comes to ``extra``. This all means that using -``extra == 'extra-1' and extra != 'extra-2'`` to appropriately express what is -needed for extra-1 to work has not been done before. It would also mean -potentially more use of an ``extra`` key as the default package version may need -to explicitly exclude all extra groups when other groups restrict what package -versions apply. - -For this to work we would either need to expand the use of the ``extra`` clause -so it can be used in ``packages.marker`` or have an ``extras`` key which -expresses a Boolean expression for under which the package should be used. In -both situations the spec around ``extra`` would need to be expanded by this PEP --- or another PEP before this one is accepted -- to lay out how Boolean -expressions would work in this case. - - -Record dependency groups -======================== - -Dependency groups have the same concerns as extras mentioned above along with -lacking any pre-existing clause for use in dependency specifiers. And so -dependency groups have the added issue that to use Boolean expressions would -require defining a new clause type. - - ================ Acknowledgements ================ From 93aff2bb877099f44283a496f99a8f174c5fbca6 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Tue, 4 Feb 2025 17:44:52 -0800 Subject: [PATCH 07/12] Add dependencies and drop recording extras and dependency groups --- peps/pep-0751.rst | 143 ++++++++-------------------------------------- 1 file changed, 23 insertions(+), 120 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index dfc5cfcb23c..44ae4450685 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -129,8 +129,9 @@ The format of the file is TOML_. Tools SHOULD write their lock files in a consistent way to minimize noise in diff output. Keys in tables -- including the top-level table -- SHOULD be -recorded in a consistent order. As well, tools SHOULD sort arrays in consistent -order. Usage of inline tables SHOULD also be kept consistent. +recorded in a consistent order (if inspiration is desired, this PEP has tried to +write down keys in a logical order). As well, tools SHOULD sort arrays in +consistent order. Usage of inline tables SHOULD also be kept consistent. ``lock-version`` @@ -245,6 +246,26 @@ order. Usage of inline tables SHOULD also be kept consistent. for the package. +``[[packages.dependencies]]`` +----------------------------- + +- **Type**: array of tables +- **Required?**: no +- **Inspiration**: PDM_, Poetry_, uv_ +- Records the other entries in ``[[packages]]`` which are direct dependencies of + this package. +- Each entry is a table which contains the minimum information required to tell + which other package entry it corresponds to where doing a key-by-key + comparison would find the appropriate package with no ambiguity (e.g. if there + are two entries for the ``spam`` package, then you can include the version + number like ``{name = "spam", version = "1.0.0"}``, or by source like + ``{name = "spam", vcs = { url = "..."}``). +- Tools MUST NOT use this information when doing installation; it is purely + informational for auditing purposes. + + + + .. Installation ``packages.direct`` @@ -1177,124 +1198,6 @@ In the end, though, it was decided that recording the dependencies is a better thing to record (if at all). -=========== -Open Issues -=========== - --------- -Auditing --------- - -Recording dependencies -====================== - -Recording the dependencies of a package is not necessary to install it. As such, -it has been left out of the PEP as it can be included via ``[tool]``. - -But knowing how costly a package is to include may be beneficial to users when -determining why a certain package was included in the lock file. A flexible -approach could be used to record the dependencies, e.g., as much detail as to -differentiate from any other entry for the same package in the file (inspired -by uv_). - - -------------------------- -Expanding the feature set -------------------------- - -This PEP is currently oriented towards standardizing on something that can -replace a ``requirements.txt`` file that acts as a lock file (e.g., what -`pip-tools`_ produces). But with an expansion of features, the file format may be -able to replace the internal lock file format used by tools like PDM_ and -Poetry_, especially when a ``pyproject.toml`` file is viewed as the ideal input -for creating a lock file. - - -Record the requirements for extras of a package -=============================================== - -A project with a ``pyproject.toml`` file may define some extras which add -dependencies to install. In the simple case this would just be a matter of -marking an entry in ``[[packages]]`` as only applying when a specific extra is -requested. Unfortunately the simple case doesn't cover all cases. - -Consider the following example where the latest release of NumPy is 2.2.1 and -the last NumPy 1 release was 1.26.4: - -.. code-block:: TOML - - [project.optional-dependencies] - extra-1 = ["numpy"] - extra-2 = ["numpy~=1.0"] - -Individually those extras cause no issue. But extra-2 does "overpower" -extra-1 when it comes to what version of NumPy to install. That leads to the -issue of needing a way to record the fact that if extra-1 is requested on its -own then NumPy 2.2.1 should be recorded in the lock file, but if extra-2 is -specified (either on its own or in conjunction with extra-1), then NumPy 1.26.4 -should be recorded. - -There are two possible solutions to this. - - -A single version across all extras in a single lock file --------------------------------------------------------- - -One solution to the problem is to do what uv_ does and lock to a single version -for a package no matter what. That would mean any use of NumPy which could occur -in any scenario would use NumPy 1.26.4. Some argue that leads to consistency as -you won't be wondering what version of NumPy you will end up with based on what -extras you select. - -But this does mean that if you want the version of NumPy to vary across extras -you will need to create separate lock files for the various NumPy versions you -want. While not technically an issue, it is ergonomically a bit annoying when -this is necessary. But it's not known how frequently varying package versions -which depend on which extra(s) are chosen occur, and when they do occur do people -still want the variance or prefer the approach uv_ has taken. - -If this solution were to be taken, then very likely an ``extras`` key would be -added which would list the extras that the entry in ``[[package]]`` should be -used for. This works thanks to extras being additive, and thus only contributing -more packages. - - -Support Boolean logic for extra selection ------------------------------------------ - -Another solution to this problem is specifying the conditions under -which a package version applies. This would mean supporting Boolean logic -to fully express the conditions under which a package applies. - -But historically extras have not been expressed this way. The use of the -``extra`` clause in :ref:`packaging:core-metadata-requires-dist` is always -singular and with a ``==`` operator. This also means the operators on ``extra`` -have not been designed to treat the extras specified as a set, and so an -expression simultaneously using ``==`` and ``!=`` are not well-defined when it -comes to ``extra``. This all means that using -``extra == 'extra-1' and extra != 'extra-2'`` to appropriately express what is -needed for extra-1 to work has not been done before. It would also mean -potentially more use of an ``extra`` key as the default package version may need -to explicitly exclude all extra groups when other groups restrict what package -versions apply. - -For this to work we would either need to expand the use of the ``extra`` clause -so it can be used in ``packages.marker`` or have an ``extras`` key which -expresses a Boolean expression for under which the package should be used. In -both situations the spec around ``extra`` would need to be expanded by this PEP --- or another PEP before this one is accepted -- to lay out how Boolean -expressions would work in this case. - - -Record dependency groups -======================== - -Dependency groups have the same concerns as extras mentioned above along with -lacking any pre-existing clause for use in dependency specifiers. And so -dependency groups have the added issue that to use Boolean expressions would -require defining a new clause type. - - ================ Acknowledgements ================ From 0cbe881fb1d78ebd10dca68cae6880fec80ae658 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 5 Feb 2025 14:52:47 -0800 Subject: [PATCH 08/12] Apply suggestions from code review Co-authored-by: Carol Willing --- peps/pep-0751.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 27efd9c48ab..b17d392f536 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -1150,7 +1150,7 @@ The ``[tool]`` table is included as it has been found to be very useful for hopes that similar benefits will materialize. But some people have been concerned that such a table will be too enticing to -tools sand will lead to files that are tool-specific and unusable by other +tools and will lead to files that are tool-specific and unusable by other tools. This could cause issues for tools trying to do installation, auditing, etc. as they would not know what details in the ``[tool]`` table are somehow critical. @@ -1193,7 +1193,7 @@ detail as to differentiate from any other entry for the same package in the file (inspired by uv_). In the end, though, it was decided that recording the dependencies is a better -thing to record (if at all). +thing to record. ================ From a256c7262bd839222dda67ffe303a41ef6bd9975 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 5 Feb 2025 14:54:14 -0800 Subject: [PATCH 09/12] Fix a link --- peps/pep-0751.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 27efd9c48ab..567e6806826 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -437,7 +437,7 @@ See ``packages.vcs.path``. - **Inspiration**: uv_, :ref:`packaging:simple-repository-api` - The size of the archive file. - Tools SHOULD provide the file size when reasonably possible (e.g. the file - size is available via the `Content-Length` header from a HEAD_ HTTP request). + size is available via the Content-Length_ header from a HEAD_ HTTP request). ``[packages.archive.hashes]`` From b842364889f2af08860dde466d76a56b3a5fd183 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 5 Feb 2025 19:50:32 -0800 Subject: [PATCH 10/12] Add a rejected idea about extras and dependency groups --- peps/pep-0751.rst | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 796c01f1ba4..561cedffbf9 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -1039,6 +1039,32 @@ decide it wasn't worth trying to do in this PEP upfront. Instead, a future PEP could propose a solution. +Recordinng possible extras and dependency groups +================================================ + +To expand the feature set of this PEP such that tools could use this PEP for +their main lock file format, recording possible extras and dependency groups +that a user may request was considered. The idea was that if you were locking +from a ``pyproject.toml`` file then you would want to lock for all +possibilities; that includes extras and dependency groups. + +To make this work one would need a way to declare when a package applied to an +extra and/or dependency group. Due to the possibility that a combination of +extras and/or dependency groups could change version requirements of a package +(e.g. extra A needed the spam package at any version, but extra B needed the +spam package older than version 2, thus making whether extra B was requested +override what extra A requested), it would require either full Boolean logic +support or only locking to a specific version of a package no matter what extras +and/or dependency groups were requested. The former would at least require +coming up with semantics around a marker for dependency groups, and the latter +would require a separate lock file any time one didn't want a single version +restriction. + +In the end, there wasn't enough interest from tools for using this PEP as +their sole lock file format to warrant doing the work to implement this feature +at this time. + + -------------- Simplification -------------- From ca5e634d0f8355500aa7cad6c663598900a792ee Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 5 Feb 2025 19:50:57 -0800 Subject: [PATCH 11/12] Suggest services look for their own lock file first, then a generic `pylock.toml` --- peps/pep-0751.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 561cedffbf9..3a55035cd98 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -114,6 +114,12 @@ e.g.: if len(filename) > 11 and filename.startswith("pylock.") and filename.endswith(".toml"): name = filename.removeprefix("pylock.").removesuffix(".toml") +The expectation is that services that install lock files automatically will +search for a lock file with the service's name, then fallback to the generic +``pylock.toml`` (e.g. a cloud host service named Spam would first look for +``pylock.spam.toml`` to install, and if that file didn't exist then install from +``pylock.toml``). + The lock file(s) SHOULD be located in the directory as appropriate for the scope of the lock file. Locking against a single ``pyproject.toml``, for instance, would place the ``pylock.toml`` in the same directory. If the lock file covered From fea24c8f5aad969735c0bef4712fbcc564825459 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Wed, 5 Feb 2025 19:57:42 -0800 Subject: [PATCH 12/12] Fix backtick typos --- peps/pep-0751.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index 3a55035cd98..7cfee1050d6 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -741,9 +741,9 @@ a suggestion): lack of source for the project. #. Get the file: - - If `path` is set, use it. - - If `url` is set, try to use it; optionally tools MAY use - `packages.index` or some tool-specific mechanism to download the + - If ``path`` is set, use it. + - If ``url`` is set, try to use it; optionally tools MAY use + ``packages.index`` or some tool-specific mechanism to download the selected wheel file (tools MUST NOT try to change what wheel file to download based on what's available; what file to install should be determined in an offline fashion for reproducibility). @@ -755,9 +755,9 @@ a suggestion): #. Get the file. - - If `path` is set, use it. - - If `url` is set, try to use it; tools MAY use - `packages.index` or some tool-specific mechanism to download the + - If ``path`` is set, use it. + - If ``url`` is set, try to use it; tools MAY use + ``packages.index`` or some tool-specific mechanism to download the file. #. Validate the file size and hash. #. Build the package.