43

I'm trying to migrate a setuptools-based project from the legacy setup.py towards modern pyproject.toml configuration.

At the same time I want to keep well established workflows based on pip-compile, i.e., a requirements.in that gets compiled to a requirements.txt (for end-user / non-library projects of course). This has important benefits as a result of the full transparency:

  • 100% reproducible installs due to pinning the full transitive closure of dependencies.
  • better understanding of dependency conflicts in the transitive closure of dependencies.

For this reason I don't want to maintain the dependencies directly inside the pyproject.toml via a dependencies = [] list, but rather externally in the pip-compiled managed requirements.txt.

This makes me wonder: Is there a way to reference a requirements.txt file in the pyproject.toml configuration, without having to fallback to a setup.py script?

sinoroc
  • 18,409
  • 2
  • 39
  • 70
bluenote10
  • 23,414
  • 14
  • 122
  • 178
  • 1
    This is a good question, but a bit upside-down in my opinion... We now have [PEP 621](https://peps.python.org/pep-0621/) that standardizes how (among other things) dependencies should be specified in `pyproject.toml`. So the question you should ask yourself should be: "how can I get `pip-compile` (and other packaging tools) to use the list of dependencies from the `pyproject.toml` as input". Indeed the standardized `pyproject.toml` should be the source of truth. – sinoroc Nov 09 '22 at 09:20
  • 3
    @sinoroc: That's mostly true for libraries, but for non-library / end-user / delivery projects it is good practice to fully pin the transitive closure, i.e., the dependencies referenced in the `pyproject.toml` should not be equivalent to the typical `requirements.in` but rather the fully compiled output. Of course, you can copy/paste the compiled output of pip-compile back into the `pyprojects.toml` but that makes an automated "bump dependency" process awkward. In such cases it is easier to separate the "machine generated" parts from the otherwise human-maintained `pyproject.toml` content. – bluenote10 Nov 09 '22 at 16:12
  • I am not sure I understand... You want to pin the dependencies in the application's project metadata? – sinoroc Nov 09 '22 at 19:08

2 Answers2

57

Use dynamic metadata:

[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}

Note that this will use a requirements.txt-like syntax; each line must conform to PEP 508, so flags like -r, -c, and -e are not supported inside this requirements.txt.

Additionally:

If you are using an old version of setuptools, you might need to ensure that all files referenced by the file directive are included in the sdist (you can do that via MANIFEST.in or using plugins such as setuptools-scm, please have a look on [sic] Controlling files in the distribution for more information).

Changed in version 66.1.0: Newer versions of setuptools will automatically add these files to the sdist.

If you want to use optional-dependencies, say, with a requirements-dev.txt, you will need to put an extra group, as follows (credit to Billbottom):

[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
optional-dependencies = {dev = { file = ["requirements-dev.txt"] }}
pigrammer
  • 2,603
  • 1
  • 11
  • 24
  • 1
    Does not work for me. It does not cause a failure during build or installation but it does not install any requirements either! And to quote from your linked page: _Currently the following fields can be listed as dynamic: version, classifiers, description, entry-points, scripts, gui-scripts and readme._ Not a word about `dependencies`. – dennis Jan 24 '23 at 07:13
  • @dennis If you scroll down on that page, you'll see a table that lists "dependencies" as being in beta for dynamic metadata. What version of setuptools do you have? – pigrammer Jan 24 '23 at 11:27
  • @dennis Also note that "`pip`-specify syntaxes, e.g. `-c/-r/-e` flags, are not supported". – pigrammer Jan 24 '23 at 11:29
  • Any idea want to put for content-type? e.g. `readme = {file = ["README.md"], content-type = "text/markdown"}` – Ray Bell Feb 15 '23 at 16:33
  • 1
    @RayBell You don't need to set `content-type`. That's only for the readme. – pigrammer Feb 16 '23 at 00:57
  • `python -m build` did not install dependencies from the `requirements` file. But `pip install -e .` did for me. – winterlight Feb 18 '23 at 20:07
  • @winterlight Sorry about that; You might want to check your setup, and if it looks right, you might want to file a bug report about that. Then again, it's possible that it isn't required to support dynamic metadata for dependencies, as it's in beta. – pigrammer Feb 18 '23 at 21:52
  • @pigrammer I think it is expected as the `build` documentation https://pypa-build.readthedocs.io/en/latest/ says "It is a simple build tool and does not perform any dependency management." – winterlight Feb 19 '23 at 06:10
  • It builds but have the feeling that it would not install anything since the requirements.txt is not included in any of the built of the 2 files `.wheel` nor `.tar.gz` – ramin Jul 27 '23 at 15:08
  • @ramin Can you try installing it and seeing if it works? – pigrammer Jul 27 '23 at 15:12
  • my project is still on "testpypi" and it didnt install the requirements even when they were in the pyproject file... I suppose it checks in the same repo for the requirements... I will check if there is a way to change the repo and make a test – ramin Jul 27 '23 at 15:20
  • @ramin See the recent edit, I added some information that might help – pigrammer Jul 27 '23 at 15:21
  • package installed... but no dependencies. I am using setuptools v 68.0.0 – ramin Jul 27 '23 at 15:29
  • This comment is only for the optional dependencies: I'm on `setuptools` 68.0.0, and to get the optional dependencies to work I had to add a group property before listing the file(s). In my case, specifying the `requirements-dev.txt` file and associating it with a `dev` group worked with the following property in the `tool.setuptools.dynamic` table: `optional-dependencies = {dev = { file = ["requirements-dev.txt"] }}` – Bilbottom Aug 09 '23 at 14:07
  • @Bilbottom do you mind if I edit that in? – pigrammer Aug 09 '23 at 21:01
  • @pigrammer feel free to! :) – Bilbottom Aug 10 '23 at 08:30
0

pip-tools can read the dependencies from pyproject.toml, just like it would from a requirements.in. That's not the same as referencing the compiled dependencies in requirements.txt, but it is one step closer: you don't need requirements.in and you can distribute requirements.txt.

Like @sinoroc, I think it may be risky to "deliver" requirements with only pinned versions. It's sufficient that one similarly set up project is set up along with your project, and you're likely to get clashes. Of course, you may be able to exclude that, or perhaps even want to exploit that nothing else is used along with your project. But that's a very specific use case.

user1010997
  • 523
  • 5
  • 13