Machine-readable build report

[layday]

Source distribution and wheel filenames are variable; they encode a variety of information (e.g. platform, Python version, distribution version) which might vary between build invocations. We might want to think about offering a way to retrieve these from the CLI (e.g. through a new option which would create a manifest) for scripts and automation tools to refer to which would provide a minor convenience over globbing.

[FFY00]

IMO this is out of scope. Maybe if this information wasn't in the file name I would agree on having a manifest with extra information.

I think it is fairly simple to write a Python module that just parses the wheel name and output json or something like that.

import argparse
import json
import os.path
import re

from typing import Any, Dict, Optional


_WHEEL_NAME_REGEX = re.compile(
    r'(?P<distribution>.+)-(?P<version>.+)'
    r'(-(?P<build_tag>.+))?-(?P<python_tag>.+)'
    r'-(?P<abi_tag>.+)-(?P<platform_tag>.+).whl'
)


def parse(name: str) -> Optional[Dict[str, str]]:
    if m := re.match(_WHEEL_NAME_REGEX, name):
        return m.groupdict()
    return None


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        'file',
        type=str,
        help='wheel file',
    )
    args = parser.parse_args()

    if info := parse(os.path.basename(args.file)):
        print(json.dumps(info, indent=4, sort_keys=True))
    else:
        print('Inavlid wheel name, see https://www.python.org/dev/peps/pep-0427/#file-name-convention')

python -m wheel2json ~/Downloads/packaging-20.7-py2.py3-none-any.whl
{
    "abi_tag": "none",
    "build_tag": null,
    "distribution": "packaging",
    "platform_tag": "any",
    "python_tag": "py2.py3",
    "version": "20.7"
}

[FFY00]

Perhaps something for https://github.com/pypa/wheel? python -m wheel info ~/Downloads/packaging-20.7-py2.py3-none-any.whl?

[layday]

I think you're misunderstanding - I don't want to parse the wheel filename into its constituent tags. I want build simply to return the filename of the distribution file the backend has produced in a machine-readable format. It could then be piped into a hypothetical installer or another tool which might expect an sdist or a wheel. For instance, if we were to produce a JSON file:

$ python -m build --write-manifest-to manifest.json
$ python -m install $(jq .wheel manifest.json)

[FFY00]

Hum, I still am not convinced. I would like to keep the CLI fairly simple. As you said

would provide a minor convenience over globbing

Unless there is a reasonable use-case this would either block or make significantly harder, I am -1.

[gaborbernat]

I think this could be achieved if we print the generated files on stdout and forward other output to sys.stderr:

twine upload $(python -m build 2>/dev/null)

Though, in general, I'm tempted not to support this. I can see a valid use case for this in the sense that currently if you want to build a package and then upload it, you need something like:

rm dist -r && python -m build . && twine upload dist/*

Because otherwise, dist might contain previous builds you don't want to upload via a simple globbing. I'm solid -1 on the JSON part, but I'm -0.5 on the stderr/stdout part... 🤔

[FFY00]

Because otherwise, dist might contain previous builds you don't want to upload via a simple globbing.

But you can simply choose other dist folder. If this was not possible I would agree with this feature, but you can... I think it is trivial to output to another folder if you need to separate things, and then move the files afterward if you want to have everything in the same folder at the end.

[gaborbernat]

How do you select a folder name that's guaranteed to be empty/not existing without invoking an rm first?

[kpfleming]

As I've been working on a project with @gaborbernat I've run into this exact problem as well. The only reason that this works properly today for twine is because twine apparently does its own glob expansion.

In a CI job I'm running python -m build --sdist --wheel with no prior knowledge of what the generated file names will be, and then I want to do an install-test of those packages. python -m pip install foo/* does not do glob expansion, so it is necessary to either know the filenames of the sdist/wheel, or to use external glob expansion to get their names.

As best I can tell PEP 517 requires the build backend to return the basename of the thing it built, and ProjectBuilder in this tool returns the full path to the thing that got built. The requested information is already available, so would a change to emit it on request be a significant effort?

[uranusjr]

FWIW if you know the package name in advance, you can do pip install --find-links foo <project> instead.

[gaborbernat]

Calling pip just to find the names IMHO is the way too heavyweight a solution.

[uranusjr]

I was responding to kpfleming’s comment, which was talking about actually installing the built wheels. If you know the project names in advance, you can pass those to pip to install the packages (instead of using path, which needs either glob expansion or knowing the dynamically-generated file names).

I made no mention of using pip just to find out the names. pip cannot even do that (without you manually parsing its debug output).

[FFY00]

Okay, I see the need, though I don't think adding a new option for the might be the best solution. python -m build is a command for users, it is designed to be fairly simple and to be as intuitive as we can. Building from an automated script, without user interaction, is a different use-case. I think adding a new option here would increase the complexity of the command line, and still be not enough to solve the kind of issues that may arise from this use-case, putting more pressure on us to add more options and further make the CLI more complex. For this reason, I believe this use-case should be handled by a different command (maybe python -m build.machine?).
The idea would be that this new command would be able to output the build information that may be necessary by automated tooling. Usually, I'd say to just use the Python module to write your customized payload, but that could be annoying, and given how common this use-case is, having a ready to use suitable interface would make sense. Having this as a separate command also opens the possibility to later split it into a different package if it becomes increasingly complex, or starts needing external dependencies.
How I'd propose this command to behave is to simply output json based on defined json schema. It would then have the option to only output a specific field instead.

So, the usage in this case would be something like:

$ python -m build.machine --output-field build.artifacts
dist/my_package-1.0.0.tar.gz
dist/my_package-1.0.0-py3-none-any.whl

TLDR: Keep python -m build a simple user-focused command and introduce a new command for use in automated scripts and that could appropriately address the requirements of that more complex use case.

[gaborbernat]

TLDR: Keep python -m build a simple user-focused command and introduce a new command for use in automated scripts and that could appropriately address the requirements of that more complex use case.

I'm personally -1 on this proposal. Would confuse more than help to have to maintain and use two separate entry-points depending on your use case. But considering twine accepts glob expressions I'm personally not too fussed about this at the moment, so I have no strong feelings of a solution, and I feel introducing two entry-points is more confusing/pricey than its benefit...

PS. Your proposal is also against UNIX design philosophies, I haven't this duality in other tools, for example, there is not an ls and an ls.machine my 2c.

[layday]

Perhaps this is something that we could roll into #192 - if the output format were to be customisable - say, if build could grow a provisional --output-format=(human|json) option, and all non-build output redirected to stderr as suggested, that'd probably meet users' needs. Imagine:

$ python -m build -w 2>&-
Built foo.whl
$ python -m build -w --output-format=json 2>&-
{"type": "build_success", "path": "foo.whl"}

This would mean we've got cook up some kind of JSON schema and we'd need to think about whether this would be more generally useful - would people care about other type messages being given in a machine-readable format?