Better documentation for how files are included in built artifacts

[AstraLuma]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

While helping with licensezero/cli#3, I was looking for "how to do package data with poetry" but found the docs were unclear or at least extremely implicit about it.

[finswimmer]

Hello @astronouth7303,

could you please add more details about what you want to add to a package and/or what information you are missing?

fin swimmer

[AstraLuma]

What setuptools calls package data: Non-module files.

The specific discussion was about licensezero.json, but this varies from static web assets to templates to UI definitions.

[thejohnfreeman]

From what I understand, every file in your package directory will be included in the package, unless it is ignored by your .gitignore. If you want to include a file that is ignored by .gitignore, e.g. generated source code in my case, then add it to [tool.poetry].include.

[meshy]

@thejohnfreeman that's not working for me -- if I explicitly include a file that's in .gitignore, I'm seeing this when running poetry build:

[IndexError]
list index out of range

Specifically, the sdist step seems to succeed, but the wheel step fails.

EDIT:

While the sdist step does not appear to throw an error, the built .tar.gz does not include the file as directed.

[thejohnfreeman]

Do you have a link to the failing example, maybe in a branch on GitHub?

[meshy]

@thejohnfreeman yes! Thanks for the quick reply.

Here's the branch: https://github.com/meshy/django-schema-graph/tree/poetry
And here's the related PR: meshy/django-schema-graph#12

EDIT:

I've noticed that there's a stacktrace leading to an IndexError in the failed CI build on that branch. Perhaps that will help diagnose this issue.

[meshy]

Here's the stacktrace from the failed CI build -- hopefully it'll help:

Traceback (most recent call last):
  File "/.../pip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/.../pip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/.../pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/.../poetry/masonry/api.py", line 39, in prepare_metadata_for_build_wheel
    builder = WheelBuilder(poetry, SystemEnv(Path(sys.prefix)), NullIO())
  File "/.../poetry/masonry/builders/wheel.py", line 44, in __init__
    super(WheelBuilder, self).__init__(poetry, env, io)
  File "/.../poetry/masonry/builders/builder.py", line 64, in __init__
    self._module = Module(
  File "/.../poetry/masonry/utils/module.py", line 68, in __init__
    PackageInclude(
  File "/.../poetry/masonry/utils/package_include.py", line 15, in __init__
    self.check_elements()
  File "/.../poetry/masonry/utils/package_include.py", line 37, in check_elements
    root = self._elements[0]
IndexError: list index out of range

[thejohnfreeman]

Sorry I didn't see this earlier. I could have solved this for you back in February! You need to change some lines in your pyproject.toml:

packages = [{ include = "schema_graph" }]
include = ["schema_graph/static/schema_graph/main.js"]

The include setting under the [tool.poetry] section is what I was talking about when I said [tool.poetry].include (and I had linked to the documentation).

[meshy]

Oh excellent! -- I didn't quite get that the first time round. Thank you for clarifying!

[brianbruggeman]

@thejohnfreeman

How do I include a full set of sub folders like botocore?

[thejohnfreeman]

@brianbruggeman include accepts globs (example).

[hyliu1989]

@thejohnfreeman It seems like the include is using relative paths relative to the root directory of the pyproject.toml. Could you give or add an example in the include and exclude section for what the relative path is based?

For example,

dummy_folder/
    pyproject.toml
    CHANGE.log
    my_package/
        __init__.py
        my_data.csv

You would specify include = ['my_package/my_data.csv'] in pyproject.toml. However, this is not compatible with setup.py and would poetry fail when the user specifies include = ['CHANGE.log'] because only the things in my_package/ will be installed to site-packages?