Proposal: Improved integration with setuptools

.github/workflows/main.yml

@@ -15,7 +15,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python_version: ["3.6", "3.7", "3.8"]
+        python_version: ["3.6", "3.9"]
         os: [ubuntu-latest, macos-latest, windows-latest]
 
     steps:
@@ -29,11 +29,16 @@ jobs:
         architecture: 'x64'
 
     - name: Install dependencies
-      run: pip install .[test]
+      run: |
+        python -m pip install -U pip codecov
+        pip install -e .[test]
 
     - name: Run test
       run: |
         python setup.py --version
-        python setup.py build_py
-        python setup.py sdist
-        python -m pytest
+        python -m build .
+        pytest -vv --cov jupyter_packaging --cov-branch --cov-report term-missing:skip-covered
+
+    - name: Coverage
+      run: |
+        codecov

README.md

@@ -1,81 +1,124 @@
-# jupyter-packaging
+# Jupyter Packaging
 
-Tools to help build and install Jupyter Python packages
+Tools to help build and install Jupyter Python packages that require a pre-build step that may include JavaScript build steps.
 
 ## Install
 
 `pip install jupyter-packaging`
 
 ## Usage
 
-There are two ways to use `jupyter-packaging` in another package.
+There are three ways to use `jupyter-packaging` in another package.
 
-The first to use a `pyproject.toml` file as outlined in [pep-518](https://www.python.org/dev/peps/pep-0518/).
+### As a Build Requirement
+
+Use a `pyproject.toml` file as outlined in [pep-518](https://www.python.org/dev/peps/pep-0518/).
 An example:
 
-```
+```toml
 [build-system]
-requires = ["jupyter_packaging~=0.6.0", "jupyterlab~=2.0", "setuptools>=40.8.0", "wheel"]
+requires = ["jupyter_packaging~=0.8.0"]
 build-backend = "setuptools.build_meta"
 ```
 
-The second method is to vendor `setupbase.py` locally alongside `setup.py` and import the helpers from `setupbase`.
+Below is an example `setup.py` using the above config.
+It assumes the rest of your metadata is in [`setup.cfg`](https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html).
+We wrap the import in a try/catch to allow the file to be run without `jupyter_packaging`
+so that `python setup.py` can be run directly when not building.
+
+```py
+from setuptools import setup
+
+try:
+    from jupyter_packaging import wrap_installers, npm_builder
+    builder = npm_builder()
+    cmdclass = wrap_installers(pre_develop=builder, pre_dist=builder)
+except ImportError:
+    cmdclass = {}
+
+setup(cmdclass=cmdclass))
+```
+
+### As a Build Backend
 
-Below is an example `setup.py` that uses the `pyproject.toml` approach:
+Use the `jupyter_packaging` build backend.
+The pre-build command is specified as metadata in `pyproject.toml`:
+
+```toml
+[build-system]
+requires = ["jupyter_packaging~=0.8.0"]
+build-backend = "jupyter_packaging.build_api"
+
+[tool.jupyter-packaging.builder]
+func = "jupyter_packaging.npm_builder"
+
+[tool.jupyter-packaging.build-args]
+build_cmd = "build:src"
+```
+
+The corresponding `setup.py` would be greatly simplified:
 
 ```py
 from setuptools import setup
-from jupyter_packaging import create_cmdclass, install_npm
-
-
-cmdclass = create_cmdclass(['js'])
-cmdclass['js'] = install_npm()
-
-setup_args = dict(
-    name             = 'PROJECT_NAME',
-    description      = 'PROJECT_DESCRIPTION',
-    long_description = 'PROJECT_LONG_DESCRIPTION',
-    version          = 'PROJECT_VERSION',
-    author           = 'Jupyter Development Team',
-    author_email     = '[email protected]',
-    url              = 'http://jupyter.org',
-    license          = 'BSD',
-    platforms        = "Linux, Mac OS X, Windows",
-    keywords         = ['ipython', 'jupyter'],
-    classifiers      = [
-        'Intended Audience :: Developers',
-        'Intended Audience :: System Administrators',
-        'Intended Audience :: Science/Research',
-        'License :: OSI Approved :: BSD License',
-        'Programming Language :: Python',
-        'Programming Language :: Python :: 2.7',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.3',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
-    ],
-    cmdclass         = cmdclass,
-    install_requires = [
-        'notebook>=4.3.0',
-    ]
-)
-
-if __name__ == '__main__':
-    setup(**setup_args)
+setup()
 ```
 
-## Development Install
+The `tool.jupyter-packaging.builder` section expects a `func` value that points to an importable
+module and a function with dot separators.  If not given, no pre-build function will run.
+
+The optional `tool.jupyter-packaging.build-args` sections accepts a dict of keyword arguments to
+give to the pre-build command.
+
+The build backend does not handle the `develop` command (`pip install -e .`).
+If desired, you can wrap just that command:
 
+```py
+import setuptools
+
+try:
+    from jupyter_packaging import wrap_installers, npm_builder
+    builder = npm_builder(build_cmd="build:dev")
+    cmdclass = wrap_installers(pre_develop=builder)
+except ImportError:
+    cmdclass = {}
+
+setup(cmdclass=cmdclass))
 ```
+
+### As a Vendored File
+
+Vendor `setupbase.py` locally alongside `setup.py` and import the module directly.
+
+```py
+import setuptools
+from setupbase import wrap_installers, npm_builder
+builder = npm_builder
+cmdclass = wrap_installers(post_develop=builder, pre_dist=builder)
+setup(cmdclass=cmdclass)
+```
+
+## Usage Notes
+
+- We recommend using `include_package_data=True` and `MANIFEST.in` to control the assets included in the [package](https://setuptools.readthedocs.io/en/latest/userguide/datafiles.html).
+- Tools like [`check-manifest`](https://github.com/mgedmin/check-manifest) or [`manifix`](https://github.com/vidartf/manifix) can be used to ensure the desired assets are included.
+- Simple uses of `data_files` can be handled in `setup.cfg` or in `setup.py`.  If recursive directories are needed use `get_data_files()` from this package.
+- Unfortunately `data_files` are not supported in `develop` mode (a limitation of `setuptools`).  You can work around it by doing a full install (`pip install .`) before the develop install (`pip install -e .`), or by adding a script to push the data files to `sys.base_prefix`.
+
+
+## Development Install
+
+```bash
 git clone https://github.com/jupyter/jupyter-packaging.git
 cd jupyter-packaging
 pip install -e .
 ```
 
 You can test changes locally by creating a `pyproject.toml` with the following, replacing the local path to the git checkout:
 
-```
+```toml
 [build-system]
-requires = ["jupyter_packaging@file://<path-to-git-checkout>", "setuptools>=40.8.0", "wheel"]
+requires = ["jupyter_packaging@file://<path-to-git-checkout>"]
 build-backend = "setuptools.build_meta"
-```
+```
+
+Note: you need to run `pip cache remove jupyter_packaging` any time changes are made to prevent `pip` from using a cached version of the source.

jupyter_packaging/__main__.py

@@ -36,5 +36,5 @@ def main(args=None):
     shutil.copy(source, destination)
 
 
-if __name__ == '__main__':
+if __name__ == '__main__':  # pragma: no cover
     main()

jupyter_packaging/build_api.py

@@ -0,0 +1,65 @@
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+import functools
+import importlib
+from pathlib import Path
+import os
+import sys
+
+from setuptools.build_meta import (
+    get_requires_for_build_wheel,
+    get_requires_for_build_sdist,
+    prepare_metadata_for_build_wheel,
+    build_sdist as orig_build_sdist,
+    build_wheel as orig_build_wheel
+)
+import tomlkit
+
+
+def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
+    """Build a wheel with an optional pre-build step."""
+    builder = _get_build_func()
+    if builder:
+        builder()
+    return orig_build_wheel(wheel_directory, config_settings=config_settings, metadata_directory=metadata_directory)
+
+
+def build_sdist(sdist_directory, config_settings=None):
+    """Build an sdist with an optional pre-build step."""
+    builder = _get_build_func()
+    if builder:
+        builder()
+    return orig_build_sdist(sdist_directory, config_settings=config_settings)
+
+
+def _get_build_func():
+    pyproject = Path('pyproject.toml')
+    if not pyproject.exists():
+        return
+    data = tomlkit.loads(pyproject.read_text(encoding='utf-8'))
+    if 'tool' not in data:
+        return
+    if 'jupyter-packaging' not in data['tool']:
+        return
+    if 'builder' not in data['tool']['jupyter-packaging']:
+        return
+    section = data['tool']['jupyter-packaging']
+    if 'func' not in section['builder']:
+        raise ValueError('Missing `func` specifier for builder')
+
+    func_data = section['builder']['func']
+    mod_name, _, func_name = func_data.rpartition('.')
+
+    # If the module fails to import, try importing as a local script
+    try:
+        mod = importlib.import_module(mod_name)
+    except ImportError:
+        try:
+            sys.path.insert(0, os.getcwd())
+            mod = importlib.import_module(mod_name)
+        finally:
+            sys.path.pop(0)
+
+    func = getattr(mod, func_name)
+    kwargs = section.get('build-args', {})
+    return functools.partial(func, **kwargs)