Skip to content

npm should not be required to install pytest-html from pypi #747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
thebaptiste opened this issue Oct 12, 2023 · 19 comments · Fixed by #758
Closed

npm should not be required to install pytest-html from pypi #747

thebaptiste opened this issue Oct 12, 2023 · 19 comments · Fixed by #758

Comments

@thebaptiste
Copy link

thebaptiste commented Oct 12, 2023

Since release 4.0.0rc5, npm is required to install pytest-html from pypi

I understand npm is only used for tests purpose (maybe I'm wrong...) ? If it is the case I think it should not be necessary to have npm pre-installed to install pytest-html

Below the error I have installing pytest-html >= 4.0.0rc5 from pypi (same thing with 4.0.0, 4.0.1 and 4.0.2) :

  Downloading pytest_html-4.0.0rc5.tar.gz (145 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 145.8/145.8 kB 20.8 MB/s eta 0:00:00
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      /bin/sh: npm: command not found
      Traceback (most recent call last):
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/hatchling/build.py", line 56, in build_wheel
          return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/site-packages/hatchling/builders/plugin/interface.py", line 150, in build
          build_hook.initialize(version, build_data)
        File "/home/baptiste/mfext/src/layers/layer7_python3_devtools/0500_extra_python_packages/tempolayer3830361/tmp/pip-wheel-dl81rpb1/pytest-html_2cea07a17db443aebda5c2ba1b9efeb4/scripts/npm.py", line 8, in initialize
          subprocess.check_output("npm ci", shell=True)
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/baptiste/mfext/build/opt/python3_core/lib/python3.11/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'npm ci' returned non-zero exit status 127.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.
@thebaptiste thebaptiste changed the title npm is required to build pytest-html npm should not be required to install pytest-html from pypi Oct 13, 2023
@BeyondEvil
Copy link
Contributor

hmm... that's weird. Like you say, it shouldn't be required. It's only required when building from source.

It's somehow related to Hatch and/or pyproject.toml - not sure how or why tho.

@BeyondEvil
Copy link
Contributor

pytest-html installs just fine on a system without npm.

jbrwe6@MAC-6CY79W0M dev % mkdir testing-npm
jbrwe6@MAC-6CY79W0M dev % cd testing-npm
jbrwe6@MAC-6CY79W0M testing-npm % python3 -m venv .venv
jbrwe6@MAC-6CY79W0M testing-npm % source .venv/bin/activate
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % pip freeze
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % npm
zsh: command not found: npm
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % pip install pytest-html
Collecting pytest-html
  Downloading pytest_html-4.0.2-py3-none-any.whl (23 kB)
*snip*
Installing collected packages: tomli, pluggy, packaging, iniconfig, exceptiongroup, pytest, MarkupSafe, pytest-metadata, jinja2, pytest-html
Successfully installed MarkupSafe-2.1.3 exceptiongroup-1.1.3 iniconfig-2.0.0 jinja2-3.1.2 packaging-23.2 pluggy-1.3.0 pytest-7.4.3 pytest-html-4.0.2 pytest-metadata-3.0.0 tomli-2.0.1
(.venv) jbrwe6@MAC-6CY79W0M testing-npm % pip freeze
exceptiongroup==1.1.3
iniconfig==2.0.0
Jinja2==3.1.2
MarkupSafe==2.1.3
packaging==23.2
pluggy==1.3.0
pytest==7.4.3
pytest-html==4.0.2
pytest-metadata==3.0.0
tomli==2.0.1

So unless you can provide steps to reproduce, I'm closing the ticket.

@RonnyPfannschmidt
Copy link
Member

It seems like the sdist needs it still, I'd recommend checking if the build results can be included and used for the sdist as well to ease things for people that work with broken ci setups or repackage

@BeyondEvil
Copy link
Contributor

BeyondEvil commented Nov 1, 2023

It seems like the sdist needs it still, I'd recommend checking if the build results can be included and used for the sdist as well to ease things for people that work with broken ci setups or repackage

Aha, so something (likely in the pyproject.toml) is missing specifically for the sdist?

Into the Hatch documentation I go then...

@BeyondEvil
Copy link
Contributor

Hmm... when I build with Hatch, the app.js is included in both sdist and wheel.

However, it seems that the zipped and tarred assets in the Github releases does not.

Something I need to test, is what the result of the way CI (using hynek/build-and-inspect-python-package@v1) builds the dist.

@BeyondEvil
Copy link
Contributor

Building with SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) python -m build --outdir dist also shows app.js as part of the sdist.

I'm out of ideas. Without steps to reproduce, I can't fix this.

@RonnyPfannschmidt
Copy link
Member

$ podman run --rm python pip install --no-binary pytest-html pytest-html
Collecting pytest-html
  Downloading pytest_html-4.0.2.tar.gz (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.1/149.1 kB 2.7 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      /tmp/pip-build-env-4pmiv9_j/overlay/lib/python3.12/site-packages/setuptools_scm/_integration/dump_version.py:65: UserWarning: template='' looks like a error, using default instead
        warnings.warn(f"{template=} looks like a error, using default instead")
      /bin/sh: 1: npm: not found
      Traceback (most recent call last):
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4pmiv9_j/overlay/lib/python3.12/site-packages/hatchling/build.py", line 56, in build_wheel
          return os.path.basename(next(builder.build(wheel_directory, ['standard'])))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4pmiv9_j/overlay/lib/python3.12/site-packages/hatchling/builders/plugin/interface.py", line 150, in build
          build_hook.initialize(version, build_data)
        File "/tmp/pip-install-_hpom3az/pytest-html_facb36f6a4cc48a083187845766b40bf/scripts/npm.py", line 8, in initialize
          subprocess.check_output("npm ci", shell=True)
        File "/usr/local/lib/python3.12/subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.12/subprocess.py", line 571, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command 'npm ci' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.2.1 -> 23.3.1

@RonnyPfannschmidt
Copy link
Member

so the build hook is always used

@ofek is there builtin hatch helper to have certain build steps happen for from source but not for from sdist

@BeyondEvil
Copy link
Contributor

If the response from Ofek is negative, what's the solution?

Remove the build-hook and build it "manually" in CI and make sure app.js is added to wheel and sdist?

@ofek
Copy link

ofek commented Nov 1, 2023

The fix should be pretty simple https://github.com/pytest-dev/pytest-html/blob/master/scripts/npm.py

Just gate those two statements by a check for something that is not shipped in either artifact like /.git/

@BeyondEvil
Copy link
Contributor

The fix should be pretty simple https://github.com/pytest-dev/pytest-html/blob/master/scripts/npm.py

Just gate those two statements by a check for something that is not shipped in either artifact like /.git/

You mean like:

import subprocess
from hatchling.builders.hooks.plugin.interface import BuildHookInterface
from pathlib import Path


class NpmBuildHook(BuildHookInterface):
    def initialize(self, version, build_data):
        app_js = Path("src/pytest_html/resources/app.js")
        if not app_js.exists():
            subprocess.check_output("npm ci", shell=True)
            subprocess.check_output("npm run build", shell=True)

@ofek
Copy link

ofek commented Nov 1, 2023

Yes but just for correctness/to be pedantic even though it's the same path:

app_js = Path(self.root, "src", "pytest_html", "resources", "app.js")

Also I do understand what you are going for there but that would not work for repeat builds like running it twice locally which is why I recommended something that is not shipped at all if possible.

@BeyondEvil
Copy link
Contributor

BeyondEvil commented Nov 1, 2023

Also I do understand what you are going for there but that would not work for repeat builds like running it twice locally which is why I recommended something that is not shipped at all if possible.

Ah, I see what you mean now! If /.git/ is present we can assume we're building from source.

Can there be cases where someones building from source but .git is not present?

import subprocess
from hatchling.builders.hooks.plugin.interface import BuildHookInterface
from pathlib import Path


class NpmBuildHook(BuildHookInterface):
    def initialize(self, version, build_data):
        building_from_source = Path(self.root, ".git").exists()
        if building_from_source:
            subprocess.check_output("npm ci", shell=True)
            subprocess.check_output("npm run build", shell=True)

@ofek
Copy link

ofek commented Nov 1, 2023

Oh yeah definitely you're right, forget what I said and continue with the path that you chose. A situation would be an archive from a GitHub release. You build artifacts in CI the proper way and it would only run once so it's fine.

@ofek
Copy link

ofek commented Nov 1, 2023

Upon further reflection perhaps you can take care of that scenario: if that directory exists at the same time as the file existing then assume a local development setup and always trigger a build.

@BeyondEvil
Copy link
Contributor

Upon further reflection perhaps you can take care of that scenario: if that directory exists at the same time as the file existing then assume a local development setup and always trigger a build.

Now I'm confused. 😅

@ofek
Copy link

ofek commented Nov 1, 2023

My apologies! Your proposal is perfect I was just trying to find a way to have local development builds always trigger JavaScript builds and for that I was thinking if /.git exists and the file you were talking about exists then you can always call NPM. Basically just another conditional logic branch that ensures rebuilding happens.

@BeyondEvil
Copy link
Contributor

BeyondEvil commented Nov 1, 2023

My apologies! Your proposal is perfect I was just trying to find a way to have local development builds always trigger JavaScript builds and for that I was thinking if /.git exists and the file you were talking about exists then you can always call NPM. Basically just another conditional logic branch that ensures rebuilding happens.

Ah.

So I should stick with my initial solution? 😅 🙇

app_js = Path(self.root, "src", "pytest_html", "resources", "app.js")
        if not app_js.exists():

@thebaptiste
Copy link
Author

thebaptiste commented Nov 14, 2023

Many thanks for the fix !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants