Ignore hashes for version control repositories

[a666]

What's the problem this feature will solve?

Right now we are having the following situation.

Having a requirements.in like:

# requirements.in
django
git+ssh://git@private.server/private_app.git@0.3#egg=private_app

and doing:

$ pip-compile --generate-hashes -o requirements.txt requirements.in && pip-sync requirements.txt

gives us the error:

ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    private_app from git+ssh://****@private.server/private_app.git@0.3#egg=private_app 
    (from -r /tmp/tmpii51x619 (line 217))

Replacing our repo with -e git+ssh://git@private.server/private_app.git@0.3#egg=private_app changes the error to

ERROR: The editable requirement posgrado_catalogs from 
git+ssh://****@private.server/private_app.git@0.3#egg=private_app
(from -r /tmp/tmp317kk_qv (line 172)) cannot be installed when requiring hashes, 
because there is no single file to hash.

Describe the solution you'd like

It would be ideal for version control repositories to skip hashing (since access and integrity is handled differently) while keeping it for every other package.

Alternative Solutions

pipenv (ugh) seems to do it.

[atugushev]

Hello @a666,

Thanks for the issue! I believe this should be implemented on pip side. See related issues:

• editable cannot be installed when requiring hashes pypa/pip#4995
• Validate VCS urls in hash-checking mode using their commit hashes pypa/pip#6469

However, there are possible solutions:

• using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip.
• split requirements files for hashable and non-hashable packages.

[a666]

Thank you @atugushev for the quick answer.

I forgot to precise that we are using your first solution for the time being.

[cjerdonek]

Hi @atugushev, what did you mean by this?

• split requirements files for hashable and non-hashable packages.

Did you mean split the requirements.in file? Or did you mean split the requirements.txt file output by pip-compile (e.g. into hashable and non-hashable requirements)? (The latter could presumably be done by parsing the output, though it's not so elegant.)

By the way, it seems like if #333 were implemented, that would be another possible solution. If one knows certain VCS requirements are included in a requirements.in file, then the names of those packages could also be passed in via #333's --exclude option to exclude those lines from the output.

[atugushev]

Did you mean split the requirements.in file?

Hello @cjerdonek! That's exactly what I mean. Yeah, --exclude seems like a good solution.

[cjerdonek]

Thanks, @atugushev! However, won't splitting the requirements.in file into VCS and non-VCS requirements mean that the dependencies of the VCS requirements (which can be hashable) won't get the benefit of the hashes, because those dependencies would be part of the requirements.txt file corresponding to the VCS requirements.in? Or am I missing part of your suggestion?

[atugushev]

@cjerdonek, yes, you are right.

[cjerdonek]

Okay, thanks a lot for confirming.

[sergeyklay]

Hello,

Not sure if this is the right place, but I'd like to show you my case so that you have more information about the issues that arise. Most of the code has been omitted to show only the gist:

tox.ini

[tox]
minversion = 3.22
envlist = py{37,38,39,310}

[testenv]
extras = testing
deps =
    -rrequirements.txt
commands =
    coverage erase
    coverage run -m pytest {posargs}

requirements.in

django

# ...

Command to compile requirements.txt file

pip-compile --generate-hashes --output-file=requirements.txt requirements.in

requirements.txt

# line line 11
asgiref==3.3.4 \
    --hash=sha256:92906c611ce6c967347bbfea733f13d6313901d54dcca88195eaeb52b2a8e8ee \
    --hash=sha256:d1216dfbdfb63826470995d31caed36225dcaf34f182e0fa257a4dd9e86f1b78
    # via django

# ...

Relevant part of setup.py

EXTRAS_REQUIRE = {
    'testing': [
        'pytest>=6.2.0',
        'pytest-cov>=2.11.1',
        'pytest-django>=4.2.0', 
        'factory-boy>=3.2.0',
        'faker>=8.1.0',
    ],
}

# ...

if __name__ == '__main__':
    setup(
        # ...

        extras_require=EXTRAS_REQUIRE,
    )

Command to install deps and run tests

tox

Output from GitHub Actions

Collecting typing-extensions
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
​ typing-extensions from https://files.pythonhosted.org/packages/2e/35/6c4fff5ab443b57116cb1aad46421fb719bed2825664e8fe77d66d99bcbc/typing_extensions-3.10.0.0-py3-none-any.whl#sha256=779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84
(from asgiref==3.3.4->-r requirements.txt (line 11))

=================================== log end ====================================

ERROR: could not install deps [-rrequirements.txt]; v = InvocationError('/home/runner/work/branch/branch/.tox/py37/bin/python -m pip install -rrequirements.txt', 1)
___________________________________ summary ____________________________________

ERROR: py37: could not install deps [-rrequirements.txt]; v = InvocationError('/home/runner/work/branch/branch/.tox/py37/bin/python -m pip install -rrequirements.txt', 1)

---

This issue occurs only for Python 3.7. Possible due this part of setup.cfg from asgiref repo:

[options]
python_requires = >=3.6
packages = find:
include_package_data = true
install_requires =
    typing_extensions; python_version < "3.8"
zip_safe = false

Possible related issues:

• Need your expertise on a possible bug in poetry 1.1.2 with export feature. cjolowicz/cookiecutter-hypermodern-python#583

---

Let me know if I can provide more information.

[nstylo]

I'd also like to see this feature. Currently we have some local dependencies checked into VCS (with git lfs) and I'd like to ignore those dependencies when running pip-compile --generate-hashes