pip10 caching issue

[zsimic]

• Pip version: 10.0.0
• Python version: 2.7.10
• Operating system: OSX 10.12.6

Description:

There seems to be an issue with caching with the recent pip 10 publish.

What I've run:

# on a clone of https://github.com/zsimic/setupmeta
rm -rf ~/Library/Caches/pip
tox -e venv   # -> works
tox -r -e venv  # -> fails

# every pip install attempt fails if/when ~/Library/Caches/pip is populated

ERROR: invocation failed (exit code 2), logfile: /Users/zsimic/dev/github/setupmeta/.venv/log/venv-1.log
ERROR: actionid: venv
msg: getenv
cmdargs: ['/Users/zsimic/dev/github/setupmeta/.venv/bin/pip', 'install', '-rtests/requirements.txt']

Collecting mock (from -r tests/requirements.txt (line 1))
/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Exception:
Traceback (most recent call last):
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 243, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 307, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/index.py", line 484, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/index.py", line 442, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/index.py", line 587, in _get_pages
    page = self._get_page(location)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/index.py", line 705, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/index.py", line 814, in get_page
    "Cache-Control": "max-age=600",
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_internal/download.py", line 397, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 640, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 218, in resolve_redirects
    **adapter_kwargs
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/cachecontrol/adapter.py", line 42, in send
    cached_response = self.controller.cached_request(request)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/cachecontrol/controller.py", line 137, in cached_request
    resp = self.serializer.loads(request, cache_data)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/cachecontrol/serialize.py", line 98, in loads
    return getattr(self, "_loads_v{0}".format(ver))(request, data)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/cachecontrol/serialize.py", line 190, in _loads_v4
    cached = msgpack.loads(data, encoding='utf-8')
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/msgpack/fallback.py", line 121, in unpackb
    ret = unpacker._unpack()
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/msgpack/fallback.py", line 600, in _unpack
    ret[key] = self._unpack(EX_CONSTRUCT)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/msgpack/fallback.py", line 600, in _unpack
    ret[key] = self._unpack(EX_CONSTRUCT)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/msgpack/fallback.py", line 560, in _unpack
    typ, n, obj = self._read_header(execute)
  File "/Users/zsimic/dev/github/setupmeta/.venv/lib/python2.7/site-packages/pip/_vendor/msgpack/fallback.py", line 391, in _read_header
    n = struct.unpack_from(">H", self._buffer, self._buff_i)[0]
TypeError: unpack_from() argument 1 must be string or read-only buffer, not bytearray

ERROR: could not install deps [-rtests/requirements.txt]; v = InvocationError('/Users/zsimic/dev/github/setupmeta/.venv/bin/pip install -rtests/requirements.txt (see /Users/zsimic/dev/github/setupmeta/.venv/log/venv-1.log)', 2)

Workarounds:

• rm -rf ~/Library/Caches/pip and try again -> works
• ln -s /dev/null ~/Library/Caches/pip -> now things work all the time

Note:

• /usr/bin/python -c 'import pip; print(pip.__version__)' -> 9.0.1
• all venvs get latest pip 10.0.0 (starting today, and issue started occurring today)

[zsimic]

Same issue occurs if I use pip 10.0.0 exclusively

~: pyenv which python
/Users/zsimic/.pyenv/versions/2.7/bin/python
~: python -c 'import pip; print(pip.__version__)'
10.0.0

If I use the pyenv-installed 2.7, pip 10.0.0 is the only one ever used, and yet it seems to still somehow corrupt its own cache (whereas with OSX system python, it could have been due to prev pip 9.0.1 partially populating that cache...)

[dstufft]

Duplicate of #5231.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.