Unable to create vrtualenv with 1.11 (when virtualenv installed by ez_install/setuptools)

[s0undt3ch]

# virtualenv /tmp/123
Overwriting /tmp/123/lib/python2.7/site.py with new content
New python executable in /tmp/123/bin/python
Installing setuptools, pip...
  Complete output from command /tmp/123/bin/python -c "import sys, pip; pip...ll\"] + sys.argv[1:])" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/pip-1.5-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/pip-1.5-py2.py3-none-any.whl/pip/log.py", line 8, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 2696, in <module>
    def _dep_map(self):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 429, in __init__
    def get_resource_stream(manager, resource_name):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 443, in add_entry
    """Is the named resource a directory?  (like ``os.path.isdir()``)"""
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1722, in find_in_zip
    except AttributeError:
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1298, in has_metadata
    def test(nodelist):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1614, in _has
    def _parts(self,zip_path):
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1488, in _zipinfo_name
    "Can't perform this operation for loaders without 'get_data()'"
AssertionError: /usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/EGG-INFO/PKG-INFO is not a subpath of /usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 9, in <module>
    load_entry_point('virtualenv==1.11', 'console_scripts', 'virtualenv')()
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 820, in main
    symlink=options.symlink)
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 988, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 956, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 898, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /tmp/123/bin/python -c "import sys, pip; pip...ll\"] + sys.argv[1:])" setuptools pip failed with error code 1

# python -V
Python 2.7.3

# pip freeze
Cheetah==2.4.4
GnuPGInterface==0.3.2
Jinja2==2.6
Landscape-Client==12.05
M2Crypto==0.21.1
MarkupSafe==0.15
PAM==0.4.2
PyYAML==3.10
SaltTesting==0.5.5
Twisted-Core==11.1.0
apache-libcloud==0.13.3
apt-xapian-index==0.44
argparse==1.2.1
boto==2.2.2
chardet==2.0.1
cloud-init==0.6.3
command-not-found==0.2.44
configobj==4.7.2
coverage==3.7.1
euca2ools==2.0.0
httplib2==0.7.2
keyring==0.9.2
language-selector==0.1
launchpadlib==1.9.12
lazr.restfulclient==0.12.0
lazr.uri==1.0.3
mock==1.0.1
msgpack-python==0.1.10
oauth==1.0.1
paramiko==1.7.7.1
pyOpenSSL==0.12
pycrypto==2.4.1
pycurl==7.19.0
pyserial==2.5
python-apt==0.8.3ubuntu7.1
python-debian==0.1.21ubuntu1
pyzmq==13.0.0
salt==2014.1.0-31-g0998b54
simplejson==2.3.2
timelib==0.2.4
ufw==0.31.1-1
unattended-upgrades==0.1
unittest-xml-reporting==1.7.0
virtualenv==1.11
wadllib==1.3.0
wsgiref==0.1.2
zope.interface==3.6.1

Is there any more info that I can provide?

[pg1671]

+1

[ionelmc]

Same issue here

virtualenv .ve --python=/usr/bin/python2.7 --system-site-packages
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in .ve/bin/python2.7
Also creating executable in .ve/bin/python
Installing setuptools, pip...
  Complete output from command /home/nestea/deploye...f6/.ve/bin/python2.7 -c "import sys, pip; pip...ll\"] + sys.argv[1:])" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/pip-1.5-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
    # See the README file for information on usage and redistribution.
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/pip-1.5-py2.py3-none-any.whl/pip/log.py", line 8, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 2696, in <module>
    # Prepare the master working set and make the ``require()`` API available
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 429, in __init__
    self.callbacks = []
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 443, in add_entry
    always appended to ``.entries``, even if it is already present.
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1722, in find_in_zip
    """Yield distributions accessible on a sys.path directory"""
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1298, in has_metadata

  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1614, in _has

  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/pkg_resources.py", line 1488, in _zipinfo_name

AssertionError: /usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/EGG-INFO/PKG-INFO is not a subpath of /usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/setuptools-2.0.2-py2.py3-none-any.whl/

[xen]

I have different problem with the same virtualenv version. Instead of traceback I'm getting Could not find an activated virtualenv (required). and site-packages folder is empty after execution of $ virtualenv-2.7 --system-site-packages venv. The same time under py3.3 everything works. Sorry, python 3 version was outdated, the same error.

[qwcode]

marking for 1.11.1

[qwcode]

I can replicate this. this occurs when virtualenv is installed with easy_install (or "python setup.py install")

cc @pfmoore

[qwcode]

this is related to the zip imports that are now occurring.
given that the exception occurs after pip tries to import pkg_resources, it might be that we fix this simply by doing pypa/pip#1422

[pfmoore]

Yep, I see it too. Looks like a setuptools bug - the assertion error occurs in pkg_resources and it looks like it's because we have a zipfile on sys.path (i.e., we're using zipimport) where the zipfile is inside a setuptools "unpacked egg" format package, Beyond that, I don't know - I don't know why pkg_resources is making this check.

Maybe @jaraco can comment?

[pfmoore]

Not sure that pypa/pip#1422 will help - we'll still be using pkg_resources, just a vendored copy.

[qwcode]

better to fix the setuptools bug, but one thought if pip is independent of setuptools, is to not add the wheels to pythonpath (which leads to the zip pkg_resources failures) but rather to make the the pip wheel executable (not exactly sure how that works), and use it as such in the virtualenv creation

[qwcode]

the setuptools issue: https://bitbucket.org/pypa/setuptools/issue/129/assertionerror-egg-info-pkg-info-is-not-a

[pfmoore]

better to fix the setuptools bug, but one thought if pip is independent of
setuptools, is to not add the wheels to pythonpath (which leads to the zip
pkg_resources failures) but rather to make the the pip wheel executable
(not exactly sure how that works), and use it as such in the virtualenv
creation

That might work. And it might also be a useful feature anyway - to have a
one-file standalone pip that can be run without installing.

[flyte]

Is this the same issue as I'm having? Virtualenv works when installed with pip but not with easy_install. I have a different stacktrace to the OP though.

The ones using ez_setup.py have these versions:

pip 1.5 from /usr/local/lib/python2.7/dist-packages/pip-1.5-py2.7.egg (python 2.7)
setuptools 2.0.2
virtualenv 1.11

whereas the ones using distribute_setup.py have

pip 1.5 from /usr/local/lib/python2.7/dist-packages/pip-1.5-py2.7.egg (python 2.7)
distribute 0.6.49
virtualenv 1.11

The following are all done on a clean vagrant 'precise64' Ubuntu box.

WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | sudo python
sudo easy_install pip
sudo pip install virtualenv
virtualenv ve

WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl http://python-distribute.org/distribute_setup.py | sudo python
sudo easy_install pip
sudo pip install virtualenv
virtualenv ve

WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | sudo python
sudo easy_install pip
sudo easy_install virtualenv
virtualenv ve # Doesn't work
sudo pip uninstall virtualenv
sudo pip install virtualenv
rm -rf ve
virtualenv ve

WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl http://python-distribute.org/distribute_setup.py | sudo python
sudo easy_install pip
sudo easy_install virtualenv
virtualenv ve # Doesn't work
sudo pip uninstall virtualenv
sudo pip install virtualenv
rm -rf ve
virtualenv ve

NOT WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | sudo python
sudo easy_install virtualenv
virtualenv ve

NOT WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py | sudo python
sudo easy_install pip
sudo easy_install virtualenv
virtualenv ve

NOT WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl http://python-distribute.org/distribute_setup.py | sudo python
sudo easy_install virtualenv
virtualenv ve

NOT WORKING:

sudo apt-get update && sudo apt-get install -y curl && curl http://python-distribute.org/distribute_setup.py | sudo python
sudo easy_install pip
sudo easy_install virtualenv
virtualenv ve

The NOT WORKING ones all produce this output:

vagrant@precise64:~$ virtualenv ve
New python executable in ve/bin/python
Installing setuptools, pip...
  Complete output from command /home/vagrant/ve/bin/python -c "import sys, pip; pip...ll\"] + sys.argv[1:])" setuptools pip:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/pip-1.5-py2.py3-none-any.whl/pip/__init__.py", line 9, in <module>
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv_support/pip-1.5-py2.py3-none-any.whl/pip/log.py", line 8, in <module>
ImportError: No module named pkg_resources
----------------------------------------
...Installing setuptools, pip...done.
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 9, in <module>
    load_entry_point('virtualenv==1.11', 'console_scripts', 'virtualenv')()
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 820, in main
    symlink=options.symlink)
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 988, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 956, in install_wheel
    'PIP_NO_INDEX': '1'
  File "/usr/local/lib/python2.7/dist-packages/virtualenv-1.11-py2.7.egg/virtualenv.py", line 898, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/vagrant/ve/bin/python -c "import sys, pip; pip...ll\"] + sys.argv[1:])" setuptools pip failed with error code 1

[qwcode]

@pfmoore not sure if you're monitoring the setuptools issue, but that's fairly well understood now, and I opened a PR for discussion, if you care to make any comment.

https://bitbucket.org/pypa/setuptools/pull-request/32

the point being is that I don't think we have to rely on a pip-only solution to this

[pfmoore]

@qwcode yes, I saw your fix, looks OK to me (although I haven't actually tested it myself, as I can't do much more than apply the patch and try reproducing this issue, which I guess you've done).

I'd certainly much rather it get fixed in setuptools. If we get a new setuptools release, I assume we need to do a quick virtualenv release to bundle the updated setuptools. Not sure if ensurepip also needs updating.