virtualenv's develop branch fails to create virtualenvs with a different Python interpreter

[mgedmin]

Steps to reproduce:

1. git clone -b develop https://github.com/pypa/virtualenv
2. cd virtualenv
3. python2.7 virtualenv.py -p python3.4 /tmp/py34

The error is

Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
ERROR: The executable /tmp/py34/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/tmp/py34')
ERROR: virtualenv is not compatible with this system or executable

I get the same error if I use -p python2.6, or -p python3.3, or -p python3.2, or -p python3.1.

I don't get the error if I use -p python2.7 or -p pypy.

I don't get the error if I switch to the master branch and use -p python3.4.

[mgedmin]

Reverting commit dde220e fixes this for me.

[mgedmin]

The irony is that dde220e was supposed to fix the same issue as this one, only in a different way.

Pinging @nvie, @saulshanabrook: can you test if this PR doesn't regress your test case (#625)?

[mgedmin]

Apologies, "this PR" is #672. I meant to post the comment there, not here. Too many open tabs!

[nvie]

The funny thing is that I cannot reproduce the behaviour you mention:

$ git clone -b develop https://github.com/pypa/virtualenv
$ cd virtualenv
$ python2.7 virtualenv.py -p python3.4 /tmp/py34
Running virtualenv with interpreter /Users/nvie/.pythonz/pythons/CPython-3.4.2/bin/python3.4
Using base prefix '/Users/nvie/.pythonz/pythons/CPython-3.4.2'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
Installing setuptools, pip...done.
$ /tmp/py34/bin/python -c 'import sys; print(sys.version_info); print(sys.prefix)'
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
/tmp/py34

[nvie]

I've tried Python versions 2.6, 2.7, 3.2, 3.3, 3.4, but none of them pose this problem. They all work.

[mgedmin]

I'm on Ubuntu 14.10.

$ python2.7 virtualenv.py -p python3.4 /tmp/py34
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
ERROR: The executable /tmp/py34/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/tmp/py34')
ERROR: virtualenv is not compatible with this system or executable

$ /tmp/py34/bin/python -c 'import sys; print(sys.version_info); print(sys.prefix)'
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
/usr

$ tree /tmp/py34
/tmp/py34
├── bin
│   ├── python -> python3.4
│   ├── python3 -> python3.4
│   └── python3.4
├── include
│   └── python3.4m -> /usr/include/python3.4m
└── lib
    └── python3.4
        ├── config-3.4dm-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4dm-x86_64-linux-gnu
        ├── config-3.4m-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4m-x86_64-linux-gnu
        ├── lib-dynload -> /usr/lib/python3.4/lib-dynload
        ├── no-global-site-packages.txt
        ├── orig-prefix.txt
        ├── plat-x86_64-linux-gnu -> /usr/lib/python3.4/plat-x86_64-linux-gnu
        ├── site-packages
        └── site.py

[mgedmin]

As opposed to

$ python3.4 virtualenv.py /tmp/py34good --no-setuptools
Using base prefix '/usr'
New python executable in /tmp/py34good/bin/python3.4
Also creating executable in /tmp/py34good/bin/python

$ /tmp/py34good/bin/python -c 'import sys; print(sys.version_info); print(sys.prefix)'
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
/tmp/py34good

$ tree /tmp/py34good
/tmp/py34good
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── python -> python3.4
│   ├── python3 -> python3.4
│   └── python3.4
├── include
│   └── python3.4m -> /usr/include/python3.4m
└── lib
    └── python3.4
        ├── abc.py -> /usr/lib/python3.4/abc.py
        ├── base64.py -> /usr/lib/python3.4/base64.py
        ├── bisect.py -> /usr/lib/python3.4/bisect.py
        ├── _bootlocale.py -> /usr/lib/python3.4/_bootlocale.py
        ├── codecs.py -> /usr/lib/python3.4/codecs.py
        ├── collections -> /usr/lib/python3.4/collections
        ├── _collections_abc.py -> /usr/lib/python3.4/_collections_abc.py
        ├── config-3.4dm-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4dm-x86_64-linux-gnu
        ├── config-3.4m-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4m-x86_64-linux-gnu
        ├── copy.py -> /usr/lib/python3.4/copy.py
        ├── copyreg.py -> /usr/lib/python3.4/copyreg.py
        ├── distutils
        │   ├── distutils.cfg
        │   └── __init__.py
        ├── _dummy_thread.py -> /usr/lib/python3.4/_dummy_thread.py
        ├── encodings -> /usr/lib/python3.4/encodings
        ├── fnmatch.py -> /usr/lib/python3.4/fnmatch.py
        ├── functools.py -> /usr/lib/python3.4/functools.py
        ├── __future__.py -> /usr/lib/python3.4/__future__.py
        ├── genericpath.py -> /usr/lib/python3.4/genericpath.py
        ├── hashlib.py -> /usr/lib/python3.4/hashlib.py
        ├── heapq.py -> /usr/lib/python3.4/heapq.py
        ├── hmac.py -> /usr/lib/python3.4/hmac.py
        ├── importlib -> /usr/lib/python3.4/importlib
        ├── imp.py -> /usr/lib/python3.4/imp.py
        ├── io.py -> /usr/lib/python3.4/io.py
        ├── keyword.py -> /usr/lib/python3.4/keyword.py
        ├── lib-dynload -> /usr/lib/python3.4/lib-dynload
        ├── linecache.py -> /usr/lib/python3.4/linecache.py
        ├── locale.py -> /usr/lib/python3.4/locale.py
        ├── no-global-site-packages.txt
        ├── ntpath.py -> /usr/lib/python3.4/ntpath.py
        ├── operator.py -> /usr/lib/python3.4/operator.py
        ├── orig-prefix.txt
        ├── os.py -> /usr/lib/python3.4/os.py
        ├── __pycache__
        │   ├── abc.cpython-34.pyc
        │   ├── _bootlocale.cpython-34.pyc
        │   ├── codecs.cpython-34.pyc
        │   ├── _collections_abc.cpython-34.pyc
        │   ├── genericpath.cpython-34.pyc
        │   ├── io.cpython-34.pyc
        │   ├── os.cpython-34.pyc
        │   ├── posixpath.cpython-34.pyc
        │   ├── site.cpython-34.pyc
        │   ├── stat.cpython-34.pyc
        │   └── _weakrefset.cpython-34.pyc
        ├── plat-x86_64-linux-gnu -> /usr/lib/python3.4/plat-x86_64-linux-gnu
        ├── posixpath.py -> /usr/lib/python3.4/posixpath.py
        ├── random.py -> /usr/lib/python3.4/random.py
        ├── re.py -> /usr/lib/python3.4/re.py
        ├── reprlib.py -> /usr/lib/python3.4/reprlib.py
        ├── rlcompleter.py -> /usr/lib/python3.4/rlcompleter.py
        ├── shutil.py -> /usr/lib/python3.4/shutil.py
        ├── site-packages
        ├── site.py
        ├── sre_compile.py -> /usr/lib/python3.4/sre_compile.py
        ├── sre_constants.py -> /usr/lib/python3.4/sre_constants.py
        ├── sre_parse.py -> /usr/lib/python3.4/sre_parse.py
        ├── stat.py -> /usr/lib/python3.4/stat.py
        ├── struct.py -> /usr/lib/python3.4/struct.py
        ├── tarfile.py -> /usr/lib/python3.4/tarfile.py
        ├── tempfile.py -> /usr/lib/python3.4/tempfile.py
        ├── types.py -> /usr/lib/python3.4/types.py
        ├── tokenize.py -> /usr/lib/python3.4/tokenize.py
        ├── token.py -> /usr/lib/python3.4/token.py
        ├── warnings.py -> /usr/lib/python3.4/warnings.py
        ├── weakref.py -> /usr/lib/python3.4/weakref.py
        └── _weakrefset.py -> /usr/lib/python3.4/_weakrefset.py

[mgedmin]

I suspect what happens here is that sys.path[0] is now deleted twice: once at the top level and once again in copy_required_modules, which makes the latter be unable to find the required bootstrap modules.

My sys.path looks like this:

$ python3.4 -c 'import sys; print("\n".join(map(repr, sys.path)))'
''
'/usr/lib/python3.4'
'/usr/lib/python3.4/plat-x86_64-linux-gnu'
'/usr/lib/python3.4/lib-dynload'
'/home/mg/.local/lib/python3.4/site-packages'
'/usr/local/lib/python3.4/dist-packages'
'/usr/lib/python3/dist-packages'

and if I run virtualenv with the -v flag, I see the expected errors about "Cannot import bootstrap module":

$ python2.7 virtualenv.py -p python3.4 /tmp/py34 -v
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
Directory /tmp/py34/lib/python3.4 already exists
Symlinking Python bootstrap modules
  Cannot import bootstrap module: os
  Ignoring built-in bootstrap module: posix
  Cannot import bootstrap module: posixpath
  Cannot import bootstrap module: nt
  Cannot import bootstrap module: ntpath
  Cannot import bootstrap module: genericpath
  Cannot import bootstrap module: fnmatch
  Cannot import bootstrap module: locale
  Cannot import bootstrap module: encodings
  Cannot import bootstrap module: codecs
  Cannot import bootstrap module: stat
  Cannot import bootstrap module: UserDict
  Cannot import bootstrap module: copy_reg
  Cannot import bootstrap module: types
  Cannot import bootstrap module: re
  Cannot import bootstrap module: sre
  Cannot import bootstrap module: sre_parse
  Cannot import bootstrap module: sre_constants
  Cannot import bootstrap module: sre_compile
  Ignoring built-in bootstrap module: zlib
  Cannot import bootstrap module: _abcoll
  Cannot import bootstrap module: warnings
  Cannot import bootstrap module: linecache
  Cannot import bootstrap module: abc
  Cannot import bootstrap module: io
  Cannot import bootstrap module: _weakrefset
  Cannot import bootstrap module: copyreg
  Cannot import bootstrap module: tempfile
  Cannot import bootstrap module: random
  Cannot import bootstrap module: __future__
  Cannot import bootstrap module: collections
  Cannot import bootstrap module: keyword
  Cannot import bootstrap module: tarfile
  Cannot import bootstrap module: shutil
  Cannot import bootstrap module: struct
  Cannot import bootstrap module: copy
  Cannot import bootstrap module: tokenize
  Cannot import bootstrap module: token
  Cannot import bootstrap module: functools
  Cannot import bootstrap module: heapq
  Cannot import bootstrap module: bisect
  Cannot import bootstrap module: weakref
  Cannot import bootstrap module: reprlib
  Cannot import bootstrap module: base64
  Cannot import bootstrap module: _dummy_thread
  Cannot import bootstrap module: hashlib
  Cannot import bootstrap module: hmac
  Cannot import bootstrap module: imp
  Cannot import bootstrap module: importlib
  Cannot import bootstrap module: rlcompleter
  Cannot import bootstrap module: operator
  Cannot import bootstrap module: _collections_abc
  Cannot import bootstrap module: _bootlocale
Directory /tmp/py34/lib/python3.4/site-packages already exists
Content /tmp/py34/lib/python3.4/site.py already in place
Content /tmp/py34/lib/python3.4/orig-prefix.txt already in place
Content /tmp/py34/lib/python3.4/no-global-site-packages.txt already in place
Directory /tmp/py34/bin already exists
New python executable in /tmp/py34/bin/python3.4
Changed mode of /tmp/py34/bin/python3.4 to 0o775
Not overwriting existing python script /tmp/py34/bin/python (you must use /tmp/py34/bin/python3.4)
Testing executable with /tmp/py34/bin/python3.4 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
ERROR: The executable /tmp/py34/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/tmp/py34')
ERROR: virtualenv is not compatible with this system or executable

I'm curious to know why virtualenv doesn't fail for you.

[mgedmin]

Is PYTHONPATH set in your environment? Because in that case there are extra items in front of the stdlib path in sys.path and this error does not occur.

$ PYTHONPATH=something:extra python3.4 -c 'import sys; print("\n".join(map(repr, sys.path)))'
''
'/home/mg/src/virtualenv/something'
'/home/mg/src/virtualenv/extra'
'/usr/lib/python3.4'
'/usr/lib/python3.4/plat-x86_64-linux-gnu'
'/usr/lib/python3.4/lib-dynload'
'/home/mg/.local/lib/python3.4/site-packages'
'/usr/local/lib/python3.4/dist-packages'
'/usr/lib/python3/dist-packages'

$ PYTHONPATH=somethingextra python2.7 virtualenv.py -p python3.4 /tmp/py34
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Not overwriting existing python script /tmp/py34/bin/python (you must use /tmp/py34/bin/python3.4)
Installing setuptools, pip...done.

[nvie]

I suspect what happens here is that sys.path[0] is now deleted twice: once at the top level and once again in copy_required_modules […]

Ohhh. I wasn't even aware this was happening when I added my patch. Therefore I might indeed have broken something. Removing it at the top level is required to make sure the following imports works as expected. I think we can safely remove the sys.path modification from the copy_required_modules() function (in favor of the top-level one). What happens if you simply remove that part?

[mgedmin]

Here's the patch that I tested on top of current develop (commit b7e489f):

diff --git a/virtualenv.py b/virtualenv.py
index 2f43427..dd23ff3 100755
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -1113,8 +1113,8 @@ def copy_required_modules(dst_prefix, symlink):
     # virtualenv.py is installed under (which might lead to py2/py3
     # incompatibility issues)
     _prev_sys_path = sys.path
-    if os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
-        sys.path = sys.path[1:]
+##  if os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
+##      sys.path = sys.path[1:]
     try:
         for modname in REQUIRED_MODULES:
             if modname in sys.builtin_module_names:

With it the following works:

$ python2.7 virtualenv.py -p python3.4 /tmp/py34 
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
Installing setuptools, pip...done.

but #671 remains a problem for me. Digging deeper, the del sys.path[0] is on line 30, but the import error I get happens before that, on line 8 (import base64). To fix that you'd have to reorder the imports (put import sys at the top) and then do the sys.path deletion before importing the rest (especially base64).

My patch in #672 fixes both this bug and #671. I ike it conceptually (the wrong Python's site-packages never get a chance to show up in sys.path) but dislike the implementation (the one-liner that exec()s the contents of a script is grotty).

[nvie]

I've given the exec() call some thought myself when I implemented my patch, but I refrained from it because it does depend on implementation details of the runtime (i.e. you're setting __file__ and __name__, but __loader__ or __package__ may also exist. Do they need to be set too? Are there more? I wanted to stay away from a solution that would force decisions there.)

Have you tried simply moving the base64 import down until after the path modification? That sounds like the correct thing to do anyway, and it may also be the simplest solution.