Switch to setuptools

[mvcisback]

From my experience setuptools is abit easier to develop with (install vs develop).

It also has great virtualenv integration.

[mvcisback]

setuptools comes with virtualenv, and as far as I can tell is the preferred distribution method (until wheels really start to replace eggs).

[mvcisback]

also setuptools and distribute merged about a year ago

[refi64]

Normally, I just use a try...catch statement to try to import setuptools, then fall back distutils. Something like:

try:
    from setuptools import setup
except:
    from distutils.core import setup

That way, the program will work on systems without setuptools, but you can use setuptools's extra commands(e.g. develop) if it is installed.

[mvcisback]

good call. I added a restriction of only falling back if its an import error.

[JukkaL]

Thanks for making the change!

I get an error about unknown encoding when using setuptools... this is easy to fix by setting up the mypy encoding in setup.py (it implements Python 3 function annotations in Python 2):

import sys

from mypy.codec import register    # <=== added this line

try:

It also seems to try to bytecode compile stub files, even though they are marked as data in setup.py, which is odd. I wonder how this could be prevented, other than by changing the extension.

[mvcisback]

So, I seem to have been mistaken when I thought it could be a drop in replacement.
data_files doesn't seem to be implemented in setuptools.

[refi64]

@mvcisback It technically works, but it must not work right in that manner. You could always try this(UNTESTED):

Create a file named MANIFEST.in with the following:

recursive-exclude stubs/3.4/*
recursive-exclude stubs/3.3/*
recursive-exclude stubs/3.2/*
recursive-exclude stubs/2.7/*

That might cause those files to not be byte-compiled.

[mvcisback]

If someone didn't want to install setuptools and such, couldn't they just get the distribution as a wheel? I think that might avoid all this headache

[JukkaL]

AFAIK, distros such as Ubuntu 12.04 can't trivially support wheels. It seems that there also needs to be a non-wheel fallback (which might be setuptools?).

How does setuptools help with virtualenv? I don't doubt that it does, I just don't know...

In the long term, I guess most people will probably use pip install mypy (once that's supported -- but we may want to provide Python 2 packages for mypy itself first) to get mypy, and only developers need to worry about dependencies. It's fine to require developers to install additional dependencies, assuming they don't clash with what they are likely to have installed already.

[mvcisback]

AFAIK, distros such as Ubuntu 12.04 can't trivially support wheels. It seems that there also needs to be a non-wheel fallback (which might be setuptools?).

The fallback would be an egg or downloading the source and using setuptools (which is easy to get on most platforms).

How does setuptools help with virtualenv? I don't doubt that it does, I just don't know...
For me the biggest 2 features are install_requirements and develop.

install_requirements allows you to add package dependencies... these will retrieved when installing.

Develop allows you to edit files and test them without constantly rerunning install. This seems trivial except its easy to forget this step and spends unnecessary time debugging.

link to comparison

It's fine to require developers to install additional dependencies, assuming they don't clash with what they are likely to have installed already.

This is what virtualenv is for. It creates a python sandbox so you don't have to worry about what they have installed already. Doing that globally is probably best left to distributions.

[mvcisback]

Also, although its abit clunky, pkg_resources serve as a great way to access datafiles and other things distributed with the package. Its alot cleaner than for example looking for the path of module using __file__

[mvcisback]

Update: This PR now also includes a change in the way data like stubs and such are accessed. This allows the removal of a lot of platform specific code and directory walking.

[JukkaL]

Sounds reasonable, but I can't get this to work. Here are my problems:

• After running python3 setup.py develop or python3 setup.py install, running mypy in PATH
  does not work -- it can't find mypy.main. It doesn't seem to install the sources.
• It still tries to compile stubs to .pyc files (which will fail) and it complains about the
  mypy encoding when byte-compiling mypy/codec/example.py. Maybe this is why it's not installing
  correctly.

I'm using Ubuntu 12.04 and Python 3.2 as my development environment.

[mvcisback]

That's very odd. I'll try with 3.2 and check back.
I'm running 3.4 on Arch Linux in a virtualenv

[mvcisback]

• After running python3 setup.py develop or python3 setup.py install, running mypy in PATH does not work -- it can't find mypy.main. It doesn't seem to install the sources.

I was able to reproduce. My virtualenv had some state because I had previously run python setup.py install.
Installing seems to behave differently because of package_dir={'': 'lib-typing/3.2'}
I think the motivation for this entry is to include typing.py, but the actual semantics of this are slightly different. '': 'lib-typing/3.2' means that lib-typing/3.2 is the root of all packages. Because of this, the egg link is placed in lib-typing/3.2 instead of the root directory.

• It still tries to compile stubs to .pyc files (which will fail) and it complains about the mypy encoding when byte-compiling mypy/codec/example.py. Maybe this is why it's not installing correctly.

I was finally able to reproduce this. I have the PYTHONDONTWRITEBYTECODE env var set so this hadn't come up. This also brought up the encoding issue...so I'll try and get these resolved and update the PR

[mvcisback]

@JukkaL aside from the environment var or using python -B setup.py develop I can't seem to not have it compile....setuptools is being alittle to smart for its own good.

I may be missing something, but the .py files are still available. Why will the pyc's cause an issue?
That said, I think #345 would also fix the byte compiling issue.

[mvcisback]

@JukkaL Also, I might be missing something...but is lib-typing supposed to be a separate project or something? 2.7 has its own setup.py.

[mvcisback]

Also, the reason why develop is currently broken:
https://bitbucket.org/pypa/setuptools/issue/230

TL;DR, package_dir is broken...we could avoid by restructuring the way the typing module (or package).

[JukkaL]

lib-typing is mostly separate from the rest of the project... it's just a dependency -- but it has to be in sync with the type checker. Maybe it should have its own setup.py (or even its own git repo).

[JukkaL]

BTW, I have fixed some of the setuptools-related issues (e.g. failures due to the codec, Python 2/3 syntax issue).

[JukkaL]

Can you check if changing the stub extensions (#345) seems to fix the remaining setuptools issues? If yes, we should prioritize to get #345 implemented first.

[mvcisback]

I'll take a look later today.

[mvcisback]

Yeah, as we expected, changing the extension fixes the byte compiling issue.
I'll make an attempt then at #345.

That makes python setup.py install work. python setup.py develop will be broken until we can figure out what to do with the typing module (or the setuptools bug gets fixed).

[JukkaL]

Maybe typing should live in a separate project, and it would just be a dependency?

I added a new issue: #377

[gvanrossum]

Hm. If we must change the filename extension used for stub files to something other than ".py" just to be able to use setuptools, then I am not in favor of it. A different extension would get in the way of various tools that are useful for stub files, e.g. automatically selecting the syntax mode for editors and IDEs. (I know you can configure this, but it's still a pain.)

[mvcisback]

@gvanrossum The filename extension was in consideration from another issue. It just had the added benefit of avoiding that issue.

I think the stubs byte compiling can be worked around even with the *.py extension.

That said, I think the alternative extension might be beneficial precisely because it doesn't work with existing infrastructure. The idea is documented in #345, but it'd be interesting to support annotations in *py files and "compiling" to *.py and *.pi. This would allow removing the typing dependency from the module for popular distribution, while allowing something that understands the annotations to optionally interpret them.

[JukkaL]

There hasn't been progress in a while. I'm going to close this PR for now -- we can continue the discussion in issue #482 and potentially reopen this PR.