Running setup.py in affiliated package is not possible unless numpy is already installed

[drdavella]

In a clean environment, running setup.py in any affiliated package will cause something like the error listed below. This is obviously unintuitive and undesirable behavior.

EDIT: This apparently only affects setup.py install/develop, but not setup.py egg_info, which is why it has not been detected in Astropy's CI builds.

It's not yet clear to me yet whether this can be fixed by putting some try/except logic around the imports below (and possibly elsewhere), or whether this is a more fundamental problem with the helpers strategy:

astropy-helpers/astropy_helpers/utils.py

Lines 59 to 79 in e2b5151

	def get_numpy_include_path():
	"""
	Gets the path to the numpy headers.
	"""
	# We need to go through this nonsense in case setuptools
	# downloaded and installed Numpy for us as part of the build or
	# install, since Numpy may still think it's in "setup mode", when
	# in fact we're ready to use it to build astropy now.
	import builtins
	if hasattr(builtins, '__NUMPY_SETUP__'):
	del builtins.__NUMPY_SETUP__
	import imp
	import numpy
	imp.reload(numpy)
	try:
	numpy_include = numpy.get_include()
	except AttributeError:
	numpy_include = numpy.get_numpy_include()
	return numpy_include

running build_ext
Traceback (most recent call last):
  File "./setup.py", line 143, in <module>
    **package_info
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/site-packages/setuptools/command/develop.py", line 138, in install_for_development
    self.run_command('build_ext')
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/ddavella/miniconda3/envs/ah-bottleneck/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/ddavella/sandbox/ah-bottleneck/astropy_helpers/astropy_helpers/commands/build_ext.py", line 292, in run
    np_include = get_numpy_include_path()
  File "/Users/ddavella/sandbox/ah-bottleneck/astropy_helpers/astropy_helpers/utils.py", line 72, in get_numpy_include_path
    import numpy
ModuleNotFoundError: No module named 'numpy'

[astrofrog]

Normally the way this should work is that if numpy is in install_requires, and if affiliated packages specify the include path as 'numpy' then we'd avoid this - but maybe something has changed recently in setuptools?

[astrofrog]

The proper solution is of course #334 but I think it might still be a while until we can rely on that.

[drdavella]

@astrofrog this problem occurs even if numpy is in install_requires.

[saimn]

numpy should be in setup_requires I think.

[saimn]

install_requires is just the runtime dependencies.

[drdavella]

@saimn then this should be part of the package template, but it is not currently.

[saimn]

@drdavella - probably because it is useful only if you build a C/Cython extension using Numpy. But it could be mentioned somewhere.

[drdavella]

@saimn I'm not sure that's true. This issue occurs for me with a new package created from cookiecutter that does not have any C extensions.

[saimn]

Ah indeed it is called even when this is not required:

astropy-helpers/astropy_helpers/commands/build_ext.py

Lines 289 to 297 in 7f11678

	def run(self):
	# For extensions that require 'numpy' in their include dirs,
	# replace 'numpy' with the actual paths
	np_include = get_numpy_include_path()
	for extension in self.extensions:
	if 'numpy' in extension.include_dirs:
	idx = extension.include_dirs.index('numpy')
	extension.include_dirs.insert(idx, np_include)
	extension.include_dirs.remove('numpy')

This call should be moved in the loop, after the if 'numpy'... check, to avoid importing numpy if not necessary.
Also there is an old issue in the template repo: astropy/package-template#107

[bsipocz]

I'm not sure that's true. This issue occurs for me with a new package created from cookiecutter that does not have any C extensions.

there may be still tons of issues with the cookiecutter, so issues and especially PRs are very much welcome for the template.

[drdavella]

@bsipocz I think the change here means no change in cookiecutter is needed right now. But I'll keep that in mind for future issues.