Coverage breaks with implicit namespaces

[nedbat]

Originally reported by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

Under PEP-0420 there are implicit namespaces.

#!python
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 119, in do_coverage
    failure = _build_coverage_report(project, logger, execution_name, execution_prefix, coverage, modules)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 183, in _build_coverage_report
    module_report_data = _build_module_report(coverage, module)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 159, in _build_module_report
    return ModuleCoverageReport(coverage._analyze(module))
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/control.py", line 889, in _analyze
    it = self._get_file_reporter(it)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/control.py", line 914, in _get_file_reporter
    file_reporter = PythonFileReporter(morf, self)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/python.py", line 101, in __init__
    if filename.endswith(('.pyc', '.pyo')):
AttributeError: module 'namespace' has no attribute 'endswith'

This happens when coverage instruments "namespace.package.module"

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/456

[nedbat]

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

I will supply a patch via GitHub and attach a reproducible case here.

[nedbat]

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

@nedbat #1 with test case. Please let me know what you think

[nedbat]

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

Also fixed this in pybuilder pybuilder/pybuilder#301

[nedbat]

@arcivanov Thanks for this. I have no experience with namespace packages. The test in the GitHub pull request uses a generated module. Can you provide a way to create an actual namespace package to test against? I'd feel better if it were as real as possible.

[nedbat]

Original comment by RobertR (Bitbucket: rbtcollins, GitHub: rbtcollins)

---

What symptoms occur when using coverage with a project using implicit namespaces?

[nedbat]

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

@rbtcollins the stacktrace you see similar to above.

[nedbat]

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

@nedbat I'll just quote the PEP-0420 here if you don't mind:

During import processing, the import machinery will continue to iterate over each directory in the parent path as it does in Python 3.2. While looking for a module or package named "foo", for each directory in the parent path:

If <directory>/foo/__init__.py is found, a regular package is imported and returned.
If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform and whether the -O flag is specified. The list here is representative.
If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent path.
Otherwise the scan continues with the next directory in the parent path.

If the scan completes without returning a module or package, and at least one directory was recorded, then a namespace package is created. The new namespace package:

Has a __path__ attribute set to an iterable of the path strings that were found and recorded during the scan.
Does not have a __file__ attribute.

The latter is the root cause of the problem if namespace is submitted as a module to be inspected/analyzed.

[nedbat]

Thanks for the pointer to the PEP, I hadn't realized Python 3.3 included this change. I tried this scenario with no problem:

$ tree .
.
├── main.py
└── namespace
    └── package
        └── module.py

2 directories, 2 files
$ cat main.py
import namespace.package.module as m

print("Var is {}".format(m.VAR))
$ cat namespace/package/module.py
VAR = 17

$ python3.5 main.py
Var is 17
$ coverage-3.5 run main.py
Var is 17
$ coverage-3.5 report -m
Name                          Stmts   Miss  Cover   Missing
-----------------------------------------------------------
main.py                           2      0   100%
namespace/package/module.py       1      0   100%
-----------------------------------------------------------
TOTAL                             3      0   100%

If you could provide more details about exactly what you are trying that is failing, I would appreciate it.

[nedbat]

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)

---

Here's the specific scenario I'm targeting (pybuilder):

• Enumerate ALL modules in the the source directory
• For every module enumerated, retrieve module analysis object.

When you enumerate modules from the file system, unless you implement the above protocol exactly you don't know if a module is a namespace or a package or a module-proper. And if namespace is supplied directly, there is a blowout on coverage side.

Please see the workaround in PyBuilder: pybuilder/pybuilder#302

[nedbat]

Fixed in changeset 36bbed8ad2e5 (bb)