Skip to content

Coverage breaks with implicit namespaces #456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nedbat opened this issue Dec 19, 2015 · 10 comments
Closed

Coverage breaks with implicit namespaces #456

nedbat opened this issue Dec 19, 2015 · 10 comments
Labels
bug Something isn't working

Comments

@nedbat
Copy link
Owner

nedbat commented Dec 19, 2015

Originally reported by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


Under PEP-0420 there are implicit namespaces.

#!python
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 119, in do_coverage
    failure = _build_coverage_report(project, logger, execution_name, execution_prefix, coverage, modules)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 183, in _build_coverage_report
    module_report_data = _build_module_report(coverage, module)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/pybuilder/plugins/python/coverage_plugin.py", line 159, in _build_module_report
    return ModuleCoverageReport(coverage._analyze(module))
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/control.py", line 889, in _analyze
    it = self._get_file_reporter(it)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/control.py", line 914, in _get_file_reporter
    file_reporter = PythonFileReporter(morf, self)
  File "/Users/arcivanov/.pyenv/versions/3.5.1/envs/project-venv/lib/python3.5/site-packages/coverage/python.py", line 101, in __init__
    if filename.endswith(('.pyc', '.pyo')):
AttributeError: module 'namespace' has no attribute 'endswith'

This happens when coverage instruments "namespace.package.module"


@nedbat
Copy link
Owner Author

nedbat commented Dec 19, 2015

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


I will supply a patch via GitHub and attach a reproducible case here.

@nedbat
Copy link
Owner Author

nedbat commented Dec 21, 2015

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


@nedbat #1 with test case. Please let me know what you think

@nedbat
Copy link
Owner Author

nedbat commented Dec 21, 2015

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


Also fixed this in pybuilder pybuilder/pybuilder#301

@nedbat
Copy link
Owner Author

nedbat commented Dec 27, 2015

@arcivanov Thanks for this. I have no experience with namespace packages. The test in the GitHub pull request uses a generated module. Can you provide a way to create an actual namespace package to test against? I'd feel better if it were as real as possible.

@nedbat
Copy link
Owner Author

nedbat commented Jan 13, 2016

Original comment by RobertR (Bitbucket: rbtcollins, GitHub: rbtcollins)


What symptoms occur when using coverage with a project using implicit namespaces?

@nedbat
Copy link
Owner Author

nedbat commented Jan 13, 2016

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


@rbtcollins the stacktrace you see similar to above.

@nedbat
Copy link
Owner Author

nedbat commented Jan 13, 2016

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


@nedbat I'll just quote the PEP-0420 here if you don't mind:

During import processing, the import machinery will continue to iterate over each directory in the parent path as it does in Python 3.2. While looking for a module or package named "foo", for each directory in the parent path:

If <directory>/foo/__init__.py is found, a regular package is imported and returned.
If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform and whether the -O flag is specified. The list here is representative.
If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent path.
Otherwise the scan continues with the next directory in the parent path.

If the scan completes without returning a module or package, and at least one directory was recorded, then a namespace package is created. The new namespace package:

Has a __path__ attribute set to an iterable of the path strings that were found and recorded during the scan.
Does not have a __file__ attribute.

The latter is the root cause of the problem if namespace is submitted as a module to be inspected/analyzed.

@nedbat
Copy link
Owner Author

nedbat commented Jan 19, 2016

Thanks for the pointer to the PEP, I hadn't realized Python 3.3 included this change. I tried this scenario with no problem:

$ tree .
.
├── main.py
└── namespace
    └── package
        └── module.py

2 directories, 2 files
$ cat main.py
import namespace.package.module as m

print("Var is {}".format(m.VAR))
$ cat namespace/package/module.py
VAR = 17

$ python3.5 main.py
Var is 17
$ coverage-3.5 run main.py
Var is 17
$ coverage-3.5 report -m
Name                          Stmts   Miss  Cover   Missing
-----------------------------------------------------------
main.py                           2      0   100%
namespace/package/module.py       1      0   100%
-----------------------------------------------------------
TOTAL                             3      0   100%

If you could provide more details about exactly what you are trying that is failing, I would appreciate it.

@nedbat
Copy link
Owner Author

nedbat commented Feb 2, 2016

Original comment by Arcadiy Ivanov (Bitbucket: arcivanov, GitHub: arcivanov)


Here's the specific scenario I'm targeting (pybuilder):

  • Enumerate ALL modules in the the source directory
  • For every module enumerated, retrieve module analysis object.

When you enumerate modules from the file system, unless you implement the above protocol exactly you don't know if a module is a namespace or a package or a module-proper. And if namespace is supplied directly, there is a blowout on coverage side.

Please see the workaround in PyBuilder: pybuilder/pybuilder#302

@nedbat
Copy link
Owner Author

nedbat commented May 8, 2016

Fixed in changeset 36bbed8ad2e5 (bb)

@nedbat nedbat closed this as completed May 8, 2016
@nedbat nedbat added critical bug Something isn't working labels Jun 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant