Defer all types whos metaclass is not ready #13579

sobolevn · 2022-09-01T10:03:52Z

Finally! I've spent more than 8 hours on this test.
It illustrates a problem with sympy from #13565

sobolevn · 2022-09-01T16:42:07Z

I got it right to pass my test, but now we have different problems:

    deferred, incomplete, progress = semantic_analyze_target(
  File "/Users/sobolev/Desktop/mypy/mypy/semanal_main.py", line 348, in semantic_analyze_target
    infer_decorator_signature_if_simple(node, analyzer)
  File "/Users/sobolev/.pyenv/versions/3.8.9/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sobolev/Desktop/mypy/mypy/build.py", line 2064, in wrap_context
    yield
  File "/Users/sobolev/Desktop/mypy/mypy/semanal_main.py", line 339, in semantic_analyze_target
    analyzer.refresh_partial(
  File "/Users/sobolev/Desktop/mypy/mypy/semanal.py", line 575, in refresh_partial
    self.refresh_top_level(node)
  File "/Users/sobolev/Desktop/mypy/mypy/semanal.py", line 584, in refresh_top_level
    self.add_implicit_module_attrs(file_node)
  File "/Users/sobolev/Desktop/mypy/mypy/semanal.py", line 616, in add_implicit_module_attrs
    assert isinstance(node, TypeInfo)
AssertionError:

Big 'thank you' goes to @ilevkivskyi for pushing me in the right direction.
Now I will need to find what is causing this new crash.

sobolevn · 2022-09-02T15:56:58Z

@ilevkivskyi maybe you have any ideas where we can safely call .add_implicit_module_attrs? Because I am out of ideas. I've tried:

Different places of semanal_main, like: https://github.com/python/mypy/pull/13579/files#diff-2924f9c93ed3b7e350efc02d9068d570cf76f32dc9485fee43df5ee11d4d9ae8R219
SemanticAnalyzer.process_top_level and other methods

Looks like I am stuck: it still fails for some cases 😒

ilevkivskyi · 2022-09-02T18:25:00Z

As I mentioned yesterday on discord, you will likely need to special-case builtins SCC, i.e. for this SCC add implicit symbols after processing the whole SCC, but for others keep the things as is.

sobolevn · 2022-09-02T21:21:58Z

Thank you for your guidance! Without it I would spend so much more time.

ilevkivskyi

Generally looks good, I have couple more suggestions/ideas.

ilevkivskyi · 2022-09-03T09:27:30Z

mypy/semanal.py

+        # We do it in the very last order, because of
+        # `builtins.dict <-> typing.Mapping <-> abc.ABCMeta`
+        # cyclic imports.
+        if file_node.fullname not in ("typing", "abc", "builtins"):


I would instead re-use core_modules (defined in semanal_main.py) here and in other place. Also you may need to use a flag for this method to avoid an import cycle with the latter (like skip_implicit_attrs).

ilevkivskyi · 2022-09-03T09:35:17Z

mypy/semanal.py

-        self.add_implicit_module_attrs(file_node)
+        # We do it in the very last order, because of
+        # `builtins.dict <-> typing.Mapping <-> abc.ABCMeta`
+        # cyclic imports.


The problem is not the cyclic import itself, but that the method assumes that e.g. dict (for __annotations__) would be never marked incomplete (i.e. it may be not defined yet, but as soon as it is, it will be TypeInfo, rather than PlaceholderNode). This makes me think maybe a cleaner solution (if it works), would be just to replace assert with an if (and simply continue if it is a placeholder, like we do if the symbol is None). Can you try this?

Btw curiously this seem to only affect tests for now, it looks in real typeshed currently dict is not an instance of ABCMeta.

Can you try this?

Done, looks like it works! Let's wait to see what tests will show us :)

sobolevn · 2022-09-03T12:52:01Z

mypy_primer fails with:

 Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 940, in main
    retcode = inner()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 933, in inner
    retcode = asyncio.run(coro)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 753, in primer
    result = await result_fut
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 414, in primer_result
    await self.setup()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 345, in setup
    repo_dir = await ensure_repo_at_revision(
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 168, in ensure_repo_at_revision
    await clone(repo_url, cwd, shallow=revision_like is None)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 134, in clone
    await run(cmd, cwd=cwd)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 110, in run
    raise subprocess.CalledProcessError(proc.returncode, cmd, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', 'https://gitlab.com/dkg/python-sop', '--depth', '1']' returned non-zero exit status 128.

I am sure it is not related, I will try to retrigger it a bit later.

AlexWaygood · 2022-09-03T12:57:26Z

mypy_primer fails with:

 Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 940, in main
    retcode = inner()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 933, in inner
    retcode = asyncio.run(coro)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 753, in primer
    result = await result_fut
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 414, in primer_result
    await self.setup()
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 345, in setup
    repo_dir = await ensure_repo_at_revision(
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 168, in ensure_repo_at_revision
    await clone(repo_url, cwd, shallow=revision_like is None)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 134, in clone
    await run(cmd, cwd=cwd)
  File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 110, in run
    raise subprocess.CalledProcessError(proc.returncode, cmd, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', 'https://gitlab.com/dkg/python-sop', '--depth', '1']' returned non-zero exit status 128.

I am sure it is not related, I will try to retrigger it a bit later.

We have the same problem on all typeshed PRs at the moment:

mypy-primer is failing on PRs typeshed#8677

github-actions · 2022-09-03T13:17:00Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

sobolevn · 2022-09-03T17:34:03Z

Thanks everyone! Next is #13565

Test that illustrates sympy metaclass problem

8fbd00a

sobolevn force-pushed the metaclass-semanal-failure branch from 5f310f2 to 8fbd00a Compare September 1, 2022 10:09

Progress!

3b51864

sobolevn changed the title ~~Test that illustrates sympy metaclass problem~~ Defer all types whos metaclass is not ready Sep 1, 2022

This comment has been minimized.

Sign in to view

sobolevn added 3 commits September 2, 2022 15:06

Fix builtins cycle import

276786f

Fix ci

4d24290

Fix ci

b502458

This comment has been minimized.

Sign in to view

Fix ci

914fc70

This comment has been minimized.

Sign in to view

sobolevn added 2 commits September 2, 2022 18:19

Fix CI

d2b5bea

Try this setup

9e2800e

This comment has been minimized.

Sign in to view

sobolevn added 2 commits September 2, 2022 23:08

Address review

9dc5b1c

Fix ci

50b5d49

This comment has been minimized.

Sign in to view

sobolevn added 2 commits September 2, 2022 23:46

Tests pass locally

7248fe1

Tests pass locally

57c5bfe

This comment has been minimized.

Sign in to view

sobolevn requested a review from ilevkivskyi September 2, 2022 21:21

ilevkivskyi reviewed Sep 3, 2022

View reviewed changes

sobolevn mentioned this pull request Sep 3, 2022

Detect metaclass conflicts, refs #13563 #13565

Closed

Try defer on incomplete implicit attrs

47fa22a

sobolevn closed this Sep 3, 2022

sobolevn reopened this Sep 3, 2022

This comment has been minimized.

Sign in to view

sobolevn requested a review from ilevkivskyi September 3, 2022 13:49

ilevkivskyi approved these changes Sep 3, 2022

View reviewed changes

sobolevn merged commit dfbaff7 into python:master Sep 3, 2022

sobolevn mentioned this pull request Sep 3, 2022

Detect Any used as a metaclass #13599

Closed

This was referenced Nov 1, 2022

typing: remove metaclass from Sized python/typeshed#9058

Merged

Sync typeshed #13977

Closed

Adding metaclass=ABCMeta to Sized breaks tuples #13986

Open

hauntsaninja mentioned this pull request Nov 29, 2022

calling .register on typing.KeysView subclass causes false positive error Member "register" is unknown python/typeshed#9296

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Defer all types whos metaclass is not ready #13579

Defer all types whos metaclass is not ready #13579

sobolevn commented Sep 1, 2022

sobolevn commented Sep 1, 2022 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sobolevn commented Sep 2, 2022

This comment has been minimized.

ilevkivskyi commented Sep 2, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sobolevn commented Sep 2, 2022

ilevkivskyi left a comment

ilevkivskyi Sep 3, 2022

ilevkivskyi Sep 3, 2022

sobolevn Sep 3, 2022

sobolevn commented Sep 3, 2022

AlexWaygood commented Sep 3, 2022 •

edited

Loading

This comment has been minimized.

github-actions bot commented Sep 3, 2022

sobolevn commented Sep 3, 2022

Defer all types whos metaclass is not ready #13579

Defer all types whos metaclass is not ready #13579

Conversation

sobolevn commented Sep 1, 2022

sobolevn commented Sep 1, 2022 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sobolevn commented Sep 2, 2022

This comment has been minimized.

ilevkivskyi commented Sep 2, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sobolevn commented Sep 2, 2022

ilevkivskyi left a comment

Choose a reason for hiding this comment

ilevkivskyi Sep 3, 2022

Choose a reason for hiding this comment

ilevkivskyi Sep 3, 2022

Choose a reason for hiding this comment

sobolevn Sep 3, 2022

Choose a reason for hiding this comment

sobolevn commented Sep 3, 2022

AlexWaygood commented Sep 3, 2022 • edited Loading

This comment has been minimized.

github-actions bot commented Sep 3, 2022

sobolevn commented Sep 3, 2022

sobolevn commented Sep 1, 2022 •

edited

Loading

AlexWaygood commented Sep 3, 2022 •

edited

Loading