gh-135700: Fix instructions in annotate have incorrect code positions #135814

AndPuQing · 2025-06-22T10:45:31Z

Issue: some instructions in __annotate__ have incorrect code positions #135700

from dis import Bytecode, dis

src = """\
22
333
__dataclass_fields__: ClassVar
"""

code = compile(src, "<string>", "exec")
print(code.co_consts)
print(dis(src))
annotate_code = code.co_consts[1]

for bc in Bytecode(annotate_code):
    print(bc.positions, bc.opname)

output

(0, <code object __annotate__ at 0x7b19f4095f20, file "<string>", line 1>, None)
  --           MAKE_CELL                0 (__conditional_annotations__)

   0           RESUME                   0

   1           LOAD_CONST               1 (<code object __annotate__ at 0x7b19f4096c30, file "<dis>", line 1>)
               MAKE_FUNCTION
               STORE_NAME               1 (__annotate__)
               BUILD_SET                0
               STORE_NAME               0 (__conditional_annotations__)

   2           NOP

   3           LOAD_NAME                0 (__conditional_annotations__)
               LOAD_SMALL_INT           0
               SET_ADD                  1
               POP_TOP
               LOAD_CONST               2 (None)
               RETURN_VALUE

Disassembly of <code object __annotate__ at 0x7b19f4096c30, file "<dis>", line 1>:
  1           RESUME                   0
              LOAD_FAST_BORROW         0 (format)
              LOAD_SMALL_INT           2
              COMPARE_OP             132 (>)
              POP_JUMP_IF_FALSE        3 (to L1)
              NOT_TAKEN
              LOAD_COMMON_CONSTANT     1 (NotImplementedError)
              RAISE_VARARGS            1
      L1:     BUILD_MAP                0

  3           LOAD_SMALL_INT           0
              LOAD_GLOBAL              0 (__conditional_annotations__)
              CONTAINS_OP              0 (in)
              POP_JUMP_IF_FALSE       10 (to L2)
              NOT_TAKEN
              LOAD_GLOBAL              2 (ClassVar)
              COPY                     2
              LOAD_CONST               1 ('__dataclass_fields__')

  1           STORE_SUBSCR
      L2:     RETURN_VALUE
None
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_FAST_BORROW
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_SMALL_INT
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) COMPARE_OP
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) POP_JUMP_IF_FALSE
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) NOT_TAKEN
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_COMMON_CONSTANT
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RAISE_VARARGS
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) BUILD_MAP
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_SMALL_INT
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_GLOBAL
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) CONTAINS_OP
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) POP_JUMP_IF_FALSE
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) NOT_TAKEN
Positions(lineno=3, end_lineno=3, col_offset=22, end_col_offset=30) LOAD_GLOBAL
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) COPY
Positions(lineno=3, end_lineno=3, col_offset=0, end_col_offset=30) LOAD_CONST
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) STORE_SUBSCR
Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) RETURN_VALUE

Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issue-135700.0qdtCl.rst

…e-135700.0qdtCl.rst Co-authored-by: sobolevn <[email protected]>

JelleZijlstra · 2025-06-22T13:38:57Z

Good catch, thanks! Would you be able to add a unit test? Also test a class with annotations so we can see how it works in a class-level __annotate__ function.

Lib/test/test_dis.py

iritkatriel · 2025-06-23T19:04:50Z

Lib/test/test_dis.py

+
+        for case_name, annotate_code in test_cases:
+            with self.subTest(case=case_name):
+                instructions = list(dis.Bytecode(annotate_code))


You can use dis.findlinestarts(annotate_code) instead.

Lib/test/test_dis.py

Co-authored-by: Irit Katriel <[email protected]>

iritkatriel · 2025-06-24T11:42:46Z

Lib/test/test_dis.py

+            ),
+        ]
+
+        for case_name, annotate_code in test_cases:


assert here that annotate_code is a code object and that annotate_code.co_name is '__annotate__'.

iritkatriel · 2025-06-24T11:44:46Z

Lib/test/test_dis.py

+                    for instr in dis.get_instructions(annotate_code)
+                    if setup_scope_begin <= instr.offset < setup_scope_end
+                    and instr.positions
+                }


Why does this need to be so complicated? Don't we just need to check that all the instructions in this code object have the same line number (excluding Nones)?

Because the important thing is to make sure the instructions in setup_annotations_scope are consistent with the end_col_offset of RESUME, not just the lineno

no fix:

Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_FAST_BORROW # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_SMALL_INT # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) COMPARE_OP # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) POP_JUMP_IF_FALSE # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) NOT_TAKEN # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) LOAD_COMMON_CONSTANT # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) RAISE_VARARGS # incorrect Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=2) BUILD_MAP # incorrect

fix:

Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RESUME Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_FAST_BORROW Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_SMALL_INT Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) COMPARE_OP Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) POP_JUMP_IF_FALSE Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) NOT_TAKEN Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) LOAD_COMMON_CONSTANT Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) RAISE_VARARGS Positions(lineno=1, end_lineno=1, col_offset=0, end_col_offset=0) BUILD_MAP

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

Ok, so we need to check they all have the same end_col_offset as well.

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

The test doesn't need to work for all inputs. Only the two inputs it runs on.

And not all annotate lines are the same, because some opcodes need to indicate the "code" where the annotation occurs.

The test doesn't need to work for all inputs. Only the two inputs it runs on.

Just to be clear, instructions within code object can correspond to different line numbers. So I made an offset restriction here, limiting the check to between RESUME and BUILD_MAP (setup_annotations_scope)

Another option is to leave dis and the bytecode alone, and write a test that asserts that something which is user-visible is fixed. How did you find this problem? A traceback pointed to the wrong place?

Another option is to leave dis and the bytecode alone, and write a test that asserts that something which is user-visible is fixed. How did you find this problem? A traceback pointed to the wrong place?

This issue appears to stem from an implementation choice in a downstream repository. For full context, you can refer to the issue #135700 linked in the PR.

My understanding is that the downstream project expects the position information for bytecode within setup_annotations_scope to have semantics similar to a RESUME instruction. Their reasoning seems to be that this bytecode is synthetic—it's generated by the compiler and isn't tied to any specific line of user source code, so there's no obvious "correct" location for it to point to.

Ultimately, this doesn't seem to be a factual bug, but rather a disagreement or ambiguity in the semantic interpretation of this position information.

Maybe @JelleZijlstra has an idea how to write a meaningful test for this.

I don't have much of an intuition on how line numbers should work for synthetic code.

@15r10nk are you able to suggest a meaningful unit test for this? Alternatively, can you confirm that the PR resolved the issue?

AndPuQing requested review from markshannon and iritkatriel as code owners June 22, 2025 10:45

bedevere-app bot added the awaiting review label Jun 22, 2025

bedevere-app bot mentioned this pull request Jun 22, 2025

some instructions in __annotate__ have incorrect code positions #135700

Open

AndPuQing and others added 2 commits June 22, 2025 18:48

fix instructions in __annotate__ have incorrect code positions

983ae79

📜🤖 Added by blurb_it.

dd3bea2

sobolevn reviewed Jun 22, 2025

View reviewed changes

Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issue-135700.0qdtCl.rst Outdated Show resolved Hide resolved

Update Misc/NEWS.d/next/Core_and_Builtins/2025-06-22-10-47-27.gh-issu…

c2f0a4d

…e-135700.0qdtCl.rst Co-authored-by: sobolevn <[email protected]>

JelleZijlstra added the needs backport to 3.14 bugs and security fixes label Jun 22, 2025

add unit tests

b79060e

sobolevn reviewed Jun 22, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

fix test

2201864

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

AndPuQing force-pushed the fix-issue-135700 branch from 9045ebe to aed06b7 Compare June 23, 2025 10:56

fix test

aed06b7

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

iritkatriel reviewed Jun 23, 2025

View reviewed changes

Lib/test/test_dis.py Outdated Show resolved Hide resolved

iritkatriel reviewed Jun 23, 2025

View reviewed changes

AndPuQing and others added 3 commits June 24, 2025 14:34

Update Lib/test/test_dis.py

d85247e

Co-authored-by: Irit Katriel <[email protected]>

Apply suggestions from code review

eca9248

Co-authored-by: Irit Katriel <[email protected]>

fix test

e14cb29

iritkatriel reviewed Jun 24, 2025

View reviewed changes

Uh oh!

gh-135700: Fix instructions in __annotate__ have incorrect code positions #135814

Are you sure you want to change the base?

gh-135700: Fix instructions in __annotate__ have incorrect code positions #135814

Uh oh!

Conversation

AndPuQing commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Jun 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndPuQing Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gh-135700: Fix instructions in annotate have incorrect code positions #135814

gh-135700: Fix instructions in annotate have incorrect code positions #135814

AndPuQing commented Jun 22, 2025 •

edited

Loading

AndPuQing Jun 24, 2025 •

edited

Loading