Skip to content

gh-130851: Don't crash when deduping unusual code constants. #130853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 5, 2025

Conversation

colesbury
Copy link
Contributor

@colesbury colesbury commented Mar 4, 2025

The bytecode compiler only generates a few different types of constants, like str, int, tuple, slices, etc. Users can construct code objects with various unusual constants, including ones that are not hashable or not even constant.

The free threaded build previously crashed with a fatal error when confronted with these constants. Instead, treat distinct objects of otherwise unhandled types as not equal for the purposes of deduplication.

The bytecode compiler only generates a few different types of constants,
like str, int, tuple, slices, etc. Users can construct code objects with
various unusual constants, including ones that are not hashable or not
even constant.

The free threaded build previously crashed with a fatal error when
confronted with these constants. Instead, treat distinct objects of
otherwise unhandled types as not equal for the purposes of deduplication.
// provide their own constants, when constructing code objects via
// types.CodeType(). If the user-provided constant is unhashable, we
// use the memory address of the object as a fallback hash value.
PyErr_Clear();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm of two minds on whether this should warn or not... I guess that's part of a bigger discussion of whether we should try to support this pretty absurd thing of stuffing arbitrary things in code objects. If we don't want to support it, this should probably warn?

@Yhg1s Yhg1s merged commit 2905690 into python:main Mar 5, 2025
52 checks passed
@miss-islington-app
Copy link

Thanks @colesbury for the PR, and @Yhg1s for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Mar 5, 2025
…thonGH-130853)

The bytecode compiler only generates a few different types of constants,
like str, int, tuple, slices, etc. Users can construct code objects with
various unusual constants, including ones that are not hashable or not
even constant.

The free threaded build previously crashed with a fatal error when
confronted with these constants. Instead, treat distinct objects of
otherwise unhandled types as not equal for the purposes of deduplication.
(cherry picked from commit 2905690)

Co-authored-by: Sam Gross <[email protected]>
@bedevere-app
Copy link

bedevere-app bot commented Mar 5, 2025

GH-130880 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Mar 5, 2025
colesbury added a commit that referenced this pull request Mar 5, 2025
…H-130853) (#130880)

The bytecode compiler only generates a few different types of constants,
like str, int, tuple, slices, etc. Users can construct code objects with
various unusual constants, including ones that are not hashable or not
even constant.

The free threaded build previously crashed with a fatal error when
confronted with these constants. Instead, treat distinct objects of
otherwise unhandled types as not equal for the purposes of deduplication.
(cherry picked from commit 2905690)

Co-authored-by: Sam Gross <[email protected]>
@colesbury
Copy link
Contributor Author

I think I broke the nogil refleaks buildbot, probably because of the immortalization in the added test:

https://buildbot.python.org/#/builders/1610/builds/966

I'll submit a follow up PR.

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable Refleaks 3.13 (tier-2) has failed when building commit e285232.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1512/builds/406) and take a look at the build logs.
  4. Check if the failure is related to this commit (e285232) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1512/builds/406

Failed tests:

  • test_perf_profiler

Failed subtests:

  • test_python_calls_appear_in_the_stack_if_perf_activated - test.test_perf_profiler.TestPerfProfilerWithDwarf.test_python_calls_appear_in_the_stack_if_perf_activated

Summary of the results of the build (if available):

==

Click to see traceback logs
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/Lib/test/test_perf_profiler.py", line 356, in test_python_calls_appear_in_the_stack_if_perf_activated
    self.assertIn(f"py::foo:{script}", stdout)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'py::foo:/tmp/test_python_iqd2xut9/tmp6h2y45yl/perftest.py' not found in 'python 3629728 87755.610646:          1 cycles:Pu: \n\t    ffffb6ca3ac0 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)\n\npython 3629728 87755.610672:          1 cycles:Pu: \n\tffffc95efcdffc78 [unknown] ([unknown])\n\tffffc95efce0049c [unknown] ([unknown])\n\tffffc95efb9a15e4 [unknown] ([unknown])\n\t    ffffb6ca3ac0 _start+0x0 (/usr/lib/ld-linux-aarch64.so.1)\n\npython 3629728 87755.611124:          1 cycles:Pu: \n\t    ffffb6c8f82c _dl_map_object_from_fd+0xb4c (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c90133 _dl_map_object+0x1e7 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8b5bf openaux+0x3f (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8a303 _dl_catch_exception+0x63 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8bb33 _dl_map_object_deps+0x553 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6ca119f dl_main+0x139f (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c9e5ff _dl_sysdep_start+0x1df (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c9fb17 _dl_start_final+0x5ab (inlined)\n\t    ffffb6c9fb17 _dl_start+0x5ab (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6ca3ad3 _start+0x13 (/usr/lib/ld-linux-aarch64.so.1)\n\npython 3629728 87755.611148:        302 cycles:Pu: \n\tffffc95efcdfff10 [unknown] ([unknown])\n\tffffc95efce004cc [unknown] ([unknown])\n\tffffc95efb9a15e4 [unknown] ([unknown])\n\t    ffffb6ca5198 __GI___close_nocancel+0x18 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8f893 _dl_map_object_from_fd+0xbb3 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c90133 _dl_map_object+0x1e7 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8b5bf openaux+0x3f (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8a303 _dl_catch_exception+0x63 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c8bb33 _dl_map_object_deps+0x553 (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6ca119f dl_main+0x139f (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c9e5ff _dl_sysdep_start+0x1df (/usr/lib/ld-linux-aarch64.so.1)\n\t    ffffb6c9fb17 _dl_start_final+0x5ab (inlined)\n\t    ff
ldbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          55ba77 unicode_hash+0x23 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          50723b PyObject_Hash+0x1f (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          67b987 traceback_hash+0x3b (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          67d1e3 _PyTraceMalloc_Init+0x113 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          652f13 pycore_create_interpreter+0x207 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65384b pyinit_config+0x6f (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65af03 pyinit_core+0xdb (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65afdb Py_InitializeFromConfig+0x97 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          68b133 pymain_init+0xff (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          68b1f7 pymain_main+0xf (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          68b28b Py_BytesMain+0x27 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          41ddf7 main+0xb (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t    ffffb6a3625b __libc_start_call_main+0x7b (/usr/lib64/libc.so.6)\n\t    ffffb6a3633b __libc_start_main@GLIBC_2.17+0x9b (inlined)\n\t          41dcef _start+0x2f (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\npython 3629728 87755.613195:     114330 cycles:Pu: \n\t          506f64 reftotal_add+0x10 (inlined)\n\t          506f64 _Py_DecRefTotal+0x10 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          55d463 intern_common+0x483 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          58e9df _PyUnicode_InternMortal+0x1b (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          58ea9b PyUnicode_InternFromString+0x1f (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          4bb4a7 descr_new+0x3f (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          4bc53f PyDescr_NewMethod+0xaf (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          53a1c3 type_add_method+0x103 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          53a26b type_add_methods+0x2b (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          53a7d3 type_ready_fill_dict+0x1f (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          54761f type_ready+0x93 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          5477b3 init_static_type+0xab (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          547993 _PyStaticType_InitBuiltin+0x23 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          5076e7 _PyTypes_InitTypes+0xa7 (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          653077 pycore_init_types+0x1b (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65368f pycore_interp_init+0x10b (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65388b pyinit_config+0xaf (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65af03 pyinit_core+0xdb (/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/python)\n\t          65afdb Py_I


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.13.cstratak-fedora-stable-aarch64.refleak/build/Lib/test/test_perf_profiler.py", line 354, in test_python_calls_appear_in_the_stack_if_perf_activated
    self.assertEqual(stderr, "")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
AssertionError: 'Warning:\nProcessed 680 events and lost 1[34 chars]\n\n' != ''
- Warning:
- Processed 680 events and lost 1 chunks!
- 
- Check IO/CPU overload!
- 

@colesbury colesbury deleted the gh-130851-code-consts branch March 19, 2025 01:56
seehwan pushed a commit to seehwan/cpython that referenced this pull request Apr 16, 2025
…thon#130853)

The bytecode compiler only generates a few different types of constants,
like str, int, tuple, slices, etc. Users can construct code objects with
various unusual constants, including ones that are not hashable or not
even constant.

The free threaded build previously crashed with a fatal error when
confronted with these constants. Instead, treat distinct objects of
otherwise unhandled types as not equal for the purposes of deduplication.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants