Skip to content

GH-124742: Add the PYTHON_GC_THRESHOLD environment variable. #124743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
+54 −0
Closed

GH-124742: Add the PYTHON_GC_THRESHOLD environment variable. #124743

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
11fdaee
Add the `PYTHON_GC_THRESHOLD` environment variable.
nascheme Sep 29, 2024
40be462
Repair doc markup.
nascheme Sep 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Doc/using/cmdline.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -976,6 +976,16 @@ conflict.
.. versionadded:: 3.4


.. envvar:: PYTHON_GC_THRESHOLD

Set the ``threshold0`` value for the garbage collector. This is the same as
calling ``gc.set_threshold(n)`` where ``n`` is value of the variable. Using
an empty value or the value ``default`` will cause the default threshold to
be used.

.. versionadded:: 3.13


.. envvar:: PYTHONMALLOC

Set the Python memory allocators and/or install debug hooks.
Expand Down
4 changes: 4 additions & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2024-09-28-20-01-40.gh-issue-124742.7kJx88.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Add the :envvar:`PYTHON_GC_THRESHOLD` environment variable. This can be used
to set the ``threshold0`` value for the garbage collector, similar to calling
``gc.set_threshold()``. If the value is empty or ``default``, the default
threshold value is used.
19 changes: 19 additions & 0 deletions Python/gc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,31 @@ _PyGC_InitState(GCState *gcstate)
#undef INIT_HEAD
}

static void
gc_set_threshold_from_env(PyInterpreterState *interp)
{
const PyConfig *config = _PyInterpreterState_GetConfig(interp);
const char *env = _Py_GetEnv(config->use_environment,
"PYTHON_GC_THRESHOLD");
if (env == NULL || strcmp(env, "default") == 0) {
return;
}
int threshold = -1;
if (_Py_str_to_int(env, &threshold) < 0) {
return; // parse failed, silently ignore
}
Comment on lines +175 to +181
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, why ignore the error? If a user accidentally sets the variable to something that doesn't work, then they might end up very confused as to why Python is ignoring their request.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping this change can be included in the 3.13 RC and so I wanted it as simple and robust as possible. In the 'main' branch, I plan to re-work it so it becomes part of the config structure and I would add an error raise there.

Maybe it is safe enough to raise an error here too. That would be consistent with the principle that "errors should not pass silently".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that making it return an error rather than nothing should still remain fairly simple, return will just become return _PyStatus_ERR("...") and whatnot.

I haven't paid too much attention to the incremental GC problem (#124567), but I'm assuming that this is a possible solution for it? That should help it go into the 3.13 release

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think this would be enough for us to feel safe about including incremental GC into 3.13. In retrospect, the incremental GC should have been opt-in for 3.13 and would become default in 3.14 if people had good experience with it.

The point of this env var is to increase the chances that people will test their programs with a higher threshold with 3.13 and then when 3.14 release nears, we would have a better idea about how "aggressive" the default GC tuning should be.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning an error _PyGC_Init results in this kind of error:

Fatal Python error: _PyGC_Init: Invalid PYTHON_GC_THRESHOLD value.
Python runtime state: preinitialized

Current thread 0x00007fe0505d6740 (most recent call first):
  <no Python frame>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense. We do the same for other env vars.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping this change can be included in the 3.13 RC and so I wanted it as simple and robust as possible.

Since we have already reached the feature-freeze stage and this is a new feature, it needs confirmation from @Yhg1s.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right. It's Thomas's call and I completely understand if he doesn't want it. There has already been a bit too much excitement with the RCs.

if (threshold > 0) {
interp->gc.young.threshold = threshold;
}
}

PyStatus
_PyGC_Init(PyInterpreterState *interp)
{
GCState *gcstate = &interp->gc;

gc_set_threshold_from_env(interp);

gcstate->garbage = PyList_New(0);
if (gcstate->garbage == NULL) {
return _PyStatus_NO_MEMORY();
Expand Down
19 changes: 19 additions & 0 deletions Python/gc_free_threading.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -838,12 +838,31 @@ _PyGC_InitState(GCState *gcstate)
gcstate->young.threshold = 2000;
}

static void
gc_set_threshold_from_env(PyInterpreterState *interp)
{
const PyConfig *config = _PyInterpreterState_GetConfig(interp);
const char *env = _Py_GetEnv(config->use_environment,
"PYTHON_GC_THRESHOLD");
if (env == NULL || strcmp(env, "default") == 0) {
return;
}
int threshold = -1;
if (_Py_str_to_int(env, &threshold) < 0) {
return; // parse failed, silently ignore
}
if (threshold > 0) {
interp->gc.young.threshold = threshold;
}
}

PyStatus
_PyGC_Init(PyInterpreterState *interp)
{
GCState *gcstate = &interp->gc;

gc_set_threshold_from_env(interp);

gcstate->garbage = PyList_New(0);
if (gcstate->garbage == NULL) {
return _PyStatus_NO_MEMORY();
Expand Down
2 changes: 2 additions & 0 deletions Python/initconfig.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,8 @@ static const char usage_envvars[] =
" on Python memory allocators. Use PYTHONMALLOC=debug to\n"
" install debug hooks.\n"
"PYTHONMALLOCSTATS: print memory allocator statistics\n"
"PYTHON_GC_THRESHOLD: set threshold0 for the garbage collector. This\n"
" threshold can also be set by gc.set_threshold().\n"
"PYTHONCOERCECLOCALE: if this variable is set to 0, it disables the locale\n"
" coercion behavior. Use PYTHONCOERCECLOCALE=warn to request\n"
" display of locale coercion and locale compatibility warnings\n"
Expand Down
Loading