GH-148726: Forward-port generational GC. (GH-148746)

The replaces the incremental GC with a forward port (from 3.13) of the generational GC.

Co-Authored-By: Neil Schemenauer <nas@arctrix.com>
Co-Authored-By: Zanie Blue <contact@zanie.dev>
Co-Authored-By: Sergey Miryanov <sergey.miryanov@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Author: 4 people

Doc/library/gc.rst

@@ -37,18 +37,11 @@ The :mod:`!gc` module provides the following functions:
 
 .. function:: collect(generation=2)
 
-   Perform a collection.  The optional argument *generation*
+   With no arguments, run a full collection.  The optional argument *generation*
    may be an integer specifying which generation to collect (from 0 to 2).  A
    :exc:`ValueError` is raised if the generation number is invalid. The sum of
    collected objects and uncollectable objects is returned.
 
-   Calling ``gc.collect(0)`` will perform a GC collection on the young generation.
-
-   Calling ``gc.collect(1)`` will perform a GC collection on the young generation
-   and an increment of the old generation.
-
-   Calling ``gc.collect(2)`` or ``gc.collect()`` performs a full collection
-
    The free lists maintained for a number of built-in types are cleared
    whenever a full collection or collection of the highest generation (2)
    is run.  Not all items in some free lists may be freed due to the
@@ -60,6 +53,9 @@ The :mod:`!gc` module provides the following functions:
    .. versionchanged:: 3.14
       ``generation=1`` performs an increment of collection.
 
+   .. versionchanged:: 3.14.5
+      ``generation=1`` performs collection of the middle generation.
+
 
 .. function:: set_debug(flags)
 
@@ -75,20 +71,19 @@ The :mod:`!gc` module provides the following functions:
 
 .. function:: get_objects(generation=None)
 
-
    Returns a list of all objects tracked by the collector, excluding the list
-   returned. If *generation* is not ``None``, return only the objects as follows:
-
-   * 0: All objects in the young generation
-   * 1: No objects, as there is no generation 1 (as of Python 3.14)
-   * 2: All objects in the old generation
+   returned. If *generation* is not ``None``, return only the objects tracked by
+   the collector that are in that generation.
 
    .. versionchanged:: 3.8
       New *generation* parameter.
 
    .. versionchanged:: 3.14
       Generation 1 is removed
 
+   .. versionchanged:: 3.14.5
+      Generation 1 is reintroduced to maintain GC behavior from 3.13.
+
    .. audit-event:: gc.get_objects generation gc.get_objects
 
 .. function:: get_stats()
@@ -124,33 +119,33 @@ The :mod:`!gc` module provides the following functions:
    Set the garbage collection thresholds (the collection frequency). Setting
    *threshold0* to zero disables collection.
 
-   The GC classifies objects into two generations depending on whether they have
-   survived a collection. New objects are placed in the young generation. If an
-   object survives a collection it is moved into the old generation.
-
-   In order to decide when to run, the collector keeps track of the number of object
+   The GC classifies objects into three generations depending on how many
+   collection sweeps they have survived.  New objects are placed in the youngest
+   generation (generation ``0``).  If an object survives a collection it is moved
+   into the next older generation.  Since generation ``2`` is the oldest
+   generation, objects in that generation remain there after a collection.  In
+   order to decide when to run, the collector keeps track of the number object
    allocations and deallocations since the last collection.  When the number of
    allocations minus the number of deallocations exceeds *threshold0*, collection
-   starts. For each collection, all the objects in the young generation and some
-   fraction of the old generation is collected.
+   starts.  Initially only generation ``0`` is examined.  If generation ``0`` has
+   been examined more than *threshold1* times since generation ``1`` has been
+   examined, then generation ``1`` is examined as well.
+   With the third generation, things are a bit more complicated,
+   see `Collecting the oldest generation <https://github.com/python/cpython/blob/ff0ef0a54bef26fc507fbf9b7a6009eb7d3f17f5/InternalDocs/garbage_collector.md#collecting-the-oldest-generation>`_ for more information.
 
    In the free-threaded build, the increase in process memory usage is also
    checked before running the collector.  If the memory usage has not increased
    by 10% since the last collection and the net number of object allocations
    has not exceeded 40 times *threshold0*, the collection is not run.
 
-   The fraction of the old generation that is collected is **inversely** proportional
-   to *threshold1*. The larger *threshold1* is, the slower objects in the old generation
-   are collected.
-   For the default value of 10, 1% of the old generation is scanned during each collection.
-
-   *threshold2* is ignored.
-
-   See `Garbage collector design <https://devguide.python.org/garbage_collector>`_ for more information.
+   See `Garbage collector design <https://github.com/python/cpython/blob/3.15/InternalDocs/garbage_collector.md>`_ for more information.
 
    .. versionchanged:: 3.14
       *threshold2* is ignored
 
+   .. versionchanged:: 3.14.5
+      *threshold2* is restored to match Python 3.13 behavior.
+
 
 .. function:: get_count()
 

Include/internal/pycore_gc.h

@@ -118,21 +118,6 @@ static inline void _PyObject_GC_SET_SHARED(PyObject *op) {
 /* Bit 1 is set when the object is in generation which is GCed currently. */
 #define _PyGC_PREV_MASK_COLLECTING ((uintptr_t)2)
 
-/* Bit 0 in _gc_next is the old space bit.
- * It is set as follows:
- * Young: gcstate->visited_space
- * old[0]: 0
- * old[1]: 1
- * permanent: 0
- *
- * During a collection all objects handled should have the bit set to
- * gcstate->visited_space, as objects are moved from the young gen
- * and the increment into old[gcstate->visited_space].
- * When object are moved from the pending space, old[gcstate->visited_space^1]
- * into the increment, the old space bit is flipped.
-*/
-#define _PyGC_NEXT_MASK_OLD_SPACE_1    1
-
 #define _PyGC_PREV_SHIFT           2
 #define _PyGC_PREV_MASK            (((uintptr_t) -1) << _PyGC_PREV_SHIFT)
 
@@ -159,13 +144,11 @@ typedef enum {
 // Lowest bit of _gc_next is used for flags only in GC.
 // But it is always 0 for normal code.
 static inline PyGC_Head* _PyGCHead_NEXT(PyGC_Head *gc) {
-    uintptr_t next = gc->_gc_next & _PyGC_PREV_MASK;
+    uintptr_t next = gc->_gc_next;
     return (PyGC_Head*)next;
 }
 static inline void _PyGCHead_SET_NEXT(PyGC_Head *gc, PyGC_Head *next) {
-    uintptr_t unext = (uintptr_t)next;
-    assert((unext & ~_PyGC_PREV_MASK) == 0);
-    gc->_gc_next = (gc->_gc_next & ~_PyGC_PREV_MASK) | unext;
+    gc->_gc_next = (uintptr_t)next;
 }
 
 // Lowest two bits of _gc_prev is used for _PyGC_PREV_MASK_* flags.
@@ -207,10 +190,6 @@ static inline void _PyGC_CLEAR_FINALIZED(PyObject *op) {
 
 extern void _Py_ScheduleGC(PyThreadState *tstate);
 
-#ifndef Py_GIL_DISABLED
-extern void _Py_TriggerGC(struct _gc_runtime_state *gcstate);
-#endif
-
 
 /* Tell the GC to track this object.
  *
@@ -220,7 +199,7 @@ extern void _Py_TriggerGC(struct _gc_runtime_state *gcstate);
  * ob_traverse method.
  *
  * Internal note: interp->gc.generation0->_gc_prev doesn't have any bit flags
- * because it's not object header.  So we don't use _PyGCHead_PREV() and
+ * because it's not an object header. So we don't use _PyGCHead_PREV() and
  * _PyGCHead_SET_PREV() for it to avoid unnecessary bitwise operations.
  *
  * See also the public PyObject_GC_Track() function.
@@ -244,19 +223,12 @@ static inline void _PyObject_GC_TRACK(
                           "object is in generation which is garbage collected",
                           filename, lineno, __func__);
 
-    struct _gc_runtime_state *gcstate = &_PyInterpreterState_GET()->gc;
-    PyGC_Head *generation0 = &gcstate->young.head;
+    PyGC_Head *generation0 = _PyInterpreterState_GET()->gc.generation0;
     PyGC_Head *last = (PyGC_Head*)(generation0->_gc_prev);
     _PyGCHead_SET_NEXT(last, gc);
     _PyGCHead_SET_PREV(gc, last);
-    uintptr_t not_visited = 1 ^ gcstate->visited_space;
-    gc->_gc_next = ((uintptr_t)generation0) | not_visited;
+    _PyGCHead_SET_NEXT(gc, generation0);
     generation0->_gc_prev = (uintptr_t)gc;
-    gcstate->young.count++; /* number of tracked GC objects */
-    gcstate->heap_size++;
-    if (gcstate->young.count > gcstate->young.threshold) {
-        _Py_TriggerGC(gcstate);
-    }
 #endif
 }
 
@@ -291,11 +263,6 @@ static inline void _PyObject_GC_UNTRACK(
     _PyGCHead_SET_PREV(next, prev);
     gc->_gc_next = 0;
     gc->_gc_prev &= _PyGC_PREV_MASK_FINALIZED;
-    struct _gc_runtime_state *gcstate = &_PyInterpreterState_GET()->gc;
-    if (gcstate->young.count > 0) {
-        gcstate->young.count--;
-    }
-    gcstate->heap_size--;
 #endif
 }
 

Include/internal/pycore_interp_structs.h

@@ -181,29 +181,14 @@ struct gc_generation {
 struct gc_generation_stats {
     PyTime_t ts_start;
     PyTime_t ts_stop;
-
-    /* heap_size on the start of the collection */
-    Py_ssize_t heap_size;
-
-    /* work_to_do on the start of the collection */
-    Py_ssize_t work_to_do;
-
     /* total number of collections */
     Py_ssize_t collections;
-
-    /* total number of visited objects */
-    Py_ssize_t object_visits;
-
     /* total number of collected objects */
     Py_ssize_t collected;
     /* total number of uncollectable objects (put into gc.garbage) */
     Py_ssize_t uncollectable;
     // Total number of objects considered for collection and traversed:
     Py_ssize_t candidates;
-
-    Py_ssize_t objects_transitively_reachable;
-    Py_ssize_t objects_not_transitively_reachable;
-
     // Total duration of the collection in seconds:
     double duration;
 };
@@ -225,11 +210,6 @@ struct gc_old_stats_buffer {
     int8_t index;
 };
 
-enum _GCPhase {
-    GC_PHASE_MARK = 0,
-    GC_PHASE_COLLECT = 1
-};
-
 /* If we change this, we need to change the default value in the
    signature of gc.collect and change the size of PyStats.gc_stats */
 #define NUM_GENERATIONS 3
@@ -244,8 +224,13 @@ struct _gc_runtime_state {
     int enabled;
     int debug;
     /* linked lists of container objects */
+#ifndef Py_GIL_DISABLED
+    struct gc_generation generations[NUM_GENERATIONS];
+    PyGC_Head *generation0;
+#else
     struct gc_generation young;
     struct gc_generation old[2];
+#endif
     /* a permanent generation which won't be collected */
     struct gc_generation permanent_generation;
     struct gc_stats *generation_stats;
@@ -259,13 +244,6 @@ struct _gc_runtime_state {
     /* a list of callbacks to be invoked when collection is performed */
     PyObject *callbacks;
 
-    Py_ssize_t heap_size;
-    Py_ssize_t work_to_do;
-    /* Which of the old spaces is the visited space */
-    int visited_space;
-    int phase;
-
-#ifdef Py_GIL_DISABLED
     /* This is the number of objects that survived the last full
        collection. It approximates the number of long lived objects
        tracked by the GC.
@@ -278,6 +256,7 @@ struct _gc_runtime_state {
        the first time. */
     Py_ssize_t long_lived_pending;
 
+#ifdef Py_GIL_DISABLED
     /* True if gc.freeze() has been used. */
     int freeze_active;
 
@@ -293,6 +272,22 @@ struct _gc_runtime_state {
 #endif
 };
 
+#ifndef Py_GIL_DISABLED
+#define GC_GENERATION_INIT \
+    .generations = { \
+        { .threshold = 2000, }, \
+        { .threshold = 10, }, \
+        { .threshold = 10, }, \
+    },
+#else
+#define GC_GENERATION_INIT \
+    .young = { .threshold = 2000, }, \
+    .old = { \
+        { .threshold = 10, }, \
+        { .threshold = 10, }, \
+    },
+#endif
+
 #include "pycore_gil.h"           // struct _gil_runtime_state
 
 /**** Import ********/

Include/internal/pycore_runtime_init.h

@@ -130,13 +130,7 @@ extern PyTypeObject _PyExc_MemoryError;
         }, \
         .gc = { \
             .enabled = 1, \
-            .young = { .threshold = 2000, }, \
-            .old = { \
-                { .threshold = 10, }, \
-                { .threshold = 0, }, \
-            }, \
-            .work_to_do = -5000, \
-            .phase = GC_PHASE_MARK, \
+            GC_GENERATION_INIT \
         }, \
         .qsbr = { \
             .wr_seq = QSBR_INITIAL, \