From fd400f0a302c8797f161738cee8d624ed480b9c5 Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Thu, 4 Dec 2025 16:21:05 +0100 Subject: [PATCH 1/7] gh-142518: Document thread-safety guarantees of list operations --- Doc/library/stdtypes.rst | 43 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 086da1a705c30f..7482b4d303c6b4 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1369,6 +1369,49 @@ Lists are mutable sequences, typically used to store collections of homogeneous items (where the precise degree of similarity will vary by application). +.. note:: + + Individual operations on :class:`list` instances such as + :meth:`~list.append`, :meth:`~list.pop`, ``lst[i] = x``, and ``x = lst[i]`` + are atomic and will not corrupt the list or crash when called concurrently + from multiple threads: + + .. code-block:: + :class: good + + # These operations are safe to call concurrently from multiple threads + lst.append(item) # atomically appends item + lst.pop() # atomically removes and returns last item + lst[i] = value # atomically replaces item at index i + item = lst[i] # atomically retrieves item at index i + + One exception to this rule is :meth:`~list.extend`. Its atomicity + guarantees depend on the iterable being passed. When the iterable is + a :class:`list`, a :class:`tuple`, a :class:`set`, a :class:`frozenset`, + a :class:`dict` or a :ref:`dictionary view object `, the + operation is atomic. Otherwise, an iterator is created which can be + concurrently modified by another thread. + + Operations that involve multiple accesses, as well as iteration, are not + atomic. For example, the following patterns are not thread-safe: + + .. code-block:: + :class: bad + + # NOT atomic: read-modify-write + lst[i] = lst[i] + 1 + + # NOT atomic: check-then-act + if lst: + item = lst.pop() + + # NOT thread-safe: iteration while modifying + for item in lst: + process(item) # another thread may modify lst + + Consider external synchronization when sharing :class:`list` instances + across threads. See :ref:`freethreading-python-howto` for more information. + .. class:: list(iterable=(), /) Lists may be constructed in several ways: From 38483c98f094f4e547dcee3ba74b541c3e726b17 Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Thu, 11 Dec 2025 14:37:57 +0100 Subject: [PATCH 2/7] Address feedback; add more details --- Doc/library/stdtypes.rst | 113 ++++++++++++++++++++++++--------------- 1 file changed, 70 insertions(+), 43 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 7482b4d303c6b4..bb3f574e873d4e 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1369,49 +1369,6 @@ Lists are mutable sequences, typically used to store collections of homogeneous items (where the precise degree of similarity will vary by application). -.. note:: - - Individual operations on :class:`list` instances such as - :meth:`~list.append`, :meth:`~list.pop`, ``lst[i] = x``, and ``x = lst[i]`` - are atomic and will not corrupt the list or crash when called concurrently - from multiple threads: - - .. code-block:: - :class: good - - # These operations are safe to call concurrently from multiple threads - lst.append(item) # atomically appends item - lst.pop() # atomically removes and returns last item - lst[i] = value # atomically replaces item at index i - item = lst[i] # atomically retrieves item at index i - - One exception to this rule is :meth:`~list.extend`. Its atomicity - guarantees depend on the iterable being passed. When the iterable is - a :class:`list`, a :class:`tuple`, a :class:`set`, a :class:`frozenset`, - a :class:`dict` or a :ref:`dictionary view object `, the - operation is atomic. Otherwise, an iterator is created which can be - concurrently modified by another thread. - - Operations that involve multiple accesses, as well as iteration, are not - atomic. For example, the following patterns are not thread-safe: - - .. code-block:: - :class: bad - - # NOT atomic: read-modify-write - lst[i] = lst[i] + 1 - - # NOT atomic: check-then-act - if lst: - item = lst.pop() - - # NOT thread-safe: iteration while modifying - for item in lst: - process(item) # another thread may modify lst - - Consider external synchronization when sharing :class:`list` instances - across threads. See :ref:`freethreading-python-howto` for more information. - .. class:: list(iterable=(), /) Lists may be constructed in several ways: @@ -1479,6 +1436,76 @@ application). list appear empty for the duration, and raises :exc:`ValueError` if it can detect that the list has been mutated during a sort. +.. admonition:: Thread safety + + Most individual operations on :class:`list` instances are atomic: + + .. code-block:: + :class: good + + # The following operations are atomic + lst1 + lst2 # atomic concatenation of two lists + x * lst # atomic repeat of lst x times + item = lst[i] # atomically retrieves item at index i + lst[i] = value # atomically replaces item at index i + lst *= x # atomically extend the list x times + + # Calls to the following list methods are atomic + lst.clear() + lst.copy() + lst.append(item) + lst.insert(idx, item) + lst.pop(idx) + lst.remove(item) + lst.reverse() + lst.sort() + + The following operations/methods are not atomic: + + .. code-block:: + :class: maybe + + lst.index(item) + lst.count(item) + item in lst + + The :meth:`~list.index` and :meth:`~list.count` methods, and the ``in`` + operator, iterate the list without holding a lock. They are safe to call + concurrently but may return results affected by concurrent modifications. + + :meth:`~list.extend` is always atomic with respect to the target list. + However, the operation is fully atomic only when the iterable that's passed + to ``extend`` is a :class:`list`, a :class:`tuple`, a :class:`set`, a + :class:`frozenset`, a :class:`dict` or a + :ref:`dictionary view object `. Otherwise, an iterator is + created which can be concurrently modified by another thread. The same + applies to inplace concatenation of list with other iterables when using + ``lst += iterable``. + + Similarly, assigning to a list slice with ``lst[i:j] = obj`` is always + atomic with respect to the target list, but ``obj`` is only locked when it + is also a :class:`list`. + + Operations that involve multiple accesses, as well as iteration, are not + atomic. For example: + + .. code-block:: + :class: bad + + # NOT atomic: read-modify-write + lst[i] = lst[i] + 1 + + # NOT atomic: check-then-act + if lst: + item = lst.pop() + + # NOT thread-safe: iteration while modifying + for item in lst: + process(item) # another thread may modify lst + + Consider external synchronization when sharing :class:`list` instances + across threads. See :ref:`freethreading-python-howto` for more information. + .. _typesseq-tuple: From 22928961a9d2ee7d0123ba4dd9f8c7c575af6636 Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Thu, 11 Dec 2025 14:43:49 +0100 Subject: [PATCH 3/7] Add everything to code blocks --- Doc/library/stdtypes.rst | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 38a6221c81252d..f477f8ac3ee80c 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1460,7 +1460,7 @@ application). lst.reverse() lst.sort() - The following operations/methods are not atomic: + The following operations/methods are not fully atomic: .. code-block:: :class: maybe @@ -1469,6 +1469,11 @@ application). lst.count(item) item in lst + lst.extend(iterable) + lst += iterable + + lst[i:j] = iterable + The :meth:`~list.index` and :meth:`~list.count` methods, and the ``in`` operator, iterate the list without holding a lock. They are safe to call concurrently but may return results affected by concurrent modifications. @@ -1482,9 +1487,9 @@ application). applies to inplace concatenation of list with other iterables when using ``lst += iterable``. - Similarly, assigning to a list slice with ``lst[i:j] = obj`` is always - atomic with respect to the target list, but ``obj`` is only locked when it - is also a :class:`list`. + Similarly, assigning to a list slice with ``lst[i:j] = iterable`` is always + atomic with respect to the target list, but ``iterable`` is only locked when + it is also a :class:`list`. Operations that involve multiple accesses, as well as iteration, are not atomic. For example: From 2b9e711b248e719873217173694d599d59aa60d6 Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Thu, 11 Dec 2025 16:59:56 +0100 Subject: [PATCH 4/7] Improve wording around atomicity; specify exact types --- Doc/library/stdtypes.rst | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index f477f8ac3ee80c..63bed7adb32877 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1475,21 +1475,20 @@ application). lst[i:j] = iterable The :meth:`~list.index` and :meth:`~list.count` methods, and the ``in`` - operator, iterate the list without holding a lock. They are safe to call + operator, iterate the list without holding a lock. They are safe to call concurrently but may return results affected by concurrent modifications. - :meth:`~list.extend` is always atomic with respect to the target list. - However, the operation is fully atomic only when the iterable that's passed - to ``extend`` is a :class:`list`, a :class:`tuple`, a :class:`set`, a - :class:`frozenset`, a :class:`dict` or a - :ref:`dictionary view object `. Otherwise, an iterator is - created which can be concurrently modified by another thread. The same - applies to inplace concatenation of list with other iterables when using - ``lst += iterable``. - - Similarly, assigning to a list slice with ``lst[i:j] = iterable`` is always - atomic with respect to the target list, but ``iterable`` is only locked when - it is also a :class:`list`. + :meth:`~list.extend` is safe to call from multiple threads. However, the + operation is fully atomic only when the iterable that's passed to ``extend`` + is a :class:`list`, a :class:`tuple`, a :class:`set`, a :class:`frozenset`, + a :class:`dict` or a :ref:`dictionary view object ` (but not + their subclasses). Otherwise, an iterator is created which can be + concurrently modified by another thread. The same applies to inplace + concatenation of list with other iterables when using ``lst += iterable``. + + Similarly, assigning to a list slice with ``lst[i:j] = iterable`` is safe + to call from multiple threads, but ``iterable`` is only locked when it is + also a :class:`list` (but not its subclasses). Operations that involve multiple accesses, as well as iteration, are not atomic. For example: From 688b25a2b0512429d93099a2b8311de07af4188c Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Wed, 17 Dec 2025 13:53:18 +0100 Subject: [PATCH 5/7] Better explain lock-free and atomicity --- Doc/library/stdtypes.rst | 99 ++++++++++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 35 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 63bed7adb32877..76ffdeb90dfd75 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1438,59 +1438,88 @@ application). .. admonition:: Thread safety - Most individual operations on :class:`list` instances are atomic: + Reading a single element from a :class:`list` is + :term:`atomic `: .. code-block:: - :class: good + :class: green + + lst[i] # list.__getitem__ - # The following operations are atomic - lst1 + lst2 # atomic concatenation of two lists - x * lst # atomic repeat of lst x times - item = lst[i] # atomically retrieves item at index i - lst[i] = value # atomically replaces item at index i - lst *= x # atomically extend the list x times - - # Calls to the following list methods are atomic - lst.clear() - lst.copy() - lst.append(item) - lst.insert(idx, item) - lst.pop(idx) - lst.remove(item) - lst.reverse() - lst.sort() - - The following operations/methods are not fully atomic: + The following methods traverse the list and use :term:`atomic ` + reads of each item to perform their function. That means that they may + return results affected by concurrent modifications: .. code-block:: :class: maybe + item in lst lst.index(item) lst.count(item) - item in lst - lst.extend(iterable) - lst += iterable + All of the above methods/operations are also lock-free. They do not block + concurrent modifications. Other operations that hold a lock will not block + these from observing intermediate states. + + All other operations from here on block using the per-object lock. + + Writing a single item via ``lst[i] = x`` is safe to call from multiple + threads and will not corrupt the list. + + The following operations return new objects and appear + :term:`atomic ` to other threads: + + .. code-block:: + :class: good + + lst1 + lst2 # concatenates two lists into a new list + x * lst # repeats lst x times into a new list + lst.copy() # returns a shallow copy of the list + + Methods that only operate on a single elements with no shifting required are + :term:`atomic `: + + .. code-block:: + :class: good + + lst.append(x) # append to the end of the list, no shifting required + lst.pop() # pop element from the end of the list, no shifting required + + The :meth:`~list.clear` method is also :term:`atomic `. + Other threads cannot observe elements being removed. + + The :meth:`~list.sort` method is not :term:`atomic `. + Other threads cannot observe intermediate states during sorting, but the + list appears empty for the duration of the sort. + + The following operations may allow lock-free operations to observe + intermediate states since they modify multiple elements in place: + + .. code-block:: + :class: maybe - lst[i:j] = iterable + lst.insert(idx, item) # shifts elements + lst.pop(idx) # idx not at the end of the list, shifts elements + lst *= x # copies elements in place - The :meth:`~list.index` and :meth:`~list.count` methods, and the ``in`` - operator, iterate the list without holding a lock. They are safe to call - concurrently but may return results affected by concurrent modifications. + The :meth:`~list.remove` method may allow concurrent modifications since + element comparison may execute arbitrary Python code (via + :meth:`~object.__eq__`). - :meth:`~list.extend` is safe to call from multiple threads. However, the - operation is fully atomic only when the iterable that's passed to ``extend`` - is a :class:`list`, a :class:`tuple`, a :class:`set`, a :class:`frozenset`, - a :class:`dict` or a :ref:`dictionary view object ` (but not - their subclasses). Otherwise, an iterator is created which can be - concurrently modified by another thread. The same applies to inplace - concatenation of list with other iterables when using ``lst += iterable``. + :meth:`~list.extend` is safe to call from multiple threads. However, its + guarantees depend on the iterable passed to it. If it is a :class:`list`, a + :class:`tuple`, a :class:`set`, a :class:`frozenset`, a :class:`dict` or a + :ref:`dictionary view object ` (but not their subclasses), the + ``extend`` operation is safe from concurrent modifications to the iterable. + Otherwise, an iterator is created which can be concurrently modified by + another thread. The same applies to inplace concatenation of a list with + other iterables when using ``lst += iterable``. Similarly, assigning to a list slice with ``lst[i:j] = iterable`` is safe to call from multiple threads, but ``iterable`` is only locked when it is also a :class:`list` (but not its subclasses). - Operations that involve multiple accesses, as well as iteration, are not + Operations that involve multiple accesses, as well as iteration, are never atomic. For example: .. code-block:: From 11e807236914b63b54b5bf6f24a58a95363ead77 Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Thu, 18 Dec 2025 18:02:06 +0200 Subject: [PATCH 6/7] Use correct ref for atomic --- Doc/library/stdtypes.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 76ffdeb90dfd75..1cdbeeb7c2d74a 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1467,7 +1467,7 @@ application). threads and will not corrupt the list. The following operations return new objects and appear - :term:`atomic ` to other threads: + :term:`atomic ` to other threads: .. code-block:: :class: good @@ -1477,7 +1477,7 @@ application). lst.copy() # returns a shallow copy of the list Methods that only operate on a single elements with no shifting required are - :term:`atomic `: + :term:`atomic `: .. code-block:: :class: good @@ -1485,7 +1485,7 @@ application). lst.append(x) # append to the end of the list, no shifting required lst.pop() # pop element from the end of the list, no shifting required - The :meth:`~list.clear` method is also :term:`atomic `. + The :meth:`~list.clear` method is also :term:`atomic `. Other threads cannot observe elements being removed. The :meth:`~list.sort` method is not :term:`atomic `. From f09430bdd42df349e1678a433799e14ffb8c1d0f Mon Sep 17 00:00:00 2001 From: Lysandros Nikolaou Date: Thu, 18 Dec 2025 18:06:55 +0200 Subject: [PATCH 7/7] Fix lint error --- Doc/library/stdtypes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 0442bceca66f3e..0ecb0627694626 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1460,7 +1460,7 @@ application). All of the above methods/operations are also lock-free. They do not block concurrent modifications. Other operations that hold a lock will not block these from observing intermediate states. - + All other operations from here on block using the per-object lock. Writing a single item via ``lst[i] = x`` is safe to call from multiple