Merge branch 'main' into os-NODEV

Author: serhiy-storchaka

Doc/c-api/bytes.rst

@@ -219,3 +219,162 @@ called with a non-bytes parameter.
    reallocation fails, the original bytes object at *\*bytes* is deallocated,
    *\*bytes* is set to ``NULL``, :exc:`MemoryError` is set, and ``-1`` is
    returned.
+
+PyBytesWriter
+-------------
+
+The :c:type:`PyBytesWriter` API can be used to create a Python :class:`bytes`
+object.
+
+.. versionadded:: next
+
+.. c:type:: PyBytesWriter
+
+   A bytes writer instance.
+
+   The API is **not thread safe**: a writer should only be used by a single
+   thread at the same time.
+
+   The instance must be destroyed by :c:func:`PyBytesWriter_Finish` on
+   success, or :c:func:`PyBytesWriter_Discard` on error.
+
+
+Create, Finish, Discard
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. c:function:: PyBytesWriter* PyBytesWriter_Create(Py_ssize_t size)
+
+   Create a :c:type:`PyBytesWriter` to write *size* bytes.
+
+   If *size* is greater than zero, allocate *size* bytes, and set the
+   writer size to *size*. The caller is responsible to write *size*
+   bytes using :c:func:`PyBytesWriter_GetData`.
+
+   On error, set an exception and return NULL.
+
+   *size* must be positive or zero.
+
+.. c:function:: PyObject* PyBytesWriter_Finish(PyBytesWriter *writer)
+
+   Finish a :c:type:`PyBytesWriter` created by
+   :c:func:`PyBytesWriter_Create`.
+
+   On success, return a Python :class:`bytes` object.
+   On error, set an exception and return ``NULL``.
+
+   The writer instance is invalid after the call in any case.
+   No API can be called on the writer after :c:func:`PyBytesWriter_Finish`.
+
+.. c:function:: PyObject* PyBytesWriter_FinishWithSize(PyBytesWriter *writer, Py_ssize_t size)
+
+   Similar to :c:func:`PyBytesWriter_Finish`, but resize the writer
+   to *size* bytes before creating the :class:`bytes` object.
+
+.. c:function:: PyObject* PyBytesWriter_FinishWithPointer(PyBytesWriter *writer, void *buf)
+
+   Similar to :c:func:`PyBytesWriter_Finish`, but resize the writer
+   using *buf* pointer before creating the :class:`bytes` object.
+
+   Set an exception and return ``NULL`` if *buf* pointer is outside the
+   internal buffer bounds.
+
+   Function pseudo-code::
+
+       Py_ssize_t size = (char*)buf - (char*)PyBytesWriter_GetData(writer);
+       return PyBytesWriter_FinishWithSize(writer, size);
+
+.. c:function:: void PyBytesWriter_Discard(PyBytesWriter *writer)
+
+   Discard a :c:type:`PyBytesWriter` created by :c:func:`PyBytesWriter_Create`.
+
+   Do nothing if *writer* is ``NULL``.
+
+   The writer instance is invalid after the call.
+   No API can be called on the writer after :c:func:`PyBytesWriter_Discard`.
+
+High-level API
+^^^^^^^^^^^^^^
+
+.. c:function:: int PyBytesWriter_WriteBytes(PyBytesWriter *writer, const void *bytes, Py_ssize_t size)
+
+   Grow the *writer* internal buffer by *size* bytes,
+   write *size* bytes of *bytes* at the *writer* end,
+   and add *size* to the *writer* size.
+
+   If *size* is equal to ``-1``, call ``strlen(bytes)`` to get the
+   string length.
+
+   On success, return ``0``.
+   On error, set an exception and return ``-1``.
+
+.. c:function:: int PyBytesWriter_Format(PyBytesWriter *writer, const char *format, ...)
+
+   Similar to :c:func:`PyBytes_FromFormat`, but write the output directly at
+   the writer end. Grow the writer internal buffer on demand. Then add the
+   written size to the writer size.
+
+   On success, return ``0``.
+   On error, set an exception and return ``-1``.
+
+
+Getters
+^^^^^^^
+
+.. c:function:: Py_ssize_t PyBytesWriter_GetSize(PyBytesWriter *writer)
+
+   Get the writer size.
+
+.. c:function:: void* PyBytesWriter_GetData(PyBytesWriter *writer)
+
+   Get the writer data: start of the internal buffer.
+
+   The pointer is valid until :c:func:`PyBytesWriter_Finish` or
+   :c:func:`PyBytesWriter_Discard` is called on *writer*.
+
+
+Low-level API
+^^^^^^^^^^^^^
+
+.. c:function:: int PyBytesWriter_Resize(PyBytesWriter *writer, Py_ssize_t size)
+
+   Resize the writer to *size* bytes. It can be used to enlarge or to
+   shrink the writer.
+
+   Newly allocated bytes are left uninitialized.
+
+   On success, return ``0``.
+   On error, set an exception and return ``-1``.
+
+   *size* must be positive or zero.
+
+.. c:function:: int PyBytesWriter_Grow(PyBytesWriter *writer, Py_ssize_t grow)
+
+   Resize the writer by adding *grow* bytes to the current writer size.
+
+   Newly allocated bytes are left uninitialized.
+
+   On success, return ``0``.
+   On error, set an exception and return ``-1``.
+
+   *size* can be negative to shrink the writer.
+
+.. c:function:: void* PyBytesWriter_GrowAndUpdatePointer(PyBytesWriter *writer, Py_ssize_t size, void *buf)
+
+   Similar to :c:func:`PyBytesWriter_Grow`, but update also the *buf*
+   pointer.
+
+   The *buf* pointer is moved if the internal buffer is moved in memory.
+   The *buf* relative position within the internal buffer is left
+   unchanged.
+
+   On error, set an exception and return ``NULL``.
+
+   *buf* must not be ``NULL``.
+
+   Function pseudo-code::
+
+       Py_ssize_t pos = (char*)buf - (char*)PyBytesWriter_GetData(writer);
+       if (PyBytesWriter_Grow(writer, size) < 0) {
+           return NULL;
+       }
+       return (char*)PyBytesWriter_GetData(writer) + pos;

Doc/c-api/init.rst

@@ -41,7 +41,6 @@ The following functions can be safely called before Python is initialized:
   * :c:func:`PyObject_SetArenaAllocator`
   * :c:func:`Py_SetProgramName`
   * :c:func:`Py_SetPythonHome`
-  * :c:func:`PySys_ResetWarnOptions`
   * the configuration functions covered in :ref:`init-config`
 
 * Informative functions:
@@ -2011,6 +2010,11 @@ Reference tracing
    is set to :c:data:`PyRefTracer_DESTROY`). The **data** argument is the opaque pointer
    that was provided when :c:func:`PyRefTracer_SetTracer` was called.
 
+   If a new tracing function is registered replacing the current a call to the
+   trace function will be made with the object set to **NULL** and **event** set to
+   :c:data:`PyRefTracer_TRACKER_REMOVED`. This will happen just before the new
+   function is registered.
+
 .. versionadded:: 3.13
 
 .. c:var:: int PyRefTracer_CREATE
@@ -2023,6 +2027,13 @@ Reference tracing
    The value for the *event* parameter to :c:type:`PyRefTracer` functions when a Python
    object has been destroyed.
 
+.. c:var:: int PyRefTracer_TRACKER_REMOVED
+
+   The value for the *event* parameter to :c:type:`PyRefTracer` functions when the
+   current tracer is about to be replaced by a new one.
+
+   .. versionadded:: 3.14
+
 .. c:function:: int PyRefTracer_SetTracer(PyRefTracer tracer, void *data)
 
    Register a reference tracer function. The function will be called when a new
@@ -2038,6 +2049,10 @@ Reference tracing
 
    There must be an :term:`attached thread state` when calling this function.
 
+   If another tracer function was already registered, the old function will be
+   called with **event** set to :c:data:`PyRefTracer_TRACKER_REMOVED` just before
+   the new function is registered.
+
 .. versionadded:: 3.13
 
 .. c:function:: PyRefTracer PyRefTracer_GetTracer(void** data)

Doc/c-api/init_config.rst

@@ -1278,6 +1278,11 @@ PyConfig
 
       Default: ``0``.
 
+      .. deprecated-removed:: 3.15 3.17
+
+         The :option:`-b` and :option:`!-bb` options will become no-op in 3.17.
+         :c:member:`~PyConfig.bytes_warning` member will be removed in 3.17.
+
    .. c:member:: int warn_default_encoding
 
       If non-zero, emit a :exc:`EncodingWarning` warning when :class:`io.TextIOWrapper`

Doc/c-api/sys.rst

@@ -316,14 +316,6 @@ accessible to C code.  They all work with the current interpreter thread's
    case *name* is deleted from the sys module. Returns ``0`` on success, ``-1``
    on error.
 
-.. c:function:: void PySys_ResetWarnOptions()
-
-   Reset :data:`sys.warnoptions` to an empty list. This function may be
-   called prior to :c:func:`Py_Initialize`.
-
-   .. deprecated-removed:: 3.13 3.15
-      Clear :data:`sys.warnoptions` and :data:`!warnings.filters` instead.
-
 .. c:function:: void PySys_WriteStdout(const char *format, ...)
 
    Write the output string described by *format* to :data:`sys.stdout`.  No

Doc/data/stable_abi.dat

@@ -59,7 +59,7 @@ Pending removal in Python 3.15
     Set :c:member:`PyConfig.program_name` instead.
   * :c:func:`!Py_SetPythonHome()`:
     Set :c:member:`PyConfig.home` instead.
-  * :c:func:`PySys_ResetWarnOptions`:
+  * :c:func:`!PySys_ResetWarnOptions`:
     Clear :data:`sys.warnoptions` and :data:`!warnings.filters` instead.
 
   The :c:func:`Py_InitializeFromConfig` API should be used with

Doc/deprecations/c-api-pending-removal-in-3.15.rst

@@ -3,6 +3,78 @@
 Remote debugging attachment protocol
 ====================================
 
+This protocol enables external tools to attach to a running CPython process and
+execute Python code remotely.
+
+Most platforms require elevated privileges to attach to another Python process.
+
+.. _permission-requirements:
+
+Permission requirements
+=======================
+
+Attaching to a running Python process for remote debugging requires elevated
+privileges on most platforms. The specific requirements and troubleshooting
+steps depend on your operating system:
+
+.. rubric:: Linux
+
+The tracer process must have the ``CAP_SYS_PTRACE`` capability or equivalent
+privileges. You can only trace processes you own and can signal. Tracing may
+fail if the process is already being traced, or if it is running with
+set-user-ID or set-group-ID. Security modules like Yama may further restrict
+tracing.
+
+To temporarily relax ptrace restrictions (until reboot), run:
+
+  ``echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope``
+
+.. note::
+
+   Disabling ``ptrace_scope`` reduces system hardening and should only be done
+   in trusted environments.
+
+If running inside a container, use ``--cap-add=SYS_PTRACE`` or
+``--privileged``, and run as root if needed.
+
+Try re-running the command with elevated privileges:
+
+  ``sudo -E !!``
+
+
+.. rubric:: macOS
+
+To attach to another process, you typically need to run your debugging tool
+with elevated privileges. This can be done by using ``sudo`` or running as
+root.
+
+Even when attaching to processes you own, macOS may block debugging unless
+the debugger is run with root privileges due to system security restrictions.
+
+
+.. rubric:: Windows
+
+To attach to another process, you usually need to run your debugging tool
+with administrative privileges. Start the command prompt or terminal as
+Administrator.
+
+Some processes may still be inaccessible even with Administrator rights,
+unless you have the ``SeDebugPrivilege`` privilege enabled.
+
+To resolve file or folder access issues, adjust the security permissions:
+
+  1. Right-click the file or folder and select **Properties**.
+  2. Go to the **Security** tab to view users and groups with access.
+  3. Click **Edit** to modify permissions.
+  4. Select your user account.
+  5. In **Permissions**, check **Read** or **Full control** as needed.
+  6. Click **Apply**, then **OK** to confirm.
+
+
+.. note::
+
+   Ensure you've satisfied all :ref:`permission-requirements` before proceeding.
+
 This section describes the low-level protocol that enables external tools to
 inject and execute a Python script within a running CPython process.
 

Doc/faq/python-video-icon.png

@@ -468,7 +468,8 @@ Dialects support the following attributes:
 .. attribute:: Dialect.skipinitialspace
 
    When :const:`True`, spaces immediately following the *delimiter* are ignored.
-   The default is :const:`False`.
+   The default is :const:`False`.  When combining ``delimiter=' '`` with
+   ``skipinitialspace=True``, unquoted empty fields are not allowed.
 
 
 .. attribute:: Dialect.strict
@@ -637,7 +638,7 @@ done::
 .. rubric:: Footnotes
 
 .. [1] If ``newline=''`` is not specified, newlines embedded inside quoted fields
-   will not be interpreted correctly, and on platforms that use ``\r\n`` linendings
+   will not be interpreted correctly, and on platforms that use ``\r\n`` line endings
    on write an extra ``\r`` will be added.  It should always be safe to specify
    ``newline=''``, since the csv module does its own
    (:term:`universal <universal newlines>`) newline handling.

Doc/howto/remote_debugging.rst

@@ -1843,9 +1843,9 @@ expression support in the :mod:`re` module).
    lowercase, :meth:`lower` would do nothing to ``'ß'``; :meth:`casefold`
    converts it to ``"ss"``.
 
-   The casefolding algorithm is
-   `described in section 3.13 'Default Case Folding' of the Unicode Standard
-   <https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G33992>`__.
+   The casefolding algorithm is `described in section 3.13.3 'Default Case
+   Folding' of the Unicode Standard
+   <https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G53253>`__.
 
    .. versionadded:: 3.3
 
@@ -2056,7 +2056,7 @@ expression support in the :mod:`re` module).
    property being one of "Lm", "Lt", "Lu", "Ll", or "Lo".  Note that this is different
    from the `Alphabetic property defined in the section 4.10 'Letters, Alphabetic, and
    Ideographic' of the Unicode Standard
-   <https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-4/#G91002>`_.
+   <https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-4/#G91002>`__.
 
 
 .. method:: str.isascii()
@@ -2196,9 +2196,9 @@ expression support in the :mod:`re` module).
    Return a copy of the string with all the cased characters [4]_ converted to
    lowercase.
 
-   The lowercasing algorithm used is
-   `described in section 3.13 'Default Case Folding' of the Unicode Standard
-   <https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G33992>`__.
+   The lowercasing algorithm used is `described in section 3.13.2 'Default Case
+   Conversion' of the Unicode Standard
+   <https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G34078>`__.
 
 
 .. method:: str.lstrip(chars=None, /)
@@ -2561,9 +2561,9 @@ expression support in the :mod:`re` module).
    character(s) is not "Lu" (Letter, uppercase), but e.g. "Lt" (Letter,
    titlecase).
 
-   The uppercasing algorithm used is
-   `described in section 3.13 'Default Case Folding' of the Unicode Standard
-   <https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G33992>`__.
+   The uppercasing algorithm used is `described in section 3.13.2 'Default Case
+   Conversion' of the Unicode Standard
+   <https://www.unicode.org/versions/Unicode17.0.0/core-spec/chapter-3/#G34078>`__.
 
 
 .. method:: str.zfill(width, /)