Docs: Explain main content node detection in visual diff (#12586)

ericholscher · humitos · web-flow · commit 227df908069a · 2025-11-25T05:41:12.000-08:00
Adds a new section to the visual diff documentation explaining how Read
the Docs detects the main content area of HTML pages for comparison.

## Changes

- Added "How main content is detected" section explaining the 4-step
priority order
- Reorganized limitations section to separate content detection
explanation
- Provided recommendations for improving detection in non-standard
documentation

This helps users understand why certain changes may or may not appear in
the visual diff and what they can do to improve detection accuracy.

---
*Generated by Copilot*

---------

Co-authored-by: Manuel Kaufmann &lt;humitos@gmail.com&gt;
diff --git a/docs/user/index.rst b/docs/user/index.rst
@@ -79,6 +79,7 @@ Read the Docs: documentation simplified
    /explanation/documentation-structure
    /guides/best-practice/links
    /security-implications
+   /reference/main-content-detection
 
 .. toctree::
    :maxdepth: 1
diff --git a/docs/user/link-previews.rst b/docs/user/link-previews.rst
@@ -29,7 +29,8 @@ Troubleshooting link previews
 
 We perform some heuristic to detect the documentation tool used to generate the page based on its HTML structure.
 This auto-detection may fail, resulting in the content rendered inside the popup being incorrect.
-If you are experiencing this, you can specify the CSS selector for the main content in :guilabel:`Settings > Addons > Advanced`,
-or you can `open an issue in the addons repository <https://github.com/readthedocs/addons>`_ so we improve our heuristic.
+If you are experiencing this, you can specify the CSS selector for the main content in :guilabel:`Settings > Addons > Advanced`.
+See :ref:`reference/main-content-detection:detection logic` for how this content is detected and guidance on choosing a good selector.
+You can also `open an issue in the addons repository <https://github.com/readthedocs/addons>`_ so we improve our heuristic.
 
 Link previews won't be generated if JavaScript is not enabled in your web browser or if all cookies are blocked.
diff --git a/docs/user/reference/main-content-detection.rst b/docs/user/reference/main-content-detection.rst
@@ -0,0 +1,145 @@
+Main content detection
+======================
+
+Read the Docs detects the main content area of HTML pages
+to focus on the documentation content itself,
+ignoring headers, footers, navigation, and other page elements.
+
+Feature usage
+-------------
+
+Different features use this detection in different ways:
+
+* :doc:`/visual-diff`: Uses a documentation-tool specific heuristic first. If you configure a custom selector it overrides that heuristic.
+* :doc:`/link-previews`: Uses the custom selector (if set) to scope links; otherwise falls back to heuristics here.
+* :doc:`/server-side-search/index`: Always uses heuristics; it ignores any configured custom selector.
+
+Detection logic
+---------------
+
+The main content node is detected using the following logic, in order of priority:
+
+#. **Elements with** ``role="main"`` **attribute**: This ARIA role is used by many static site generators and themes to indicate the main content area.
+#. **The** ``<main>`` **HTML tag**: The semantic HTML5 element for main content.
+#. **Parent of the first** ``<h1>`` **tag**: If no explicit main content markers are found, the system assumes all sections are siblings under a common parent, and uses the parent of the first heading as the main content container.
+#. **The** ``<body>`` **tag**: As a last resort, if none of the above are found, the entire body is used.
+
+.. tip::
+
+   Following the ARIA_ conventions will also improve the accessibility of your site.
+   See also https://webaim.org/techniques/semanticstructure/.
+
+.. _ARIA: https://www.w3.org/TR/wai-aria/
+
+Improving detection
+-------------------
+
+If your documentation uses a non-standard structure,
+Read the Docs may not correctly identify the main content area.
+
+To improve detection, consider:
+
+- Adding a ``role="main"`` attribute to your main content container
+- Using a ``<main>`` HTML tag in your theme
+- Ensuring your main content has at least one ``<h1>`` heading
+
+Configuring a custom selector
+-----------------------------
+
+If the automatic detection does not work for your project, you can explicitly set the CSS selector for the main content node in your project settings:
+
+#. Go to your project's :guilabel:`Settings`.
+#. Click :guilabel:`Addons`.
+#. Open :guilabel:`Advanced`
+#. Fill in the :guilabel:`CSS main content selector` field (for example: ``div#main`` or ``.my-content``). Leave it blank to use automatic detection.
+#. Save the settings.
+
+When this selector is configured, it overrides the heuristic detection only for addons that honor it (currently Visual Diff and Link Previews). Choose a stable container whose structure does not change between builds to avoid spurious diffs or missed link previews. It will not affect search indexing unless we add support in the future.
+
+Examples of good selectors:
+
+- ``div#content`` (an id that wraps all page content)
+- ``div[role="main"]`` (ARIA role usage)
+
+.. warning::
+
+   Avoid overly broad selectors like ``body`` or ones matching multiple nodes (e.g. a class applied to multiple elements).
+
+Example structures
+------------------
+
+Using role="main" or <main> tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: html
+   :emphasize-lines: 10-12
+
+   <html>
+      <head>
+         ...
+      </head>
+      <body>
+         <div>
+            This content isn't processed
+         </div>
+
+         <div role="main">
+            All content inside the main node is processed
+         </div>
+
+         <footer>
+            This content isn't processed
+         </footer>
+      </body>
+   </html>
+
+Inferring from first h1 tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If a main node isn't found,
+we try to infer the main node from the parent of the first section with a ``h1`` tag.
+Example:
+
+.. code-block:: html
+   :emphasize-lines: 10-20
+
+   <html>
+      <head>
+         ...
+      </head>
+      <body>
+         <div>
+            This content isn't processed
+         </div>
+
+         <div id="parent">
+            <h1>First title</h1>
+            <p>
+               The parent of the h1 title will
+               be taken as the main node,
+               this is the div tag.
+            </p>
+
+            <h2>Second title</h2>
+            <p>More content</p>
+         </div>
+      </body>
+   </html>
+
+Fallback to body tag
+~~~~~~~~~~~~~~~~~~~~
+
+If a section title isn't found, we default to the ``body`` tag.
+Example:
+
+.. code-block:: html
+   :emphasize-lines: 5-7
+
+   <html>
+      <head>
+         ...
+      </head>
+      <body>
+         <p>Content</p>
+      </body>
+   </html>
diff --git a/docs/user/server-side-search/index.rst b/docs/user/server-side-search/index.rst
@@ -74,11 +74,20 @@ Analytics
 
 .. _Elasticsearch: https://www.elastic.co/products/elasticsearch
 
-
 Search as you type
 ------------------
 
 Search as-you-type allows users to quickly find exactly what they are looking for while typing.
 It also saves recent searches, for future reference.
 
 Try it by pressing :guilabel:`/` (forward slash) and typing.
+
+How main content is detected
+----------------------------
+
+Server Side Search indexes the "main content" of HTML pages,
+ignoring headers, footers, navigation, and other page elements that aren't part of the documentation content itself.
+This keeps results focused and avoids repeated elements like nav menus from polluting relevance.
+
+For details on how the main content area is detected,
+see :ref:`reference/main-content-detection:detection logic`.
diff --git a/docs/user/visual-diff.rst b/docs/user/visual-diff.rst
@@ -82,16 +82,26 @@ This is useful if your ``latest`` version doesn't point the default branch of yo
 
    This option can be changed by contacting :doc:`/support`.
 
+How main content is detected
+----------------------------
+
+The visual diff compares the "main content" of HTML pages,
+ignoring headers, footers, navigation, and other page elements that aren't part of the documentation content itself.
+This helps avoid false positives, like all pages being marked as changed because of a date or commit hash being updated in the footer.
+
+For details on how the main content area is detected,
+see :ref:`reference/main-content-detection:detection logic`.
+
+.. tip::
+
+  If the heuristic root element picked by Visual Diff is wrong for your project theme, set the :guilabel:`CSS main content selector` under :guilabel:`Settings > Addons`. Visual Diff honors this override; other features like Server Side Search do not.
+
 Limitations and known issues
 ----------------------------
 
 - The diff considers HTML files only.
 - The diff is done between the files from the latest successful build of the pull request and the default base version (latest by default).
   If your pull request gets out of sync with its base branch, the diff may not be accurate, and may show unrelated files and sections as changed.
-- The diff is done by comparing the "main content" of the HTML files.
-  This means that some changes outside the main content, like header or footer, may not be detected.
-  This is done to avoid showing changes that are not relevant to the documentation content itself.
-  Like all pages being marked as changed because of a date or commit hash being updated in the footer.
 - Invisible changes. Some sections may be highlighted as changed, even when they haven't actually visually changed.
   This can happen when the underlying HTML changes without a corresponding visual change, for example, if a link's URL is updated
 - Tables may be shown to have changes when they have not actually changed.
