fix(api-markdown-documenter): Fixed an issue where MarkdownRenderer would emit trailing \ characters when processing certain summary comments. (#25933)

The issue here is somewhat complicated. The following is an attempt to
provide a high-level-ish overview of what's going on.

## The Problem

Keeping the description at a high-level, `CommonMark` specifies 2 ways
to force an explicit line break: either ending a line with 2 spaces or
ending a line with a backslash.
- https://commonmark.org/help/tutorial/03-paragraphs.html

`mdast-util-to-markdown`, the library we use to emit prefers the latter,
as it is more explicit.

In most cases, this is irrelevant, and Markdown parsers that adhere to
`CommonMark` should tolerate this. Docusaurus doesn't appear to, causing
spurious trailing backslash characters to appear in *some* of our API
documentation. E.g.
https://fluidframework.com/docs/api/fluid-framework/allowedtypesfullevaluated-typealias

But our library shouldn't need to output forced line breaks. We build
our Markdown AST trees manually, and so we control the output structure.
We generally avoid explicit line breaks, and instead use `Paragraph`
nodes when separating blocks of text.

But for contents parsed by `TSDoc`, we _transform_ their parsed content,
making certain assumptions about how the parser behaves.

The case that was causing problems for us is a case where `TSDoc` fails
to trim trailing whitespace from comment blocks (generally this
manifests in the summary component). In particular, when a summary
comment is followed by 2 or more modifier tags on a subsequent line,
TSDoc fails to trim the summary comment, yielding something with
trailing explicit line breaks.

### Example

Given the following comment:

```typescript 
/**
 * I am a comment
 * 
 */
```

The TSDoc parser will yield a summary block of `I am a comment`

But given the following comment:

```typescript
/**
 * I am a comment
 * 
 * @Sealed @public
 */
```

The TSDoc parser will yield a summary block of `I am a comment \n`

## The Solution

The solution is to better trim paragraph contents given to us from the
`TSDoc` parser. Previously, we would trim any leading/trailing line
breaks we encountered. Now, we also trim and leading/trailing whitespace
content.

This simplifies the Paragraph trees we hand off to
`mdast-util-to-markdown`, removing the need for them to inject trailing
`\` characters to force line breaks.


[AB#53737](https://dev.azure.com/fluidframework/235294da-091d-4c29-84fc-cdfc3d90890b/_workitems/edit/53737)

Author: Josmithr

tools/api-markdown-documenter/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @fluid-tools/api-markdown-documenter
 
+## 0.23.2
+
+### 🐞 Bug Fixes
+
+- Fixed an issue where `MarkdownRenderer` would emit trailing `\` characters when processing certain summary comments.
+
 ## 0.23.1
 
 ### 🐞 Bug Fixes

tools/api-markdown-documenter/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@fluid-tools/api-markdown-documenter",
-	"version": "0.23.1",
+	"version": "0.23.2",
 	"description": "Processes .api.json files generated by API-Extractor and generates Markdown documentation from them.",
 	"homepage": "https://fluidframework.com",
 	"repository": {

tools/api-markdown-documenter/src/api-item-transforms/TsdocNodeTransforms.ts

@@ -157,7 +157,7 @@ function transformTsdocSectionContent(
  * 2. Remove line break nodes adjacent to paragraph nodes.
  *
  * 3. Remove leading and trailing line breaks within the paragraph (see
- * {@link trimLeadingAndTrailingLineBreaks}).
+ * {@link trimParagraphContents}).
  *
  * 4. Trim leading whitespace from first child if it is plain-text, and trim trailing whitespace from
  * last child if it is plain-text.
@@ -170,8 +170,8 @@ function transformTsdocParagraph(
 	// To ensure we map the content correctly, we should scan the child list for matching open/close tags,
 	// and map the subsequence to an "html" node.
 
-	// Trim leading and trailing line breaks, which are redundant in the context of a paragraph.
-	const adjustedChildren = trimLeadingAndTrailingLineBreaks(node.nodes);
+	// Trim leading and trailing whitespace and line breaks, which are redundant in the context of a paragraph.
+	const adjustedChildren = trimParagraphContents(node.nodes);
 
 	// Transform child items into Documentation domain
 	const transformedChildren: PhrasingContent[] = [];
@@ -698,29 +698,37 @@ function combineAdjacentPlainText(nodes: readonly PhrasingContent[]): PhrasingCo
 }
 
 /**
- * Trims an line break nodes found at the beginning or end of the list.
- *
- * @remarks Useful for cleaning up {@link ParagraphNode} child contents, since leading and trailing
- * newlines are effectively redundant.
+ * Trims any whitespace and line break nodes found at the beginning or end of the list of paragraph contents.
+ * In the context of a Paragraph, such content is redundant.
  */
-function trimLeadingAndTrailingLineBreaks(nodes: readonly DocNode[]): DocNode[] {
+function trimParagraphContents(nodes: readonly DocNode[]): DocNode[] {
 	if (nodes.length === 0) {
 		return [];
 	}
 
 	let startIndex = 0;
 	let endIndex = nodes.length - 1;
 
-	for (const node of nodes) {
+	function canTrimNode(node: DocNode): boolean {
 		if (node.kind === DocNodeKind.SoftBreak) {
+			return true;
+		}
+		if (node.kind === DocNodeKind.PlainText) {
+			return (node as DocPlainText).text.trim().length === 0;
+		}
+		return false;
+	}
+
+	for (const node of nodes) {
+		if (canTrimNode(node)) {
 			startIndex++;
 		} else {
 			break;
 		}
 	}
 
 	for (let i = nodes.length - 1; i > startIndex; i--) {
-		if (nodes[i].kind === DocNodeKind.SoftBreak) {
+		if (canTrimNode(nodes[i])) {
 			endIndex--;
 		} else {
 			break;

tools/api-markdown-documenter/src/api-item-transforms/test/TsdocNodeTransforms.test.ts

@@ -870,6 +870,37 @@ describe("Tsdoc node transformation tests", () => {
 				},
 			]);
 		});
+
+		describe("Regression tests", () => {
+			// TSDoc currently does something funny when parsing comments like the following case.
+			// If the summary comment is followed by a line with multiple modifier tags, TSDoc doesn't seem to trim trailing whitespace and newlines from the summary.
+			// This results in us generating paragraph content with trailing lines and whitespace that aren't desired.
+			// This test exists to ensure we handle this case correctly and don't emit extra whitespace.
+			it("Summary content parsed with trailing line break", () => {
+				// Note
+				const comment = `/**
+ * I am a regression test.
+ *
+ * @sealed @beta
+ */`;
+				const context = parser.parseString(comment);
+				const summarySection = context.docComment.summarySection;
+
+				const result = transformTsdocSection(summarySection, transformOptions);
+
+				expect(result).to.deep.equal([
+					{
+						type: "paragraph",
+						children: [
+							{
+								type: "text",
+								value: "I am a regression test.",
+							},
+						],
+					},
+				]);
+			});
+		});
 	});
 
 	// Test TODOs:

tools/api-markdown-documenter/src/test/snapshots/markdown/simple-suite-test/deep-config/test-suite-a/index.md

@@ -88,8 +88,8 @@ const foo = bar;
 
 ## Namespaces
 
-| Namespace | Alerts | Description |
-| - | - | - |
-| [TestBetaNamespace](/test-suite-a/testbetanamespace-namespace/) | `Beta` | A namespace tagged as `@beta`. |
-| [TestModule](/test-suite-a/testmodule-namespace/) | | |
-| [TestNamespace](/test-suite-a/testnamespace-namespace/) | | Test Namespace |
+| Namespace | Alerts | Modifiers | Description |
+| - | - | - | - |
+| [TestBetaNamespace](/test-suite-a/testbetanamespace-namespace/) | `Beta` | `sealed` | A namespace tagged as `@beta`. |
+| [TestModule](/test-suite-a/testmodule-namespace/) | | | |
+| [TestNamespace](/test-suite-a/testnamespace-namespace/) | | | Test Namespace |

tools/api-markdown-documenter/src/test/snapshots/markdown/simple-suite-test/deep-config/test-suite-a/testbetanamespace-namespace/index.md

@@ -9,6 +9,7 @@ A namespace tagged as `@beta`.
 <h2 id="testbetanamespace-signature">Signature</h2>
 
 ```typescript
+/** @sealed */
 export declare namespace TestBetaNamespace
 ```
 

tools/api-markdown-documenter/src/test/snapshots/markdown/simple-suite-test/default-config/test-suite-a/index.md

@@ -88,11 +88,11 @@ const foo = bar;
 
 ## Namespaces
 
-| Namespace | Alerts | Description |
-| - | - | - |
-| [TestBetaNamespace](/test-suite-a/testbetanamespace-namespace/) | `Beta` | A namespace tagged as `@beta`. |
-| [TestModule](/test-suite-a/testmodule-namespace/) | | |
-| [TestNamespace](/test-suite-a/testnamespace-namespace/) | | Test Namespace |
+| Namespace | Alerts | Modifiers | Description |
+| - | - | - | - |
+| [TestBetaNamespace](/test-suite-a/testbetanamespace-namespace/) | `Beta` | `sealed` | A namespace tagged as `@beta`. |
+| [TestModule](/test-suite-a/testmodule-namespace/) | | | |
+| [TestNamespace](/test-suite-a/testnamespace-namespace/) | | | Test Namespace |
 
 ## Function Details
 

tools/api-markdown-documenter/src/test/snapshots/markdown/simple-suite-test/default-config/test-suite-a/testbetanamespace-namespace/index.md

@@ -9,6 +9,7 @@ A namespace tagged as `@beta`.
 <h2 id="testbetanamespace-signature">Signature</h2>
 
 ```typescript
+/** @sealed */
 export declare namespace TestBetaNamespace
 ```
 

tools/api-markdown-documenter/src/test/snapshots/markdown/simple-suite-test/flat-config/test-suite-a.md

@@ -85,11 +85,11 @@ const foo = bar;
 
 # Namespaces
 
-| Namespace | Alerts | Description |
-| - | - | - |
-| [TestBetaNamespace](docs/test-suite-a#testbetanamespace-namespace) | `Beta` | A namespace tagged as `@beta`. |
-| [TestModule](docs/test-suite-a#testmodule-namespace) | | |
-| [TestNamespace](docs/test-suite-a#testnamespace-namespace) | | Test Namespace |
+| Namespace | Alerts | Modifiers | Description |
+| - | - | - | - |
+| [TestBetaNamespace](docs/test-suite-a#testbetanamespace-namespace) | `Beta` | `sealed` | A namespace tagged as `@beta`. |
+| [TestModule](docs/test-suite-a#testmodule-namespace) | | | |
+| [TestNamespace](docs/test-suite-a#testnamespace-namespace) | | | Test Namespace |
 
 # Interface Details
 
@@ -1181,6 +1181,7 @@ A namespace tagged as `@beta`.
 <h3 id="testbetanamespace-signature">Signature</h3>
 
 ```typescript
+/** @sealed */
 export declare namespace TestBetaNamespace
 ```
 

tools/api-markdown-documenter/src/test/test-data/simple-suite-test/test-suite-a.api.json

@@ -569,7 +569,7 @@
 				{
 					"kind": "Namespace",
 					"canonicalReference": "test-suite-a!TestBetaNamespace:namespace",
-					"docComment": "/**\n * A namespace tagged as `@beta`.\n *\n * @remarks\n *\n * Tests release level inheritance.\n *\n * @beta\n */\n",
+					"docComment": "/**\n * A namespace tagged as `@beta`.\n *\n * @remarks\n *\n * Tests release level inheritance.\n *\n * @beta @sealed\n */\n",
 					"excerptTokens": [
 						{
 							"kind": "Content",