20251115_02-Update

- Attacked the TODO list with some simple glyph fixes.

Author: unixwzrd

CHANGELOG.md

@@ -9,7 +9,8 @@
 - README: clarified installation modes (standard, editable, and NLP extras), tightened wording, and refreshed badges/links.
 - Setup: improved guidance printed by `setup.sh` after environment creation; clarified quick start steps.
 - Requirements: synced with current packaging to ensure local venv installs match `pyproject.toml` expectations.
-- No behavior changes to the cleaner; tests unchanged.
+- Normalization: fold fullwidth square brackets 【】 to ASCII [] by default; add `--keep-fullwidth-brackets` to preserve them; dagger `†` remains untouched.
+- Minor behavior change: default folding of fullwidth brackets; use the new flag to opt out.
 
 ## 2025-09-18
 

README.md

@@ -102,7 +102,7 @@ Once installed and activated:
 ```bash
 (LLaSA-speech) [unixwzrd@xanax: bin]$ cleanup-text --help
 
-usage: cleanup-text [-h] [-i] [-Q] [-D] [-n] [-o OUTPUT] [-t] [-p] [infile ...]
+usage: cleanup-text [-h] [-i] [-Q] [-D] [--keep-fullwidth-brackets] [-n] [-o OUTPUT] [-t] [-p] [infile ...]
 
 Clean Unicode quirks from text. If no input files are given, reads from STDIN and writes to STDOUT (filter mode). If input files are given, creates cleaned files with .clean before the extension (e.g., foo.txt -> foo.clean.txt). Use -o - to force output to STDOUT for all input files, or -o <file> to specify a single output file (only with one
 input file).
@@ -116,6 +116,8 @@ options:
   -Q, --keep-smart-quotes
                         Preserve Unicode smart quotes (do not convert to ASCII)
   -D, --keep-dashes     Preserve Unicode EN/EM dashes (do not convert to ASCII)
+  --keep-fullwidth-brackets
+                        Preserve fullwidth square brackets (【】) (do not fold to ASCII)
   -n, --no-newline      Do not add a newline at the end of the output file (suppress final newline).
   -o OUTPUT, --output OUTPUT
                         Output file name, or '-' for STDOUT. Only valid with one input file, or use '-' for STDOUT with multiple files.
@@ -127,6 +129,7 @@ options:
 
 - `-Q`, `--keep-smart-quotes`: Preserve Unicode smart quotes (curly single/double quotes). Useful when preparing prose/blog posts where typographic quotes are intentional. Default behavior converts them to ASCII for shell/CI safety.
 - `-D`, `--keep-dashes`: Preserve EN/EM dashes. Useful when stylistic punctuation is desired in prose. Default behavior converts EM dash to ` - ` and EN dash to `-`.
+- `--keep-fullwidth-brackets`: Preserve fullwidth square brackets (`【】`). By default, they are folded to ASCII `[]` to keep monospace alignment in terminals and fixed-width tables.
 - `-R`, `--report`: Audit text for anomalies, human-readable.
 - `-J`, `--json`: Audit text for anomalies, JSON format.
 - `-T`, `--threshold`: Fail CI if anomalies exceed threshold.

docs/cleanup-text.md

@@ -1,6 +1,6 @@
-# Unicode Text Cleaner (`cleanup-text`) - v1.1.0
+# Unicode Text Cleaner (`cleanup-text`) - v1.1.2
 
-*Last updated: 2025-09-18*
+*Last updated: 2025-11-15*
 
 A robust command-line tool to normalize and clean problematic Unicode characters, invisible characters, and formatting quirks from text files. Designed to make code and text more human, linter-friendly, and free of "AI tells" or watermarks.
 
@@ -85,17 +85,18 @@ tests/test_all.sh --help
 - Use the `-i` flag if you need to preserve invisible Unicode characters for special use cases.
 - Use the `-n` flag if you need to suppress the final newline (rare).
 
-## TODO (Alignment & Bracket Normalization)
-
-- Add optional folding of fullwidth square brackets to ASCII in `unicodefix.transforms.clean_text`:
-  - Map `【` → `[` and `】` → `]` under a new flag (e.g., `preserve_fullwidth_brackets: bool = False`).
-  - Preserve dagger glyph `†` and inline spans (e.g., `†L147-L156`).
-  - Rationale: terminal table alignment (fixed-width) and monospace column layout can drift with fullwidth characters.
-- Consider expanding flags to preserve typographic punctuation while still removing invisible/control chars:
-  - Existing: `preserve_quotes`, `preserve_dashes`
-  - Proposed: `preserve_fullwidth_brackets`, `preserve_fullwidth_variants`
-- Provide helper for ASCII-only display normalization for terminals while retaining original text for auditing/search.
-- Document patterns: (1) global pre-clean before render, (2) render-time folding behind a toggle.
+## Alignment & Fullwidth Brackets
+
+- Fullwidth square brackets are now folded to ASCII by default to preserve monospace alignment in terminals and fixed-width tables:
+  - `【` → `[`, `】` → `]`
+  - Use `--keep-fullwidth-brackets` to preserve `【】`.
+  - The dagger glyph `†` (e.g., `†L147-L156`) is preserved.
+- A small helper exists for display-only folding:
+  - `unicodefix.transforms.fold_for_terminal_display(text)` applies the same folding without other cleaning.
+  - Useful when you want ASCII rendering for terminals while keeping the original text for auditing/search.
+- Patterns:
+  1) Global pre-clean before render (default behavior).
+  2) Render-time folding behind a toggle (use the helper).
 
 ## Changelog
 

docs/test-suite.md

@@ -1,6 +1,6 @@
-# Test Suite for cleanup-text - v1.1.0
+# Test Suite for cleanup-text - v1.1.2
 
-*Last updated: 2025-09-18*
+*Last updated: 2025-11-15*
 
 ## Overview
 

src/__init__.py

@@ -1,2 +1,2 @@
 __all__ = ["clean_text", "handle_newlines"]
-__version__ = "1.1.0"
+__version__ = "1.1.2"

src/unicodefix/cli.py

@@ -103,6 +103,7 @@ def run_filter_mode(args) -> None:
         preserve_invisible=args.invisible,
         preserve_quotes=args.keep_smart_quotes,
         preserve_dashes=args.keep_dashes,
+        preserve_fullwidth_brackets=args.keep_fullwidth_brackets,
     )
     cleaned = handle_newlines(cleaned, args.no_newline)
     # VSCode quirk: append only to stdout
@@ -142,6 +143,7 @@ def process_file(infile: str, args) -> None:
                     preserve_invisible=args.invisible,
                     preserve_quotes=args.keep_smart_quotes,
                     preserve_dashes=args.keep_dashes,
+                    preserve_fullwidth_brackets=args.keep_fullwidth_brackets,
                 )
                 cleaned = handle_newlines(cleaned, args.no_newline)
                 _write_text(infile, cleaned, eol)
@@ -167,6 +169,7 @@ def process_file(infile: str, args) -> None:
             preserve_invisible=args.invisible,
             preserve_quotes=args.keep_smart_quotes,
             preserve_dashes=args.keep_dashes,
+            preserve_fullwidth_brackets=args.keep_fullwidth_brackets,
         )
         cleaned = handle_newlines(cleaned, args.no_newline)
 
@@ -202,6 +205,8 @@ def main():
                         help="Preserve Unicode smart quotes")
     parser.add_argument("-D", "--keep-dashes", action="store_true",
                         help="Preserve Unicode EN/EM dashes")
+    parser.add_argument("--keep-fullwidth-brackets", action="store_true",
+                        help="Preserve fullwidth square brackets (【】)")
     parser.add_argument("-n", "--no-newline", action="store_true",
                         help="Do not add a final newline")
     parser.add_argument("-o", "--output",

src/unicodefix/transforms.py

@@ -23,6 +23,7 @@ def clean_text(
     preserve_invisible: bool = False,
     preserve_quotes: bool = False,
     preserve_dashes: bool = False,
+    preserve_fullwidth_brackets: bool = False,
 ) -> str:
     """
     Normalize problematic/invisible Unicode to safe ASCII where appropriate.
@@ -62,6 +63,15 @@ def clean_text(
         text = re.sub(r"\s*\u2014\s*", " - ", text)  # EM → space-dash-space
         text = text.replace("\u2013", "-")           # EN → dash
 
+    # Fold select fullwidth punctuation that affects monospace alignment
+    if not preserve_fullwidth_brackets:
+        FULLWIDTH_FOLD = {
+            "\u3010": "[",  # 【
+            "\u3011": "]",  # 】
+        }
+        if any(ch in text for ch in FULLWIDTH_FOLD):
+            text = text.translate(str.maketrans(FULLWIDTH_FOLD))
+
     # Zs separators → ASCII space
     text = re.sub(r"[\u00A0\u1680\u2000-\u200A\u202F\u205F\u3000]", " ", text)
 
@@ -89,3 +99,16 @@ def handle_newlines(text: str, no_newline: bool = False) -> str:
     if no_newline:
         return text
     return text if text.endswith(("\n", "\r", "\r\n")) else text + "\n"
+
+
+def fold_for_terminal_display(text: str) -> str:
+    """
+    Fold a minimal set of width-breaking Unicode punctuation for better terminal alignment.
+    - Fullwidth square brackets 【】 → ASCII [].
+    - Intentionally does not touch † (dagger) and similar glyphs.
+    """
+    mapping = {
+        "\u3010": "[",  # 【
+        "\u3011": "]",  # 】
+    }
+    return text.translate(str.maketrans(mapping))