diff --git a/README.md b/README.md
index 31549234..9ff1d961 100644
--- a/README.md
+++ b/README.md
@@ -4,15 +4,32 @@
[](https://ci.eclipse.org/tm4e/job/TM4E/)
[](LICENSE)
-TM4E provides a Java implementation of the [TextMate grammar](https://en.wikipedia.org/wiki/TextMate#Language_grammars) engine, based on the [microsoft/vscode-textmate](https://github.com/microsoft/vscode-textmate) project, that can be embedded in any Java application,
-along with Eclipse IDE integration for syntax highlighting and VS Code-style [Language Configuration](https://code.visualstudio.com/api/references/contribution-points#contributes.languages) (brackets, auto-closing, on-enter behavior, and more).
-TM4E is an [official Eclipse.org project](https://projects.eclipse.org/projects/technology.tm4e), and follows the usual Eclipse.org processes and IP requirements.
+## Table of contents
+
+- [About TM4E](#about-tm4e)
+- [Key Features](#key-features)
+- [Installation](#-installation)
+- [Getting started as an end user](#-getting-started-as-an-end-user)
+- [Who is using TM4E?](#-who-is-using-tm4e)
+- [Architecture and modules](#-architecture-and-modules)
+- [Integrating TM4E](#-integrating-tm4e)
+- [Contributing](#-contributing)
+- [License](#-license)
+
+
+## About TM4E
-**Example**: TypeScript editor using TM4E-based highlighting (from [Eclipse Wild Web Developer](https://github.com/eclipse/wildwebdeveloper)):
+TM4E provides a Java implementation of the [TextMate grammar](https://en.wikipedia.org/wiki/TextMate#Language_grammars) engine, based on the [microsoft/vscode-textmate](https://github.com/microsoft/vscode-textmate) project.
+It can be embedded in any Java application, along with Eclipse IDE integration for syntax highlighting and VS Code-style [Language Configuration](https://code.visualstudio.com/api/references/contribution-points#contributes.languages) (brackets, auto-closing, on-enter behavior, and more).
+
+**Example**: TypeScript editor using TM4E-based highlighting (from [Eclipse Wild Web Developer](https://github.com/eclipse/wildwebdeveloper))
+TM4E is an [official Eclipse.org project](https://projects.eclipse.org/projects/technology.tm4e), and follows the usual Eclipse.org processes and IP requirements.
+
+
## Key features
- TextMate-based syntax highlighting for Eclipse editors, including the Generic Editor and TM4E-powered custom editors.
@@ -20,7 +37,8 @@ TM4E is an [official Eclipse.org project](https://projects.eclipse.org/projects/
- Themable colorization through TextMate themes, with built-in Light/Dark themes and additional themes contributed by plugins.
- A TM4E secondary partitioner that understands embedded languages and exposes normalized grammar scopes that other tooling can consume.
-## 📦 Installation
+
+## 📥 Installation
### 1) In Eclipse IDE or RCP applications
@@ -42,7 +60,6 @@ TM4E version |Min. Eclipse version | Min. Java version
[0.6.2](https://download.eclipse.org/tm4e/releases/0.6.2/)+ | [2022-09](https://projects.eclipse.org/releases/2022-09) ([4.25](https://projects.eclipse.org/projects/eclipse/releases/4.25.0)) | 17
[0.6.0](https://download.eclipse.org/tm4e/releases/0.6.0/)+ | [2022-03](https://projects.eclipse.org/releases/2022-03) ([4.23](https://projects.eclipse.org/projects/eclipse/releases/4.23.0)) | 17
-
### 2) As a Java API with Maven
**Snapshot** binaries of `org.eclipse.tm4e.core` are available at https://repo.eclipse.org/content/repositories/tm4e-snapshots/org/eclipse/org.eclipse.tm4e.core/.
@@ -82,7 +99,16 @@ Then reference the dependency in your pom.xml as follows:
```
-## 👥 Who is using TM4E?
+## 🚀 Getting started as an end user
+
+If you are using an Eclipse IDE or other tool that ships TM4E-powered editors, you usually do not install TM4E directly.
+
+- For day-to-day usage, configuration, and troubleshooting of TM4E-powered editors, see the [User Guide](docs/user-guide.md).
+- For usage questions and general help, use [GitHub discussions](https://github.com/eclipse-tm4e/tm4e/discussions).
+- To report problems with TM4E-powered editors, file issues in [GitHub issues](https://github.com/eclipse-tm4e/tm4e/issues) and include details about your IDE, TM4E version, and steps to reproduce.
+
+
+## 🙋 Who is using TM4E?
Here are some projects that use TM4E:
@@ -104,15 +130,6 @@ Here are some projects that use TM4E:
- [Phaser Editor 2D](https://github.com/PhaserEditor2D/PhaserEditor) - an IDE for the creation of HTML5 games.
-## 📚 End users
-
-If you are using an Eclipse IDE or other tool that ships TM4E-powered editors, you usually do not install TM4E directly.
-
-- For day-to-day usage, configuration, and troubleshooting of TM4E-powered editors, see the [User Guide](docs/user-guide.md).
-- For usage questions and general help, use [GitHub discussions](https://github.com/eclipse-tm4e/tm4e/discussions).
-- To report problems with TM4E-powered editors, file issues in [GitHub issues](https://github.com/eclipse-tm4e/tm4e/issues) and include details about your IDE, TM4E version, and steps to reproduce.
-
-
## 🧩 Architecture and modules
TM4E is split into three main bundles plus an optional language pack feature:
@@ -120,19 +137,19 @@ TM4E is split into three main bundles plus an optional language pack feature:
- [org.eclipse.tm4e.core](https://github.com/eclipse-tm4e/tm4e/tree/main/org.eclipse.tm4e.core) - Java TextMate engine (a port of [vscode-textmate](https://github.com/Microsoft/vscode-textmate)), UI-agnostic and usable with any Java UI toolkit.
- [org.eclipse.tm4e.ui](https://github.com/eclipse-tm4e/tm4e/tree/main/org.eclipse.tm4e.ui) - Eclipse integration layer providing [`TMPresentationReconciler`](org.eclipse.tm4e.ui/src/main/java/org/eclipse/tm4e/ui/text/TMPresentationReconciler.java), TextMate-based partitioning, theming, preference pages, hovers, and diagnostics.
- [org.eclipse.tm4e.languageconfiguration](https://github.com/eclipse-tm4e/tm4e/tree/main/org.eclipse.tm4e.languageconfiguration) - VS Code-style `language-configuration.json` support for bracket handling, auto-close, on-enter behavior, and comment toggling.
-- [org.eclipse.tm4e.language_pack](https://github.com/eclipse-tm4e/tm4e/tree/main/org.eclipse.tm4e.language_pack) - Optional language pack feature that bundles TextMate grammars and language configurations so many common languages work out of the box in the Eclipse IDE.
+- [org.eclipse.tm4e.language_pack](https://github.com/eclipse-tm4e/tm4e/tree/main/org.eclipse.tm4e.language_pack) - Optional language pack feature that bundles 50+ TextMate grammars and language configurations so many common languages work out of the box in the Eclipse IDE.
## 🔌 Integrating TM4E
If you are an Eclipse plug-in developer or Java tool author and want to consume TM4E:
-- Start with the [Plugin Developer Guide](docs/plugin-developer-guide.md) for how to depend on TM4E, register grammars and language configurations, and wire TM4E into editors.
+- Start with the [Plugin Developer Guide](docs/plugin-developer-guide.md) for how to depend on TM4E, contribute grammars, themes, and language configurations, and wire TM4E into editors.
- For integration questions, best practices, and design discussions, use [GitHub discussions](https://github.com/eclipse-tm4e/tm4e/discussions).
- To report bugs or request new APIs or integration features, file issues in [GitHub issues](https://github.com/eclipse-tm4e/tm4e/issues).
-## 🛠 Contributing
+## 🤝 Contributing
If you want to work on TM4E itself:
diff --git a/docs/plugin-developer-guide.md b/docs/plugin-developer-guide.md
index 83a36aa4..6e7da298 100644
--- a/docs/plugin-developer-guide.md
+++ b/docs/plugin-developer-guide.md
@@ -3,16 +3,17 @@
This guide is for Eclipse plugin developers who want to add TextMate-based syntax highlighting and language configuration features to their editors.
It explains the main TM4E concepts, how to contribute grammars and themes, how to connect editors to TM4E, and how to use the TM partitioner and language configuration support.
+
## Table of contents
1. [Core Concepts](#core-concepts)
1. [Contributing Grammars](#contributing-grammars)
1. [External Grammar Injection](#external-grammar-injection)
+1. [Contributing Language Configurations](#contributing-language-configurations)
+1. [Contributing Themes](#contributing-themes)
1. [Wiring Editors to TM4E](#wiring-editors-to-tm4e)
1. [Using `TextEditor`](#using-texteditor)
1. [Using the Generic Editor](#using-the-generic-editor)
-1. [Themes for Plugin Authors](#themes-for-plugin-authors)
-1. [Using Language Configuration](#using-language-configuration)
1. [Using the TM Partitioner from Code](#using-the-tm-partitioner-from-code)
1. [Diagnostics: Token and Scope Hover](#diagnostics-token-and-scope-hover)
1. [Samples and Tools](#samples-and-tools)
@@ -30,6 +31,7 @@ Language configuration JSON files add structural editor behavior on top of token
Finally, a TM-specific document partitioner runs as a secondary partitioner.
It exposes the normalized language scope for each region of a document, so that features like content assist or hovers can react to the active embedded language.
+
## Contributing Grammars
Grammars are contributed through the `org.eclipse.tm4e.registry.grammars` extension point. TM4E supports the common TextMate formats:
@@ -75,6 +77,83 @@ The screenshots below show the effect before and after enabling these injections
 
+
+## Contributing Language Configurations
+
+Language configuration describes editor behavior beyond pure tokenization. TM4E consumes VS Code style `language-configuration.json` files and exposes them through the `org.eclipse.tm4e.languageconfiguration` bundle.
+
+### 1) Contributing a language-configuration file
+
+As a plug-in author you normally contribute a JSON configuration file with the `org.eclipse.tm4e.languageconfiguration.languageConfigurations` extension point:
+
+```xml
+
+
+
+```
+
+The `contentTypeId` in the example refers to the Rust editor from the Corrosion project. Replace it with the content type for your own language so that TM4E can apply the configuration to the correct files. The `path` is resolved relative to your plug-in and the JSON file must be listed in `build.properties`.
+
+The configuration file can contain sections such as `comments`, `brackets`, `autoClosingPairs`, `surroundingPairs`, `folding`, `wordPattern`, and `onEnterRules`. When both the configuration and the current implementation support a feature, TM4E enables:
+
+- auto closing and surrounding pairs
+- matching bracket highlighting
+- on-enter indentation and comment continuation
+- toggle line/block comments
+- folding
+
+### 2) JSON format and compatibility with VS Code
+
+TM4E reads `language-configuration.json` files in the same way as VS Code, with the following points to keep in mind:
+
+- `onEnterRules` are declared in the JSON file.
+- Regular expressions are provided as JSON strings, so backslashes must be escaped just as in VS Code.
+- Indent actions are expressed as string values (for example `"Indent"`, `"Outdent"`, `"IndentOutdent"`, or `"none"`), which TM4E maps internally to the corresponding indentation behavior.
+
+The recommended way to define language configuration is always via JSON `language-configuration.json` files, either contributed through the `languageConfigurations` extension point or added by users in the preferences. Directly implementing TM4E's internal language-configuration interfaces in Java is not required for typical plug-ins and is not considered a stable API surface.
+
+### 3) How TM4E uses language configuration at runtime
+
+Users can attach additional configuration files or disable specific features through the `TextMate > Language Configuration` preference page; this guide focuses on the plug-in-side extension mechanism.
+
+For editors based on the Generic Editor, TM4E contributes:
+
+- auto edit strategies that drive on-enter indentation and auto closing/surrounding pairs based on the configuration
+- character pair matchers that implement matching bracket highlighting
+- folding reconcilers that use language configuration for region folding
+
+Custom `TextEditor`-based editors can reuse the same TM4E strategies and matchers by installing them in their own `SourceViewerConfiguration` if they want consistent behavior.
+
+TM4E also defines commands for toggling line comments and adding or removing block comments that respect the active language configuration. These commands are bound to standard text editor key bindings where applicable and can be invoked from the IDE's key bindings and command framework.
+
+
+## Contributing Themes
+
+TM4E ships with built-in Light and Dark themes that are linked to the Eclipse appearance themes, but plugins can contribute additional CSS-based themes through the `org.eclipse.tm4e.ui.themes` extension point.
+
+```xml
+
+
+
+```
+
+Themes can be flagged as more suitable for light or dark appearances and can be associated with specific grammar scopes so that, for example, a dedicated theme applies whenever a particular language is active.
+You declare one or more `` elements and then add `themeAssociation` elements that link themes to one or more scopes and optional dark/light variants.
+The exact attributes and options are described in the `themes` extension point schema.
+
+At runtime you can also force a specific theme for an editor by calling `setThemeId` on the presentation reconciler:
+
+```java
+var r = new org.eclipse.tm4e.ui.text.TMPresentationReconciler();
+r.setThemeId(org.eclipse.tm4e.ui.themes.ThemeIdConstants.Monokai);
+```
+
+
## Wiring Editors to TM4E
Editors must be explicitly wired to TM4E so that tokenization and presentation updates are driven by the TextMate engine.
@@ -157,79 +236,6 @@ For a new language with its own file extension, a minimal `plugin.xml` wiring th
The grammar file `syntaxes/MyLang.tmLanguage.json` must be packaged in your plug-in and listed in `build.properties`. With this setup, opening a `.mylang` file in the Generic Editor will cause TM4E to resolve the `com.example.mylang` content type, load the `source.mylang` grammar, and drive TextMate-based highlighting for that editor.
-## Themes for Plugin Authors
-
-TM4E ships with built-in Light and Dark themes that are linked to the Eclipse appearance themes, but plugins can contribute additional CSS-based themes through the `org.eclipse.tm4e.ui.themes` extension point.
-
-```xml
-
-
-
-```
-
-Themes can be flagged as more suitable for light or dark appearances and can be associated with specific grammar scopes so that, for example, a dedicated theme applies whenever a particular language is active.
-You declare one or more `` elements and then add `themeAssociation` elements that link themes to one or more scopes and optional dark/light variants.
-The exact attributes and options are described in the `themes` extension point schema.
-
-At runtime you can also force a specific theme for an editor by calling `setThemeId` on the presentation reconciler:
-
-```java
-var r = new org.eclipse.tm4e.ui.text.TMPresentationReconciler();
-r.setThemeId(org.eclipse.tm4e.ui.themes.ThemeIdConstants.Monokai);
-```
-
-## Using Language Configuration
-
-Language configuration describes editor behavior beyond pure tokenization. TM4E consumes VS Code style `language-configuration.json` files and exposes them through the `org.eclipse.tm4e.languageconfiguration` bundle.
-
-### 1) Contributing a language-configuration file
-
-As a plug-in author you normally contribute a JSON configuration file with the `org.eclipse.tm4e.languageconfiguration.languageConfigurations` extension point:
-
-```xml
-
-
-
-```
-
-The `contentTypeId` in the example refers to the Rust editor from the Corrosion project. Replace it with the content type for your own language so that TM4E can apply the configuration to the correct files. The `path` is resolved relative to your plug-in and the JSON file must be listed in `build.properties`.
-
-The configuration file can contain sections such as `comments`, `brackets`, `autoClosingPairs`, `surroundingPairs`, `folding`, `wordPattern`, and `onEnterRules`. When both the configuration and the current implementation support a feature, TM4E enables:
-
-- auto closing and surrounding pairs
-- matching bracket highlighting
-- on-enter indentation and comment continuation
-- toggle line/block comments
-- folding
-
-### 2) JSON format and compatibility with VS Code
-
-TM4E reads `language-configuration.json` files in the same way as VS Code, with the following points to keep in mind:
-
-- `onEnterRules` are declared in the JSON file.
-- Regular expressions are provided as JSON strings, so backslashes must be escaped just as in VS Code.
-- Indent actions are expressed as string values (for example `"Indent"`, `"Outdent"`, `"IndentOutdent"`, or `"none"`), which TM4E maps internally to the corresponding indentation behavior.
-
-The recommended way to define language configuration is always via JSON `language-configuration.json` files, either contributed through the `languageConfigurations` extension point or added by users in the preferences. Directly implementing TM4E's internal language-configuration interfaces in Java is not required for typical plug-ins and is not considered a stable API surface.
-
-### 3) How TM4E uses language configuration at runtime
-
-Users can attach additional configuration files or disable specific features through the `TextMate > Language Configuration` preference page; this guide focuses on the plug-in-side extension mechanism.
-
-For editors based on the Generic Editor, TM4E contributes:
-
-- auto edit strategies that drive on-enter indentation and auto closing/surrounding pairs based on the configuration
-- character pair matchers that implement matching bracket highlighting
-- folding reconcilers that use language configuration for region folding
-
-Custom `TextEditor`-based editors can reuse the same TM4E strategies and matchers by installing them in their own `SourceViewerConfiguration` if they want consistent behavior.
-
-TM4E also defines commands for toggling line comments and adding or removing block comments that respect the active language configuration. These commands are bound to standard text editor key bindings where applicable and can be invoked from the IDE's key bindings and command framework.
## Using the TM Partitioner from Code
@@ -277,6 +283,7 @@ If your feature uses `IContentType` rather than scopes, `TMPartitions.getContent
Convenience methods such as `TMPartitions.hasPartitioning(doc)` and `TMPartitions.getPartition(doc, offset)` help detect whether TM4E is active and retrieve the current region.
Generic Editor extension points still refer to the editor's primary partitioning via their `partitionType` attribute; you always query TM4E's secondary partitioner from code instead of configuring it directly in the extension XML.
+
## Diagnostics: Token and Scope Hover
When debugging grammars, themes, or partitioning, it is often helpful to see exactly which TextMate token and scopes TM4E has computed at a given location in the editor.
@@ -289,10 +296,15 @@ When the hover is enabled, hovering over a token shows:
- The full list of TextMate scopes applied to the token.
- The JFace text partition information at the caret (offset, length, type) and the TM4E grammar scope for that region.
+
+
You can enable or disable this hover on the main `TextMate` preference page (`Window > Preferences > TextMate`) by toggling the "Show TextMate token info hover" checkbox.
This preference applies to editors that use the Generic Editor framework and have TM4E enabled.
The hover is primarily intended for plugin and grammar authors as a troubleshooting tool rather than as an end-user feature.
+
+
+
## Samples and Tools
The `org.eclipse.tm4e.samples` project in this repository contains working examples of grammars, language configurations, editors, and extension point contributions.
@@ -304,6 +316,7 @@ These pages let you inspect and configure grammars, themes, task tags, and langu
For interactive grammar development there is also a separate [Previewer](https://marketplace.eclipse.org/content/previewer) plugin that provides a view showing live syntax highlighting for the grammar file you are editing.
Combined with TM4E this allows you to iterate quickly on grammars and themes while validating the scopes that the tokenizer produces.
+
## Further Reading
For a tutorial-style introduction to implementing TM4E-powered editors and understanding how TextMate grammars and language-configuration files fit together, see:
diff --git a/docs/user-guide.md b/docs/user-guide.md
index a7c99cb5..696a2677 100644
--- a/docs/user-guide.md
+++ b/docs/user-guide.md
@@ -2,6 +2,7 @@
This guide is for Eclipse users who work with editors powered by TM4E. It explains what TM4E does in the UI, how to install and configure it, and how to troubleshoot common problems.
+
## Table of contents
1. [Getting TM4E in Eclipse](#getting-tm4e-in-eclipse)