TM4E Grammar Processing and Syntax Coloring Divergence from TextMate

[jchamlin]

I'm working on a way to get TM4E to be able to emulate DevStyle's Darkest Dark theme so I can use it as a base and extend it with additional scopes it does not cover, like invalid and invalid.illegal. And since I can't "export" the dynamic theme from the TM4E Theme's page or find it anywhere else in raw form, I've come up with some creative ways to grab all of it's definitions from a workspace export. Nonetheless, I'm getting pretty close to a tmTheme that replicates its defaults of whatever's setup within Eclipse and that will give me the ability to start with it as a base and add more scopes.

While I was doing that I wated a rapid way to test my tmTheme and adjust it. So, I started using TmLanguage and TmTheme Playground at https://youngspe.dev/textmate-grammar-playground/. I loaded in the java.tmLanguage.json from https://github.com/eclipse-tm4e/tm4e/blob/main/org.eclipse.tm4e.language_pack/syntaxes/java/java.tmLanguage.json, and my theme, and some sample code, and I noticed something. I don't think TM4E is determining the same scopes as a standard TextMate engine does. For example, extends Someclass, the Someclass ends up with scopes:
entity.other.inherited-class.java
meta.definition.class.inherited.classes.java
meta.class.java
source.java

And I have a theme rule:
{
"scope": "entity.other",
"settings": {
"foreground": "#D197D9"
}
},

And the Someclass ends up D197D9 (pinkish) in the Playground, but white in the Generic Code Editor in TM4E. According to TextMate selector semantics, a selector entity.other should match any scope beginning with entity.other.*, including entity.other.inherited-class.java. So, it should show up as pinkish in TM4E but it does not.

I can't find a way to see the scope inside of the Generic Code Editor (is there a way?) so I don't know what scope(s) it decided to apply to the element, and why it's not applying the scope rule from my theme.

I've attached my sample Java code that I'm using to test the grammar (and develop the theme itself), my tmTheme, and a copy of screenshots where you can see the divergence, not just in that scope, but in MANY other scopes too (method names, class names, variables, annotations, etc). Inspecting the screenshots you can see many differences: lots of things that should be the base white are colored aqua (annotations, classes after "new" keyword, method names, etc). Many Java class names that should be red are rendered in that same aqua keyword color. Generic types should be another color but are also the generic aqua color. variables should be a darker blue but are colored differently in TM4E depending on context. Some thigns are correct: comments, keywords, Javadoc and single line comments, quoted strings, etc. But most of the other things are not correct. If I were to guess, it looks like it's not applyting the proper scopes to the elements.

EclipseTM4EThemeTool.java

Dark-MyEclipse2.tmTheme.json

[sebthom]

I can't find a way to see the scope inside of the Generic Code Editor (is there a way?) so I don't know what scope(s) it decided to apply to the element, and why it's not applying the scope rule from my theme.

Did you try this? https://github.com/eclipse-tm4e/tm4e/blob/main/docs/adopter-guide.md#diagnostics-token-and-scope-hover

[jchamlin]

Thanks! That's super-useful. I just tried it. Attached screenshot. and here's the text:

BaseTool (8 chars)

Token Type: meta.java.other.definition.class.entity.inherited.classes.inherited-class
TextMate Scopes:
- source.java@org.eclipse.tm4e.language_pack
- meta.class.java
- meta.definition.class.inherited.classes.java
- entity.other.inherited-class.java

So, it says it has entity.other.inherited-class.java which should match my theme selector "entity.other" and color it pink. I even tried adding "entity.other.inherited-class" and "entity.other.inherited-class.java" with the same color, and it sitll didn't change colors (still white). I switched themes and back, and restarted Eclipse just to make sure. I don't have any of those other scopes defined in my theme (nohting source.* or meta.*). So, it should have matched my scope "entity.other". I'm not sure why it didn't. Interestingly enough, the default MyEclipse Dark theme does manage to make it pink. But my theme does not despite having what appears to be the right selector.

[jchamlin]

I just wanted to post an update. I managed to get TM4E working and the coloring to match DevStyle's Darkest Dark theme almost perfectly. The Java editor has access to some syntactic information that the Java TextMate grammar doesn't (or doesn't expose), so the Eclipse native Java editor can do a little bit more with coloring than TextMate, but in the end I got it to match about 95% of the colors and styling. And where it didn't match, I'm okay with the compromise. But what I had to do was a serious hacky workaround. For every generic style, like "constant.numeric" and "entity.other" and "keyword.operator" that failed to match (and it was about 80% of them) I added the specific TM4E Token Type to the selectors. There were often 3-10 more individual tokens types to add to each one, but afterwards, I got a pretty close render. At least for Java files. I'm sure this will break down on HTML, CSS, Python, and other languages because my theme won't have those token types in it. It would be better if T4ME just matched the generic selector, like it should, and like TextMate does. Here's one example of my workaround. This one is for the "extends BaseTool" example above. "entity.other" and 'entity.other.inherited-class" and "entity.other.inherited-class.java" did not match, but adding the very specifici TM4E token types got it to match. So my theme selector for that one looks like this:
"scope": [
"entity.other",
"entity.other.inherited-class.java",
"meta.definition.class.inherited.classes",
"meta.java.other.definition.class.entity.inherited.classes.inherited-class"
],
"settings": {
"foreground": "#79ABFF"
}

I've attached the full tmTheme file, so you can see which selectors failed and where I had to add the specific token types to get them to match. I've also attached a screenshot of the comparison: DevStyle Darkest Dark on the left, and TM4E runing my MyEclipse-Dark2.tmTheme.json file. It's very close.

Anyway, I look forward to a fix where I dont' have to use the TM4E specific token types to get a theme to apply.

Dark-MyEclipse2.tmTheme.json

[sebthom]

Can you please install this snapshot release https://github.com/eclipse-tm4e/tm4e/actions/runs/20494776077/artifacts/4963886435 (updatesite) and check if it fixes the issue. thanks!

[jchamlin]

It's better. But there are many few things that still aren't working right. Most seem to be related to choosing a less specific scope over a more specific scope.

entity.name.funciton should apply instead of entity.name to entity.name.function.java. But it's applying entity.name instead. The more specific scope should win.

punctuation.definition.string should apply to punctuation.definition.string.begin.java and punctuation.definition.string.end.java but it's not applying anything (not styled)

storage.type should apply instead storage to storage.type.java, but it's applying storage instead of storage.type.

storage.type.annotation should apply instead of storate or storage.type to storage.type.annotation.java, but its applying storage instead of storage.type.annotation.

storage.type.primitive should apply instead of storage.type to storage.type.primitive.java, but its applying storage.type instead of storage.type.primitive.

entity.other should apply to entity.other.inherited-class.java, but it's not applying anything (not styled).

storage.modifier.package should apply to storage.modifier.package.java but its applying storage.modifier instead.

There are many more, but I thought I would stop here, since many of the symtoms seem to be related to possibly the same root cause.

[jchamlin]

I guess I didn't actually update to 0.17.2, because MyEclipse 2025.2.0 (the latest release) has TM4E pinned to 0.14.1. So, when I went to install from your patched update site, it produced an error saying it couldn't be installed, resolved to a lower version, and didn't actually install 0.17.2. It installed 0.14.1. And I missed it. So, I wasn't able to actually test out your changes.

[sebthom]

I merged the PR already. The changes are now part of https://download.eclipse.org/tm4e/snapshots/
Please give it a try.

If you still cannot install it, then please try with a vanilla Eclipse distribution and not MyEclipse. Depending on how they installed TM4E it might not be possible to upgrade it.

[jchamlin]

I'm on Eclipse-2025 with the latest TM4E build (0.17.2) now. I haven't tested everything yet, I'm about halfway through, but so far things are looking much better. I'll post again with final results and let you know if there are any remaining issues. Thank you for the quick turnaround.