You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,7 +75,20 @@ Languages in the `native` class support all features
75
75
of this library and are extensively tested. `advanced` languages are tested but do not support the full feature set. Languages of the `basic` class are not tested and
76
76
only support the feature set of the backend. They can still be used for tokenization and AST parsing.
77
77
78
+
## How to contribute
79
+
**Your language is not natively supported by code.tokenize or the tokenization seems to be incorrect?** Then change it!
80
+
81
+
While code.tokenize is developed mainly as an helper library for internal research projects, we welcome pull requests of any sorts (if it is a new feature or a bug fix).
82
+
83
+
**Want to help to test more languages?**
84
+
Our goal is to support as many languages as possible at a `native` level. However, languages on `basic` level are completly untested. You can help by testing `basic` languages and reporting issues in the tokenization process!
85
+
78
86
## Release history
87
+
* 0.2.0
88
+
* Major API redesign!
89
+
* CHANGE: AST parsing is now done by an external library: [code_ast](https://github.com/cedricrupb/code_ast)
90
+
* CHANGE: Visitor pattern instead of custom tokenizer
91
+
* CHANGE: Custom visitors for language dependent tokenization
79
92
* 0.1.0
80
93
* The first proper release
81
94
* CHANGE: Language specific tokenizer configuration
0 commit comments