Skip to content

Commit 5f5dc55

Browse files
committed
Preparation for 0.2.0 release
1 parent ce2661c commit 5f5dc55

File tree

2 files changed

+16
-2
lines changed

2 files changed

+16
-2
lines changed

README.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,20 @@ Languages in the `native` class support all features
7575
of this library and are extensively tested. `advanced` languages are tested but do not support the full feature set. Languages of the `basic` class are not tested and
7676
only support the feature set of the backend. They can still be used for tokenization and AST parsing.
7777

78+
## How to contribute
79+
**Your language is not natively supported by code.tokenize or the tokenization seems to be incorrect?** Then change it!
80+
81+
While code.tokenize is developed mainly as an helper library for internal research projects, we welcome pull requests of any sorts (if it is a new feature or a bug fix).
82+
83+
**Want to help to test more languages?**
84+
Our goal is to support as many languages as possible at a `native` level. However, languages on `basic` level are completly untested. You can help by testing `basic` languages and reporting issues in the tokenization process!
85+
7886
## Release history
87+
* 0.2.0
88+
* Major API redesign!
89+
* CHANGE: AST parsing is now done by an external library: [code_ast](https://github.com/cedricrupb/code_ast)
90+
* CHANGE: Visitor pattern instead of custom tokenizer
91+
* CHANGE: Custom visitors for language dependent tokenization
7992
* 0.1.0
8093
* The first proper release
8194
* CHANGE: Language specific tokenizer configuration

setup.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
setup(
77
name = 'code_tokenize',
88
packages = ['code_tokenize'],
9-
version = '0.1.0',
9+
version = '0.2.0',
1010
license='MIT',
1111
description = 'Fast program tokenization and structural analysis in Python',
1212
long_description = long_description,
@@ -19,7 +19,8 @@
1919
install_requires=[
2020
'tree_sitter',
2121
'GitPython',
22-
'requests'
22+
'requests',
23+
'code-ast'
2324
],
2425
classifiers=[
2526
'Development Status :: 3 - Alpha',

0 commit comments

Comments
 (0)