Skip to content

Master feature #5293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
casionone merged 22 commits into from
Nov 23, 2025
Merged

Master feature #5293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f5a82b9
update version to 1.8.0
casionone Aug 13, 2025
dac7e73
Upgrade `LINKIS_VERSION` to `1.8.0` to Fix Integration Test Failure i…
kazutoiris Aug 20, 2025
4ee7ea4
Fix KIND image loading, script typo, and cache directory creation (#5…
kazutoiris Aug 29, 2025
75f23a1
chore: prepare to release 1.8.0 (#5254)
kazutoiris Sep 2, 2025
7697578
support azure (#5214)
v-kkhuang Sep 22, 2025
7d58d35
Add OAuth2 authentication support (#5253)
kazutoiris Sep 28, 2025
59c29d7
fix azure compile error (#5264)
aiceflower Sep 30, 2025
feb03e5
remove default token (#5265)
aiceflower Oct 7, 2025
049ad8e
fix token security (#5266)
aiceflower Oct 8, 2025
cf6bc34
Merge upstream changes and apply additional improvements (#5279)
kazutoiris Oct 30, 2025
7a63e0f
docs(readme): remove underlines between badges (#5283)
kazutoiris Oct 31, 2025
2a3d1b1
feat(ci): add support for spark and hive engine integration tests (#5…
kazutoiris Nov 21, 2025
b87f07a
merge 1.9.0
kinghao007 Nov 21, 2025
2e52cac
fix compile error
kinghao007 Nov 21, 2025
83a64e1
add licence
kinghao007 Nov 21, 2025
c0223f8
fix third-party dependencies
kinghao007 Nov 21, 2025
d35117a
merge web
kinghao007 Nov 22, 2025
94752a2
optimize pythonModule
kinghao007 Nov 22, 2025
e2aca04
Update version to 1.8.0
aiceflower Nov 22, 2025
b996e38
Update package.json
aiceflower Nov 22, 2025
ae9238c
Update NOTICE
aiceflower Nov 22, 2025
c807674
Merge pull request #5294 from WeDataSphere/master-feature-web
aiceflower Nov 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
11 changes: 7 additions & 4 deletions .github/workflows/dlc.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
{
"ignorePatterns": [
{
"pattern": "^http://localhost"
"ignorePatterns": [
{
"pattern": "^http://localhost"
},
{
"pattern": "^http://127.0.0.1"
},
{
"pattern": "^http://127.0.0.1"
"pattern": "^(https?://)?([a-zA-Z0-9-]+\\.)*bilibili\\.com"
}
],
"timeout": "10s",
Expand Down
15 changes: 8 additions & 7 deletions .github/workflows/integration-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,8 @@ jobs:
sleep 20
done

bash ./linkis-dist/helm/scripts/prepare-for-spark.sh

#show linkis pod logs
#POD_NAME=`kubectl get pods -n linkis -l app.kubernetes.io/instance=linkis-demo-cg-linkismanager -o jsonpath='{.items[0].metadata.name}'`
#kubectl logs -n linkis ${POD_NAME} -f --tail=10000
Expand All @@ -170,16 +172,15 @@ jobs:

# Execute test by linkis-cli
POD_NAME=`kubectl get pods -n linkis -l app.kubernetes.io/instance=linkis-demo-mg-gateway -o jsonpath='{.items[0].metadata.name}'`
kubectl exec -it -n linkis ${POD_NAME} -- bash -c " \
kubectl exec -n linkis ${POD_NAME} -- bash -c " \
sh /opt/linkis/bin/linkis-cli -engineType shell-1 -codeType shell -code \"pwd\" ";

kubectl exec -it -n linkis ${POD_NAME} -- bash -c " \
kubectl exec -n linkis ${POD_NAME} -- bash -c " \
sh /opt/linkis/bin/linkis-cli -engineType python-python2 -codeType python -code 'print(\"hello\")' "

#todo
#kubectl exec -it -n linkis ${POD_NAME} -- bash -c " \
#sh /opt/linkis/bin/linkis-cli -engineType hive-3.1.3 -codeType hql -code 'show databases' "
kubectl exec -n linkis ${POD_NAME} -- bash -c " \
sh /opt/linkis/bin/linkis-cli -engineType hive-3.1.3 -codeType hql -code 'show databases' "

#kubectl exec -it -n linkis ${POD_NAME} -- bash -c " \
#sh /opt/linkis/bin/linkis-cli -engineType spark-3.2.1 -codeType sql -code 'show databases' "
kubectl exec -n linkis ${POD_NAME} -- bash -c " \
sh /opt/linkis/bin/linkis-cli -engineType spark-3.2.1 -codeType sql -code 'show databases' "
shell: bash
109 changes: 54 additions & 55 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
</h2>

<p align="center">
<strong> Linkis builds a computation middleware layer to facilitate connection,
<strong> Linkis builds a computation middleware layer to facilitate connection,
governance and orchestration between the upper applications and the underlying data engines. </strong>
</p>
<p align="center">
Expand All @@ -14,41 +14,40 @@
</p>

<p align="center">
<a href="https://linkis.apache.org/docs/latest/introduction/" >
<img src="https://img.shields.io/badge/document-English-blue.svg" alt="EN docs" />
</a>
<a href="https://linkis.apache.org/zh-CN/docs/latest/introduction/">
<img src="https://img.shields.io/badge/文档-简体中文-blue.svg" alt="简体中文文档" />
</a>
<a href="https://linkis.apache.org/docs/latest/introduction/" ><!--
--><img src="https://img.shields.io/badge/document-English-blue.svg" alt="EN docs" /><!--
--></a>
<a href="https://linkis.apache.org/zh-CN/docs/latest/introduction/"><!--
--><img src="https://img.shields.io/badge/文档-简体中文-blue.svg" alt="简体中文文档" /><!--
--></a>
</p>

<p align="center">
<a target="_blank" href="https://search.maven.org/search?q=g:org.apache.linkis%20AND%20a:linkis">
<img src="https://img.shields.io/maven-central/v/org.apache.linkis/linkis.svg?label=maven%20central" />
</a>
<a target="_blank" href="https://github.com/apache/linkis/blob/master/LICENSE">
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg?label=license" />
</a>
<a target="_blank" href="https://www.oracle.com/technetwork/java/javase/downloads/index.html">
<img src="https://img.shields.io/badge/JDK-8-green.svg" />
</a>
<a target="_blank" href="https://github.com/apache/linkis/actions">
<img src="https://github.com/apache/linkis/actions/workflows//build-backend.yml/badge.svg" />
</a>

<a target="_blank" href='https://github.com/apache/linkis'>
<img src="https://img.shields.io/github/forks/apache/linkis.svg" alt="github forks"/>
</a>
<a target="_blank" href='https://github.com/apache/linkis'>
<img src="https://img.shields.io/github/stars/apache/linkis.svg" alt="github stars"/>
</a>
<a target="_blank" href='https://github.com/apache/linkis'>
<img src="https://img.shields.io/github/contributors/apache/linkis.svg" alt="github contributors"/>
</a>
<a target="_blank" href="https://badges.toozhao.com/stats/01G7TRNN1PH9PMSCYWDF3EK4QT">
<img src="https://badges.toozhao.com/badges/01G7TRNN1PH9PMSCYWDF3EK4QT/green.svg" />
</a>

<a target="_blank" href="https://search.maven.org/search?q=g:org.apache.linkis%20AND%20a:linkis"><!--
--><img src="https://img.shields.io/maven-central/v/org.apache.linkis/linkis.svg?label=maven%20central" /><!--
--></a>
<a target="_blank" href="https://github.com/apache/linkis/blob/master/LICENSE"><!--
--><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg?label=license" /><!--
--></a>
<a target="_blank" href="https://www.oracle.com/technetwork/java/javase/downloads/index.html"><!--
--><img src="https://img.shields.io/badge/JDK-8-green.svg" /><!--
--></a>
<a target="_blank" href="https://github.com/apache/linkis/actions"><!--
--><img src="https://github.com/apache/linkis/actions/workflows//build-backend.yml/badge.svg" /><!--
--></a>
<a target="_blank" href='https://github.com/apache/linkis'><!--
--><img src="https://img.shields.io/github/forks/apache/linkis.svg" alt="github forks"/><!--
--></a>
<a target="_blank" href='https://github.com/apache/linkis'><!--
--><img src="https://img.shields.io/github/stars/apache/linkis.svg" alt="github stars"/><!--
--></a>
<a target="_blank" href='https://github.com/apache/linkis'><!--
--><img src="https://img.shields.io/github/contributors/apache/linkis.svg" alt="github contributors"/><!--
--></a>
<a target="_blank" href="https://badges.toozhao.com/stats/01G7TRNN1PH9PMSCYWDF3EK4QT"><!--
--><img src="https://badges.toozhao.com/badges/01G7TRNN1PH9PMSCYWDF3EK4QT/green.svg" /><!--
--></a>

</p>
<br/>

Expand Down Expand Up @@ -89,21 +88,21 @@ Apache Linkis | DeepWiki : https://deepwiki.com/apache/linkis

# Engine Type

| **Engine name** | **Support underlying component version<br/>(default dependency version)** | **Linkis Version Requirements** | **Included in Release Package By Default** | **Description** |
|:---- |:---- |:---- |:---- |:---- |
|Spark|Apache >= 2.0.0, <br/>CDH >= 5.4.0, <br/>(default Apache Spark 3.2.1)|\>=1.0.3|Yes|Spark EngineConn, supports SQL , Scala, Pyspark and R code|
|Hive|Apache >= 1.0.0, <br/>CDH >= 5.4.0, <br/>(default Apache Hive 3.1.3)|\>=1.0.3|Yes |Hive EngineConn, supports HiveQL code|
|Python|Python >= 2.6, <br/>(default Python2*)|\>=1.0.3|Yes |Python EngineConn, supports python code|
|Shell|Bash >= 2.0|\>=1.0.3|Yes|Shell EngineConn, supports Bash shell code|
|JDBC|MySQL >= 5.0, Hive >=1.2.1, <br/>(default Hive-jdbc 2.3.4)|\>=1.0.3|No |JDBC EngineConn, already supports ClickHouse, DB2, DM, Greenplum, kingbase, MySQL, Oracle, PostgreSQL and SQLServer, can be extended quickly Support other DB, such as SQLite|
|Flink |Flink >= 1.12.2, <br/>(default Apache Flink 1.12.2)|\>=1.0.2|No |Flink EngineConn, supports FlinkSQL code, also supports starting a new Yarn in the form of Flink Jar Application|
|Pipeline|-|\>=1.0.2|No|Pipeline EngineConn, supports file import and export|
|openLooKeng|openLooKeng >= 1.5.0, <br/>(default openLookEng 1.5.0)|\>=1.1.1|No|openLooKeng EngineConn, supports querying data virtualization engine with Sql openLooKeng|
|Sqoop| Sqoop >= 1.4.6, <br/>(default Apache Sqoop 1.4.6)|\>=1.1.2|No|Sqoop EngineConn, support data migration tool Sqoop engine|
|Presto|Presto >= 0.180|\>=1.2.0|No|Presto EngineConn, supports Presto SQL code|
|ElasticSearch|ElasticSearch >=6.0|\>=1.2.0|No|ElasticSearch EngineConn, supports SQL and DSL code|
|Trino | Trino >=371 | >=1.3.1 | No | Trino EngineConn， supports Trino SQL code |
|Seatunnel | Seatunnel >=2.1.2 | >=1.3.1 | No | Seatunnel EngineConn， supportt Seatunnel SQL code |
| **Engine name** | **Support underlying component version<br/>(default dependency version)** | **Linkis Version Requirements** | **Included in Release Package By Default** | **Description** |
| :-------------- | :------------------------------------------------------------------------ | :------------------------------ | :----------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Spark | Apache >= 2.0.0, <br/>CDH >= 5.4.0, <br/>(default Apache Spark 3.2.1) | \>=1.0.3 | Yes | Spark EngineConn, supports SQL , Scala, Pyspark and R code |
| Hive | Apache >= 1.0.0, <br/>CDH >= 5.4.0, <br/>(default Apache Hive 3.1.3) | \>=1.0.3 | Yes | Hive EngineConn, supports HiveQL code |
| Python | Python >= 2.6, <br/>(default Python2*) | \>=1.0.3 | Yes | Python EngineConn, supports python code |
| Shell | Bash >= 2.0 | \>=1.0.3 | Yes | Shell EngineConn, supports Bash shell code |
| JDBC | MySQL >= 5.0, Hive >=1.2.1, <br/>(default Hive-jdbc 2.3.4) | \>=1.0.3 | No | JDBC EngineConn, already supports ClickHouse, DB2, DM, Greenplum, kingbase, MySQL, Oracle, PostgreSQL and SQLServer, can be extended quickly Support other DB, such as SQLite |
| Flink | Flink >= 1.12.2, <br/>(default Apache Flink 1.12.2) | \>=1.0.2 | No | Flink EngineConn, supports FlinkSQL code, also supports starting a new Yarn in the form of Flink Jar Application |
| Pipeline | - | \>=1.0.2 | No | Pipeline EngineConn, supports file import and export |
| openLooKeng | openLooKeng >= 1.5.0, <br/>(default openLookEng 1.5.0) | \>=1.1.1 | No | openLooKeng EngineConn, supports querying data virtualization engine with Sql openLooKeng |
| Sqoop | Sqoop >= 1.4.6, <br/>(default Apache Sqoop 1.4.6) | \>=1.1.2 | No | Sqoop EngineConn, support data migration tool Sqoop engine |
| Presto | Presto >= 0.180 | \>=1.2.0 | No | Presto EngineConn, supports Presto SQL code |
| ElasticSearch | ElasticSearch >=6.0 | \>=1.2.0 | No | ElasticSearch EngineConn, supports SQL and DSL code |
| Trino | Trino >=371 | >=1.3.1 | No | Trino EngineConn， supports Trino SQL code |
| Seatunnel | Seatunnel >=2.1.2 | >=1.3.1 | No | Seatunnel EngineConn， supportt Seatunnel SQL code |

# Download

Expand Down Expand Up @@ -152,13 +151,13 @@ npm run build
```

### Bundled with MySQL JDBC Driver
Due to the MySQL licensing restrictions, the MySQL Java Database Connectivity (JDBC) driver is not bundled with the
Due to the MySQL licensing restrictions, the MySQL Java Database Connectivity (JDBC) driver is not bundled with the
official released linkis image by default. However, at current stage, linkis still relies on this library to work properly.
To solve this problem, we provide a script which can help to creating a custom image with mysql jdbc from the official
To solve this problem, we provide a script which can help to creating a custom image with mysql jdbc from the official
linkis image by yourself, the image created by this tool will be tagged as `linkis:with-jdbc` by default.

```shell
$> LINKIS_IMAGE=linkis:1.3.1
$> LINKIS_IMAGE=linkis:1.3.1
$> ./linkis-dist/docker/scripts/make-linkis-image-with-mysql-jdbc.sh
```

Expand All @@ -167,7 +166,7 @@ Please refer to [Quick Deployment](https://linkis.apache.org/docs/latest/deploym

# Examples and Guidance
- [User Manual](https://linkis.apache.org/docs/latest/user-guide/how-to-use)
- [Engine Usage Documents](https://linkis.apache.org/docs/latest/engine-usage/overview)
- [Engine Usage Documents](https://linkis.apache.org/docs/latest/engine-usage/overview)
- [API Documents](https://linkis.apache.org/docs/latest/api/overview)

# Documentation & Vedio
Expand All @@ -186,19 +185,19 @@ Below is the Linkis architecture diagram. You can find more detailed architectur

# Contributing

Contributions are always welcomed, we need more contributors to build Linkis together. either code, or doc, or other supports that could help the community.
Contributions are always welcomed, we need more contributors to build Linkis together. either code, or doc, or other supports that could help the community.
For code and documentation contributions, please follow the [contribution guide](https://linkis.apache.org/community/how-to-contribute).

# Contact Us


- Any questions or suggestions please kindly submit an [issue](https://github.com/apache/linkis/issues).
- Any questions or suggestions please kindly submit an [issue](https://github.com/apache/linkis/issues).
- By mail [dev@linkis.apache.org](mailto:dev@linkis.apache.org)
- You can scan the QR code below to join our WeChat group to get more immediate response

<img src="https://linkis.apache.org/Images/wedatasphere_contact_01.png" width="256"/>

# Who is Using Linkis

We opened an issue [[Who is Using Linkis]](https://github.com/apache/linkis/issues/23) for users to feedback and record who is using Linkis.
We opened an issue [[Who is Using Linkis]](https://github.com/apache/linkis/issues/23) for users to feedback and record who is using Linkis.
Since the first release of Linkis in 2019, it has accumulated more than **700** trial companies and **1000+** sandbox trial users, which involving diverse industries, from finance, banking, tele-communication, to manufactory, internet companies and so on.
Loading
Loading