Skip to content

Commit b4d9cc4

Browse files
bstadlbauerbstadlbauer
authored
Add operator go client (#731)
1 parent 97dfbbc commit b4d9cc4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+5246
-0
lines changed

‎.gitattributes‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,5 @@
11
dask_kubernetes/_version.py export-subst
22
* text=auto
3+
4+
dask_kubernetes/operator/go_client/pkg/apis/kubernetes.dask.org/v1/zz_generated.deepcopy.go linguist-generated=true
5+
dask_kubernetes/operator/go_client/pkg/client/** linguist-generated=true

‎.github/workflows/update-go-package-version.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# As we're using CalVer, we need to update the major version of the go package every year
2+
# More info in this discussion: https://github.com/dask/dask-kubernetes/pull/731#discussion_r1226569925
3+
name: "Update go package version"
4+
5+
on:
6+
schedule:
7+
- cron: "0 0 2 1 *" # 00:00 UTC on the 2nd of January every year
8+
9+
jobs:
10+
update-major-go-version:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v2
14+
- name: Update go package version
15+
run: |
16+
PAST_YEAR=$(date -d "last year" '+%Y')
17+
CURRENT_YEAR=$(date '+%Y')
18+
sed -i "s/v${PAST_YEAR}/v${CURRENT_YEAR}/g" go.mod
19+
sed -i "s/v${PAST_YEAR}/v${CURRENT_YEAR}/g" dask_kubernetes/operator/go_client/hack/regenerate-code.sh
20+
sed -i "s/v${PAST_YEAR}/v${CURRENT_YEAR}/g" dask_kubernetes/operator/go_client/pkg/apis/kubernetes.dask.org/v1/register.go
21+
./dask_kubernetes/operator/go_client/hack/regenerate-code.sh
22+
- uses: peter-evans/create-pull-request@v4
23+
with:
24+
commit-message: "Update major go package version"
25+
title: "Update major go package version"
26+
body: "Update major go package version"
27+
branch: "update-go-package-version"
28+
branch-suffix: "timestamp"
29+
delete-branch: true

‎.pre-commit-config.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,32 @@ repos:
2222
language: script
2323
types: [yaml]
2424
pass_filenames: true
25+
- id: regenerate-go-code
26+
name: regenerate-go-code
27+
entry: dask_kubernetes/operator/go_client/hack/regenerate-code.sh
28+
language: script
29+
files: |
30+
(?x)^(
31+
go\.mod
32+
|go\.sum
33+
|dask_kubernetes/operator/go_client/pkg/apis/.*\.go
34+
)$
35+
exclude: ^dask_kubernetes/operator/go_client/pkg/apis/.*zz_generated.*\.go$
36+
pass_filenames: false
37+
- id: lint-go-code
38+
name: lint-go-code
39+
entry: dask_kubernetes/operator/go_client/hack/lint.sh
40+
language: script
41+
files: ^dask_kubernetes/operator/go_client/.*\.go$
42+
pass_filenames: false
43+
- id: build-go-code
44+
name: build-go-code
45+
entry: golang:1.19.10-bookworm bash -c 'GOCACHE=/src/.cache go build ./...'
46+
language: docker_image
47+
files: |
48+
(?x)^(
49+
go\.mod
50+
|go\.sum
51+
|dask_kubernetes/operator/go_client/.*\.go
52+
)$
53+
pass_filenames: false

‎CODEOWNERS‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,7 @@
11
* @jacobtomlinson
2+
dask_kubernetes/operator/customresources/ @jacobtomlinson @bstadlbauer
3+
4+
# Go operator
5+
go.mod @jacobtomlinson @bstadlbauer
6+
go.sum @jacobtomlinson @bstadlbauer
7+
dask_kubernetes/operator/go_client/ @jacobtomlinson @bstadlbauer

‎CONTRIBUTING.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,28 @@ For general information on how to contribute see https://docs.dask.org/en/latest
66

77
For local development, it is often desirable to run the scheduler process on one’s local machine & the workers on minikube.
88
For more information on setting this up see https://kubernetes.dask.org/en/latest/testing.html.
9+
10+
11+
### Dask operator - `go` client
12+
We're also releasing a `go` client for the dask operator. In case you make changes to the custom resource definitions in `dask_kubernetes/operator/customresources/`, please also check whether this has an
13+
effect on any of the types in `dask_kubernetes/operator/go_client/pkg/apis/kubernetes.dask.org/v1/types.go` and adapt them accordingly.
14+
15+
This being a Python repository, with a few (mostly auto-generated) `go` files, we've decided on an approach of linting an and code-generating `go` files within Docker. So in case you want to update the `go` files, no local `go` installation is required, but you will need to have `docker` ([docs](https://www.docker.com/)) installed.
16+
17+
#### Linting
18+
In case you would like to lint the `go` files, run
19+
```bash
20+
./dask_kubernetes/operator/go_client/hack/lint.sh
21+
```
22+
Linting will also happen automatically upon commit and on CI, using `pre-commit`.
23+
24+
#### Code generation
25+
In case you have changed the custom resource definitions, please port your changes to `dask_kubernetes/operator/go_client/pkg/apis/kubernetes.dask.org/v1/types.go` and then run
26+
27+
```bash
28+
./dask_kubernetes/operator/go_client/hack/regenerate-code.sh
29+
```
30+
to regenerate all the files. Code-generation will also happen automatically upon commit and on CI, using `pre-commit`.
31+
32+
#### Versioning
33+
`go` is very opinionated when it comes to versioning, especially regarding breaking changes. It's versioning system is designed around [Semantic Versioning](https://semver.org/) which is different from [Calendar Versioning](https://calver.org/) used throughout `dask`. In `go`, each major version is it's own package ([docs](https://go.dev/doc/modules/release-workflow#breaking)), which is why need to increase the version (i.e. the year from CalVer) in `./go.mod` as well as in `./dask_kubernetes/operator/go_client/hack/regenerate-code.sh` yearly for this work.

‎dask_kubernetes/operator/go_client/hack/Dockerfile‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
FROM golang:1.19.9
2+
3+
RUN git clone \
4+
--depth=1 \
5+
--branch v0.24.15 \
6+
https://github.com/kubernetes/code-generator.git \
7+
$GOPATH/src/k8s.io/code-generator
8+
9+
RUN touch $HOME/boilerplate.txt

‎dask_kubernetes/operator/go_client/hack/lint.sh‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/bin/bash -e
2+
3+
CURRENT_DIR=$(dirname ${BASH_SOURCE})
4+
REPO_DIR=$(realpath "$CURRENT_DIR/../../../..")
5+
6+
docker run \
7+
--rm \
8+
-v "${REPO_DIR}":/app \
9+
-v ~/.cache/golangci-lint/v1.53.2:/root/.cache \
10+
-w /app \
11+
golangci/golangci-lint:v1.53.2 \
12+
golangci-lint \
13+
run -v \
14+
--enable goimports \
15+
--enable gofmt \
16+
--timeout 3m0s \
17+
./...

‎dask_kubernetes/operator/go_client/hack/regenerate-code.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash -e
2+
3+
CURRENT_DIR=$(dirname ${BASH_SOURCE})
4+
REPO_DIR=$(realpath "$CURRENT_DIR/../../../..")
5+
MAJOR_VERSION="v2023"
6+
7+
PROJECT_MODULE="github.com/dask/dask-kubernetes/${MAJOR_VERSION}"
8+
IMAGE_NAME="kubernetes-codegen:latest"
9+
10+
CUSTOM_RESOURCE_NAME="kubernetes.dask.org"
11+
CUSTOM_RESOURCE_VERSION="v1"
12+
13+
echo "Building codegen Docker image..."
14+
docker build -f "${CURRENT_DIR}/Dockerfile" \
15+
-t "${IMAGE_NAME}" \
16+
.
17+
18+
echo "Generating client codes..."
19+
docker run --rm \
20+
-v "${REPO_DIR}:/go/src/${PROJECT_MODULE}" \
21+
-w "/go/src/${PROJECT_MODULE}" \
22+
"${IMAGE_NAME}" \
23+
/go/src/k8s.io/code-generator/generate-groups.sh \
24+
all \
25+
$PROJECT_MODULE/dask_kubernetes/operator/go_client/pkg/client \
26+
$PROJECT_MODULE/dask_kubernetes/operator/go_client/pkg/apis \
27+
$CUSTOM_RESOURCE_NAME:$CUSTOM_RESOURCE_VERSION \
28+
-h /root/boilerplate.txt

‎dask_kubernetes/operator/go_client/pkg/apis/kubernetes.dask.org/v1/doc.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
// +k8s:deepcopy-gen=package,register
2+
3+
// Package v1 is the v1 version of the API.
4+
// +groupName=kubernetes.dask.org
5+
package v1

‎dask_kubernetes/operator/go_client/pkg/apis/kubernetes.dask.org/v1/register.go‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
package v1
2+
3+
import (
4+
daskoperator "github.com/dask/dask-kubernetes/v2023/dask_kubernetes/operator/go_client/pkg/apis"
5+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
6+
"k8s.io/apimachinery/pkg/runtime"
7+
"k8s.io/apimachinery/pkg/runtime/schema"
8+
)
9+
10+
const (
11+
Version = "v1"
12+
)
13+
14+
var (
15+
SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
16+
AddToScheme = SchemeBuilder.AddToScheme
17+
)
18+
19+
// SchemeGroupVersion is the group version used to register these objects.
20+
var SchemeGroupVersion = schema.GroupVersion{Group: daskoperator.GroupName, Version: Version}
21+
22+
// Resource takes an unqualified resource and returns a Group-qualified GroupResource.
23+
func Resource(resource string) schema.GroupResource {
24+
return SchemeGroupVersion.WithResource(resource).GroupResource()
25+
}
26+
27+
// addKnownTypes adds the set of types defined in this package to the supplied scheme.
28+
func addKnownTypes(scheme *runtime.Scheme) error {
29+
scheme.AddKnownTypes(
30+
SchemeGroupVersion,
31+
&DaskCluster{},
32+
&DaskClusterList{},
33+
&DaskWorkerGroup{},
34+
&DaskWorkerGroupList{},
35+
&DaskJob{},
36+
&DaskJobList{},
37+
&DaskAutoscaler{},
38+
&DaskAutoscalerList{},
39+
)
40+
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
41+
return nil
42+
}

0 commit comments

Comments
 (0)