@@ -7,12 +7,50 @@ It is inspired by and re-uses much of the Kubernetes controller plumbing from th
77## Use Cases
88
99` k8s-dt-node-labeller ` was developed in order to facilitate targeted deployments into hybrid (e.g. armhf, arm64)
10- Kubernetes clusters, as well as for targeting heterogeneous accelerators in Edge deployments.
10+ Kubernetes clusters, as well as for targeting heterogeneous accelerators in Edge deployments - specifically those that
11+ show up as platform devices (FPGAs, embedded GPUs, etc.) as opposed to those that are dynamically discoverable at
12+ run-time via bus enumeration (e.g. USB, PCI).
1113
14+ ## Overview
15+
16+ ` k8s-dt-node-labeller ` is primarily concerned with top-level system characteristics, and therefore limits itself to
17+ walking through the top-level ` devicetree ` node looking for ` compatible ` properties to expose as Kubernetes node labels.
18+ As ` k8s-dt-node-labeller ` requires access to the target node's ` devicetree ` instance, the labeller must be run directly
19+ on the target node that is to be labelled. An overview of the general labelling flow (in the context of a Kubernetes
20+ DaemonSet) is provided below:
21+
22+ ![ Overview Diagram] [ overview ]
23+
24+ It is expected that the node labeller is run when the node is registered with the cluster, such that pods relying on
25+ specific labels are able to be scheduled and placed appropriately.
26+
27+ Labelling is possible in a number of different ways:
28+ - Running Node-Local via the CLI App
29+ - Running Node-Local via Docker
30+ - Deploying as a Kubernetes DaemonSet
31+
32+ Regardless of the method of deployment, the labels will be generated and applied to the local node.
33+
34+ ### Limitations
35+
36+ ` k8s-dt-node-labeller ` only generates labels from properties at the top-level of the ` devicetree ` and those specifically
37+ defined by the user by design. It does not recurse down the tree, nor does it make any attempt to enumerate busses that
38+ are better served by bus-specific device detection logic. While the labeller can attest to the existence of a node in
39+ the ` devicetree ` , it offers no device-specific information or control - all of which would need to be implemented
40+ through a device-specific device plugin.
41+
42+ ## Installation
43+
44+ If planning to use the CLI App directly, this can be compiled and installed as other Go projects:
45+
46+ ```
47+ $ go get github.com/adaptant-labs/k8s-dt-node-labeller
48+ ```
49+
50+ For Docker containers and Kubernetes deployment instructions, see below.
1251## Usage
1352
14- The node labeller is expected to be run node-local, and will need to be invoked on each individual node
15- requiring its own specific devicetree parsing and labelling.
53+ General usage is as follows:
1654
1755```
1856$ k8s-dt-node-labeller --help
@@ -24,10 +62,9 @@ Usage: k8s-dt-node-labeller [flags] [-n devicetree nodes...]
2462 Paths to a kubeconfig. Only required if out-of-cluster.
2563 -n string
2664 Additional devicetree node names
27-
2865```
2966
30- By default, ` compatible ` strings from the top-level ` / ` node are discovered and converted to node labels.
67+ A dry-run is possible by specifying the ` -d ` flag:
3168
3269```
3370$ k8s-dt-node-labeller -d
@@ -37,7 +74,8 @@ beta.devicetree.org/nvidia-jetson-nano: 1
3774beta.devicetree.org/nvidia-tegra210: 1
3875```
3976
40- additional node specifications are possible via the ` -n ` flag, as below:
77+ By default, ` compatible ` strings from the top-level ` / ` node are discovered and converted to node labels. Additional
78+ node specifications are possible via the ` -n ` flag, as below:
4179
4280```
4381$ k8s-dt-node-labeller -d -n gpu pwm-fan
@@ -50,6 +88,11 @@ beta.devicetree.org/nvidia-gm20b: 1
5088beta.devicetree.org/pwm-fan: 1
5189```
5290
91+ In the case of Docker or Kubernetes Deployment, note that these arguments can also be passed on to the container
92+ directly.
93+
94+ ### Running Node-Local via the CLI App
95+
5396In order to submit the labels to the cluster, ensure a valid kubeconfig can be found (e.g. by setting the ` KUBECONFIG `
5497environment variable, or through specifying the location with the ` -kubeconfig ` parameter) and execute the node
5598labeller as-is:
@@ -67,7 +110,35 @@ $ k8s-dt-node-labeller
67110...
68111```
69112
70- After which the node labels can be viewed from ` kubectl ` :
113+ ### Running Node-Local via Docker
114+
115+ Multi-arch Docker images are available on Docker Hub at [ adaptant/k8s-dt-node-labeller] . These may be run as-is
116+ in-cluster, or out of cluster with an appropriate ` KUBECONFIG ` passed through.
117+
118+ Note that as the labeller requires access to the host's ` /sys/firmware ` directory, this must either be passed through
119+ explicitly, or the container run in privileged mode:
120+
121+ ```
122+ $ docker run --privileged adaptant/k8s-dt-node-labeller
123+ ...
124+ ```
125+
126+ ### Deploying as a Kubernetes DaemonSet
127+
128+ An example deployment configuration for a DaemonSet is provided in ` k8s-dt-labeller-ds.yaml ` , which can be directly
129+ applied to the running cluster:
130+
131+ ```
132+ $ kubectl apply -f k8s-dt-labeller-ds.yaml
133+ ```
134+
135+ This will create a special ` dt-labeller ` service account, cluster role, and binding with the permission to list and
136+ reconcile nodes. Note that as the labeller requires access to an unmasked ` /sys/firmware ` , it must also be run in a
137+ privileged securityContext.
138+
139+ ### Verifying Labelling
140+
141+ After the labeller has run, the node labels can be viewed from ` kubectl ` :
71142
72143```
73144$ kubectl describe node jetson-nano
@@ -87,6 +158,10 @@ Labels: beta.devicetree.org/nvidia-jetson-nano=1
87158 ...
88159```
89160
161+ ## Features and bugs
162+
163+ Please file feature requests and bugs in the [ issue tracker] [ tracker ] .
164+
90165## Acknowledgements
91166
92167This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant
@@ -97,6 +172,9 @@ agreement No 825480 ([SODALITE]).
97172` k8s-dt-node-labeller ` is licensed under the terms of the Apache 2.0 license, the full
98173version of which can be found in the LICENSE file included in the distribution.
99174
175+ [ tracker ] : https://github.com/adaptant-labs/k8s-dt-node-labeller/issues
100176[ devicetree ] : https://www.devicetree.org
101177[ SODALITE ] : https://www.sodalite.eu
102- [ amdgpu-node-labeller ] : https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller
178+ [ overview ] : https://raw.githubusercontent.com/adaptant-labs/k8s-dt-node-labeller/master/overview.png
179+ [ amdgpu-node-labeller ] : https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/master/cmd/k8s-node-labeller
180+ [ adaptant/k8s-dt-node-labeller ] : https://hub.docker.com/repository/docker/adaptant/k8s-dt-node-labeller
0 commit comments