通过 Operator 的方案，可以对 Kubernetes 的功能进行友好地扩展。Operatpr = CRD + Controller。首先通过 yaml 定义，生成 CRD ，然后 Controller 不断地监听 etcd 中的数据，执行相应动作。开发 Operator 时，有很多繁琐且重复的事情。KubeBuilder 可以帮助我们快速生成骨架代码，开发一个 Kubernetes 的扩展功能， 更多介绍可以参考文档：Kubernetes 复杂有状态应用管理框架 – Operator 。本篇文档，主要是尝试使用 KubeBuilder 开发一个 Operator 。
1. 环境准备
我准备的是，单节点 Kubernetes 1.15.3 。
将集群 /etc/kubernetes/admin.conf 拷贝到本地 ~/.kube/config 即可。
2. Hello，Kubebuilder
2.1 安装 kubebuilder、kustomize
以 OS X 为例，当前 kubebuilder 版本为 2.x :
1
2
| brew install kubebuilder
brew install kustomize
|
2.2 初始化项目
1
2
3
4
| export GOPATH=$(go env GOPATH)
mkdir -p $GOPATH/src/github.com/kube-api
cd $GOPATH/src/github.com/kube-api
kubebuilder init --domain k8s.chenshaowen.com --license apache2 --owner "chenshaowen"
|
初始化工程之后，KubeBuilder 会生成一系列配置和代码框架。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| tree -L 3
.
├── Dockerfile
├── Makefile
├── PROJECT
├── bin
│ └── manager
├── config
│ ├── certmanager
│ │ ├── certificate.yaml
│ │ ├── kustomization.yaml
│ │ └── kustomizeconfig.yaml
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ ├── manager_prometheus_metrics_patch.yaml
│ │ ├── manager_webhook_patch.yaml
│ │ └── webhookcainjection_patch.yaml
│ ├── manager
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── rbac
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ └── role_binding.yaml
│ └── webhook
│ ├── kustomization.yaml
│ ├── kustomizeconfig.yaml
│ └── service.yaml
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go
8 directories, 28 files
|
2.3. 新增 API
1
2
3
4
5
6
| kubebuilder create api --group groupa --version v1beta1 --kind ApiExampleA
Create Resource [y/n]
y
Create Controller [y/n]
y
......
|
KubeBuilder 会将 CRD 和 Controller 添加到工程中。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
| tree -L 3
.
├── Dockerfile
├── Makefile
├── PROJECT # 新增 resources 部分
├── api
│ └── v1beta1 # 新增 API 描述
│ ├── apiexamplea_types.go
│ ├── groupversion_info.go
│ └── zz_generated.deepcopy.go
├── bin
│ └── manager
├── config
│ ├── certmanager
│ │ ├── certificate.yaml
│ │ ├── kustomization.yaml
│ │ └── kustomizeconfig.yaml
│ ├── crd # 新增 CRD 定义
│ │ ├── kustomization.yaml
│ │ ├── kustomizeconfig.yaml
│ │ └── patches
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ ├── manager_prometheus_metrics_patch.yaml
│ │ ├── manager_webhook_patch.yaml
│ │ └── webhookcainjection_patch.yaml
│ ├── manager
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── rbac
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ └── role_binding.yaml
│ ├── samples # 新增创建 CRD 对象示例
│ │ └── groupa_v1beta1_apiexamplea.yaml
│ └── webhook
│ ├── kustomization.yaml
│ ├── kustomizeconfig.yaml
│ └── service.yaml
├── controllers # 新增 Controller
│ ├── apiexamplea_controller.go
│ └── suite_test.go
├── go.mod # 新增依赖包
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go # 新增处理逻辑
14 directories, 36 files
|
2.4 修改配置，适配国内环境
在容器中进行编译时，由于国内网络问题，会导致拉取不到 Go 依赖包和依赖镜像，需要修改 Dockerfile 文件。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| @@ -7,7 +7,7 @@ COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
-RUN go mod download
+RUN GOPROXY=https://gocenter.io go mod download
# Copy the go source
COPY main.go main.go
@@ -19,7 +19,7 @@ RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager
# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
-FROM gcr.io/distroless/static:latest
+FROM gcr.azk8s.cn/distroless/static:latest
WORKDIR /
COPY --from=builder /workspace/manager .
ENTRYPOINT ["/manager"]
|
2.5. 构建并推送镜像测试
- 本地构建并推送镜像
由于使用的是远程 Kubernetes 环境，需要借助镜像仓库进行部署。
1
2
3
4
5
6
7
| @@ -1,6 +1,6 @@
# Image URL to use all building/pushing image targets
-IMG ?= controller:latest
+IMG ?= docker.io/shaowenchen/controller:latest
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:trivialVersions=true"
|
1
2
3
| docker login docker.io -u shaowenchen
Password:
Login Succeeded
|
1
| make docker-build & make docker-push
|
也可以通过，在执行 make docker-build 等命令时，增加 IMG 变量修改镜像名称。
- 部署到 Kubernetes
1
2
3
| kubectl get crd
NAME CREATED AT
apiexampleas.groupa.k8s.chenshaowen.com 2019-09-24T07:24:45Z
|
1
2
3
| kubectl get deploy -n kube-api-system
NAME READY UP-TO-DATE AVAILABLE AGE
kube-api-controller-manager 1/1 1 1 46s
|
1
| kubectl apply -f config/samples/groupa_v1beta1_apiexamplea.yaml
|
1
2
3
4
| kubectl get apiexampleas.groupa.k8s.chenshaowen.com
NAME AGE
apiexamplea-sample 61s
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl get apiexampleas.groupa.k8s.chenshaowen.com apiexamplea-sample -o yaml
apiVersion: groupa.k8s.chenshaowen.com/v1beta1
kind: ApiExampleA
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"groupa.k8s.chenshaowen.com/v1beta1","kind":"ApiExampleA","metadata":{"annotations":{},"name":"apiexamplea-sample","namespace":"default"},"spec":{"foo":"bar"}}
creationTimestamp: "2019-09-24T07:29:16Z"
generation: 1
name: apiexamplea-sample
namespace: default
resourceVersion: "635450"
selfLink: /apis/groupa.k8s.chenshaowen.com/v1beta1/namespaces/default/apiexampleas/apiexamplea-sample
uid: 05398ab4-7d4a-4f2e-af30-b59e61680c7e
spec:
foo: bar
|
3. 在 Project 中写入逻辑
通过上面的操作，我们新增了 Kubernetes 对象类型 apiexampleas.groupa.k8s.chenshaowen.com （ApiExampleA），并实例化对象进行操作。
这些操作仅仅只是对 etcd 数据的操作，没有触发有效动作。下面，尝试往工程中注入一点自定义逻辑。实现一个简单的功能：给自定义的 CRD 增加两个字段，FirstName、SecondName ；创建对象时，在 Controller 中获取这两个字段，并输出到日志中。
- 修改代码
在 api/v1beta1/apiexamplea_types.go 文件 ApiExampleASpec 结构体中增加两个字段：
1
2
3
4
5
6
| type ApiExampleASpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
FirstName string `json:"firstname"` // add
SecondName string `json:"secondname"` // add
}
|
首先在 import 中新增 log 包：
1
2
3
| import (
"log" // add
...
|
然后在 Reconcile 函数中，处理逻辑
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| func (r *ApiExampleAReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
// _ = context.Background()
// _ = r.Log.WithValues("apiexamplea", req.NamespacedName)
// // your logic here
// return ctrl.Result{}, nil
ctx := context.Background()
_ = r.Log.WithValues("apiexamplea", req.NamespacedName)
obja := &groupav1beta1.ApiExampleA{}
if err := r.Get(ctx, req.NamespacedName, obja); err != nil {
log.Println(err, "unable to fetch New Object")
} else {
log.Println("fetch New Object:", obja.Spec.FirstName, obja.Spec.SecondName)
}
return ctrl.Result{}, nil
}
|
- 修改镜像 tag
修改镜像 tag 是为了远程 Kubernetes 集群远程部署 Deployment 时，能够重新拉取镜像，使用指定的镜像进行部署。
修改 Makefile 文件中的变量 IMG 为：IMG ?= docker.io/shaowenchen/controller:1
- 构建发布
1
| make & make docker-build & make docker-push
|
- 部署
- 生成一个 CRD 对象实例
在 config/samples/groupa_v1beta1_apiexamplea.yaml 文件中修改 name 值，并在 spec 字段新增两个字段：
1
2
3
4
5
6
7
8
9
| apiVersion: groupa.k8s.chenshaowen.com/v1beta1
kind: ApiExampleA
metadata:
name: apiexamplea-sample2
spec:
# Add fields here
# foo: bar
firstname: shaowen
secondname: chen
|
创建 CRD 对象
1
| kubectl create -f config/samples/groupa_v1beta1_apiexamplea.yaml
|
查看 Controller 的 Pod Name
1
2
3
| kubectl get pod -n kube-api-system
NAME READY STATUS RESTARTS AGE
kube-api-controller-manager-7d8bb9fc6f-8bmg9 2/2 Running 0 22m
|
查看创建日志
1
2
3
4
5
6
7
8
9
| kubectl logs kube-api-controller-manager-7d8bb9fc6f-8bmg9 -c manager -n kube-api-system
2019-09-25T07:09:51.124Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2019-09-25T07:09:51.216Z INFO controller-runtime.controller Starting EventSource {"controller": "apiexamplea", "source": "kind source: /, Kind="}
2019-09-25T07:09:51.217Z INFO setup starting manager
2019-09-25T07:09:51.217Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2019-09-25T07:10:08.111Z INFO controller-runtime.controller Starting Controller {"controller": "apiexamplea"}
2019-09-25T07:10:08.111Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"kube-api-system","name":"controller-leader-election-helper","uid":"bf307b9a-829f-478e-9306-68b6c47671fa","apiVersion":"v1","resourceVersion":"871886"}, "reason": "LeaderElection", "message": "kube-api-controller-manager-7d8bb9fc6f-8bmg9_77815cc0-df63-11e9-b8c4-e6f6cfac380f became leader"}
2019-09-25T07:10:08.211Z INFO controller-runtime.controller Starting workers {"controller": "apiexamplea", "worker count": 1}
2019/09/25 07:10:08 fetch New Object: shaowen chen
|
4. 参考
