在說 operator 之前讓我們先來了解下什麼是 CRD (自定義資源描述,Custom Resource Definition)。
CRD
在 Kubernetes 中我們使用的 Deployment, DamenSet,StatefulSet, Service,Ingress, ConfigMap, Secret 這些都是資源,而對這些資源的創建、更新、刪除的動作都會被成為為事件(Event),Kubernetes 的 Controller Manager 負責事件監聽,並觸發相應的動作來滿足期望(Spec),這種方式也就是聲明式,即用戶只需要關心應用程序的最終狀態。當我們在使用中發現現有的這些資源不能滿足我們的需求的時候,Kubernetes 提供了自定義資源(Custom Resource)和 opertor 為應用程序提供基於 kuberntes 擴展。
CRD 則是對自定義資源的描述(Custom Resource Definition),也就是介紹這個資源有什麼屬性呀,這些屬性的類型是什麼,結構是怎樣的這類。
我們看一個postgres-operator
的CRD:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: postgresqls.acid.zalan.do
labels:
app.kubernetes.io/name: postgres-operator
annotations:
"helm.sh/hook": crd-install
spec:
group: acid.zalan.do
names:
kind: postgresql
listKind: postgresqlList
plural: postgresqls
singular: postgresql
shortNames:
- pg
additionalPrinterColumns:
- name: Team
type: string
description: Team responsible for Postgres CLuster
JSONPath: .spec.teamId
- name: Version
type: string
description: PostgreSQL version
JSONPath: .spec.postgresql.version
- name: Pods
type: integer
description: Number of Pods per Postgres cluster
JSONPath: .spec.numberOfInstances
- name: Volume
type: string
description: Size of the bound volume
JSONPath: .spec.volume.size
...
從上面的 CRD 文件可以看到 CRD 主要包括apiVersion
、kind
、metadata
和spec
四個部分。其中最關鍵的是apiVersion
和kind
,apiVersion
表示資源所屬組織和版本,apiVersion
一般由APIGourp
和Version
組成,這裡的 APIGourp 是apiextensions.k8s.io
,Version 是v1beta1
,相關信息可以通過kubectl api-resoures
查看。kind
表示資源類型,這裡是CustomResourceDefinition
,表示是一個自定義的資源描述。
operator是什麼
CoreOS 在 2016 年底提出了 Operator 的概念,當時的一段官方定義如下:
An Operator represents human operational knowledge in software, to reliably manage an application.
operator 是一種 kubernetes 的擴展形式,利用自定義資源對象(Custom Resource)來管理應用和組件,允許用戶以 Kubernetes 的聲明式 API 風格來管理應用及服務。operator 定義了一組在 Kubernetes 集群中打包和部署複雜業務應用的方法,operator主要是為解決特定應用或服務關於如何運行、部署及出現問題時如何處理提供的一種特定的自定義方式。比如:
- 按需部署應用服務(總不能用一大堆configmap來管理吧,也會很混亂~w(゚Д゚)w)
- 實現應用狀態的備份和還原,完成版本升級,比如
- 數據庫 schema 或額外的配置設置的改動
- 為分佈式應用進行master選舉,例如etcd,或者master-slave架構的mysql集群。
operator SDK
operator SDK —— operator framework,是 CoreOS 公司開發和維護的用於快速創建 operator 的工具,可以幫助我們快速構建 operator 應用,類似的工具還有:
- KUDO (Kubernetes 通用聲明式 Operator)
- kubebuilder,kubernetes SIG 在維護的一個項目
- Metacontroller,可與 Webhook 結合使用,以實現自己的功能。
如果希望查看 Operator 生態,可以上 operatorhub.io ,也可以將自己創建的應用發佈上去。
operator 安裝
安裝 operator sdk:
export RELEASE_VERSION=v0.13.0
curl -LO https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
chmod +x operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu && sudo mkdir -p /usr/local/bin/ && sudo cp operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu /usr/local/bin/operator-sdk && rm operator-sdk-${RELEASE_VERSION}-x86_64-linux-gnu
基於模板創建項目
用operator sdk 創建項目模板,這裡用官方提供的一個sample-controller的模板:
operator-sdk new <controller-name> --repo github.com/kubernetes/sample-controller
項目結構目錄創建完成,如下:
$ operator-sdk new test-controller --repo github.com/kubernetes/sample-controller
$ tree
.
├── build
│ ├── bin
│ │ ├── entrypoint
│ │ └── user_setup
│ └── Dockerfile
├── cmd
│ └── manager
│ └── main.go
├── deploy
│ ├── operator.yaml
│ ├── role_binding.yaml
│ ├── role.yaml
│ └── service_account.yaml
├── go.mod
├── go.sum
├── pkg
│ ├── apis
│ │ └── apis.go
│ └── controller
│ └── controller.go
├── tools.go
└── version
└── version.go
創建CRD
創建CRD:
operator-sdk add api --api-version=<api的版本> --kind=<類型名稱>
創建CRD後,多出來了文件夾:
$ operator-sdk add api --api-version=test.k8s.realibox.com/v1 --kind=Realibox
INFO[0000] Generating api version test.k8s.realibox.com/v1 for kind Realibox.
INFO[0000] Created pkg/apis/test/group.go
INFO[0002] Created pkg/apis/test/v1/realibox_types.go
INFO[0002] Created pkg/apis/addtoscheme_test_v1.go
INFO[0002] Created pkg/apis/test/v1/register.go
INFO[0002] Created pkg/apis/test/v1/doc.go
INFO[0002] Created deploy/crds/test.k8s.realibox.com_v1_realibox_cr.yaml
INFO[0004] Created deploy/crds/test.k8s.realibox.com_realiboxes_crd.yaml
INFO[0004] Running deepcopy code-generation for Custom Resource group versions: [test:[v1], ]
INFO[0014] Code-generation complete.
INFO[0014] Running CRD generation for Custom Resource group versions: [test:[v1], ]
INFO[0014] Created deploy/crds/test.k8s.realibox.com_realiboxes_crd.yaml
INFO[0014] CRD generation complete.
INFO[0014] API generation complete.
$ tree
...
├── pkg
│ ├── apis
│ │ ├── addtoscheme_test_v1.go
│ │ ├── apis.go
│ │ └── test
│ │ ├── group.go
│ │ └── v1
│ │ ├── doc.go
│ │ ├── realibox_types.go
│ │ ├── register.go
│ │ └── zz_generated.deepcopy.go
│ └── controller
│ └── controller.go
...
test 文件夾下面放得就是 CRD,我們通過pkg/apis/test/v1/*_types.go
文件定義我們的CRD結構,主要是Spec
和Status
:
vim pkg/apis/test/v1/realibox_types.go
...
// RealiboxSpec defines the desired state of Realibox
type RealiboxSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "operator-sdk generate k8s" to regenerate code after modifying this file
// Add custom validation using kubebuilder tags: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
}
type RealiboxStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "operator-sdk generate k8s" to regenerate code after modifying this file
// Add custom validation using kubebuilder tags: https://book-v1.book.kubebuilder.io/beyond_basics/generating_crd.html
}
...
這裡我們只改Spec
字段,將RealiboxSpec
結構體改為:
type RealiboxSpec struct {
Domain string `json:"domain,omitempty"`
OSS string `json:"oss,omitempty"`
Size string `json:"size,omitempty"`
}
更新CRD文件:
operator-sdk generate k8s
operator-sdk generate crds
CRD本質是一種k8s的資源,因此要使用crd,需要在K8s集群上創建CRD:
kubectl apply -f deploy/crds/test.k8s.realibox.com_realiboxes_crd.yaml
查看集群CRD:
$ kubectl get crd
NAME CREATED AT
clusterauthtokens.cluster.cattle.io 2020-08-29T06:41:42Z
clusteruserattributes.cluster.cattle.io 2020-08-29T06:41:42Z
realiboxes.test.k8s.realibox.com 2020-08-29T07:57:44Z
編寫controller
創建好 CRD 後,我們可以編寫 controller 了,先創建一個 controller 監聽和核對新創建的realibox資源類型:
命令行說明:
operator-sdk add controller --api-version=<api的版本> --kind=<類型名稱>
運行結果:
$ operator-sdk add controller --api-version=test.k8s.realibox.com/v1 --kind=Realibox
$ tree
...
├── pkg
│ ├── apis
│ │ ├── addtoscheme_test_v1.go
│ │ ├── apis.go
│ │ └── test
│ │ ├── group.go
│ │ └── v1
│ │ ├── doc.go
│ │ ├── realibox_types.go
│ │ ├── register.go
│ │ └── zz_generated.deepcopy.go
│ └── controller
│ ├── add_realibox.go
│ ├── controller.go
│ └── realibox
│ └── realibox_controller.go
...
在pkg/controller
目錄下生成了controller代碼,在pkg/controller/realibox/realibox_controller.go
編寫代碼邏輯即可,在這裡,我將CR信息在創建pod之前打印到日誌裡:
...
func (r *ReconcileRealibox) Reconcile(request reconcile.Request) (reconcile.Result, error) {
...
reqLogger.Info(fmt.Sprintf("Domain: %v created, oss info:%v, size: %v",instance.Spec.Domain,instance.Spec.OSS, instance.Spec.Size))
// Define a new Pod object
pod := newPodForCR(instance)
...
}
...
下面就可以運行 controller 了。
注:如果希望對集群進行更多地複雜操作,可以使用client-go
來操作 Kubernetes 的資源,client-go是一個對 Kubernetes API 進行封裝的庫,由 Kubernetes 官方提供,還是十分好用的。
運行 controller
運行controller有兩種方法,可以在本地直接運行controller,也可以打包到k8s運行。
本地運行controller
在本地運行controller直接go run
就可以了:
export WATCH_NAMESPACE=default
go run cmd/manager/main.go
注意:不管是在本地運行還是遠程運行都需要先在集群中創建CRD
運行好後我們可以編寫一個CR資源,提交到k8s集群中:
apiVersion: test.k8s.realibox.com/v1
kind: Realibox
metadata:
name: example-realibox
spec:
domain: "realibox.com"
oss: "aliyun.com"
size: "3Gb"
通過kubectl提交到集群,可以看到controller程序已經收到CR註冊內容打印出來了:
...
{"level":"info","ts":1598689291.273161,"logger":"controller_realibox","msg":"Domain: realibox.com created, oss info:aliyun.com, size: 3Gb","Request.Namespace":"default","Request.Name":"example-realibox"}
{"level":"info","ts":1598689291.2731829,"logger":"controller_realibox","msg":"Skip reconcile: Pod already exists","Request.Namespace":"default","Request.Name":"example-realibox","Pod.Namespace":"default","Pod.Name":"example-realibox-pod"}
...
打包提交到k8s運行
如果我們controller完成,我們可以將其打包放到k8s上運行:
打包鏡像
這裡用的阿里雲鏡像倉庫:
operator-sdk build registry.cn-shenzhen.aliyuncs.com/shikanon/realibox-operator-test:v0.1 --image-builder docker
將打包好鏡像上傳鏡像倉庫:
docker login [email protected] registry.cn-shenzhen.aliyuncs.com
docker push registry.cn-shenzhen.aliyuncs.com/shikanon/realibox-operator-test:v0.1
更改deploy/operator.yaml
裡面的鏡像名稱:
...
spec:
serviceAccountName: test-controller
containers:
- name: test-controller
# Replace this with the built image name
image: registry.cn-shenzhen.aliyuncs.com/shikanon/realibox-operator-test:v0.1
command:
- test-controller
imagePullPolicy: Always
...
將deploy下的YAML文件提交到集群中
kubectl apply -f deploy/service_account.yaml
kubectl apply -f deploy/role.yaml
kubectl apply -f deploy/role_binding.yaml
kubectl apply -f deploy/operator.yaml
查看狀態,確保全部成功:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/test-controller-75bf886d9c-whjdn 1/1 Running 0 23s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 123m
service/test-controller-metrics ClusterIP 10.43.100.251 <none> 8383/TCP,8686/TCP 12s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/test-controller 1/1 1 1 23s
NAME DESIRED CURRENT READY AGE
replicaset.apps/test-controller-75bf886d9c 1 1 1 23s
提交CR
這和單機運行是一樣的,編寫一個CR提交到集群,然後可以通過controller的pod日誌查看到變化:
$ kubectl logs -f test-controller-75bf886d9c-whjdn
...
{"level":"info","ts":1598690827.2699623,"logger":"controller_realibox","msg":"Reconciling Realibox","Request.Namespace":"default","Request.Name":"example-realibox"}
{"level":"info","ts":1598690827.270006,"logger":"controller_realibox","msg":"Domain: realibox.com created, oss info:aliyun.com, size: 3Gb","Request.Namespace":"default","Request.Name":"example-realibox"}
{"level":"info","ts":1598690827.2700245,"logger":"controller_realibox","msg":"Skip reconcile: Pod already exists","Request.Namespace":"default","Request.Name":"example-realibox","Pod.Namespace":"default","Pod.Name":"example-realibox-pod"}
...
參考文獻
- https://kubernetes.io/zh/docs/concepts/extend-kubernetes/operator/
- https://liqiang.io/post/kubernetes-all-about-crd-part06-kubebuilder-and-operator-sdk-d6e0858e
- https://coreos.com/blog/introducing-operators.html
- https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-building-kubernetes-operators-and-stateful-apps
- https://mp.weixin.qq.com/s/aXdbBJlUAQp6YPvZU3isSQ