背景
我們知道，如果在Kubernetes中支持GPU設備調度，需要做如下的工作：
-
節點上安裝nvidia驅動
-
節點上安裝nvidia-docker
-
集群部署gpu device plugin，用於為調度到該節點的pod分配GPU設備。
除此之外，如果你需要監控集群GPU資源使用情況，你可能還需要安裝DCCM exporter結合Prometheus輸出GPU資源監控信息。
要安裝和管理這麼多的組件，對於運維人員來說壓力不小。基於此，NVIDIA開源了一款叫NVIDIA GPU Operator的工具，該工具基於Operator Framework實現，用於自動化管理上面我們提到的這些組件。
NVIDIA GPU Operator有以下的組件構成：
-
安裝nvidia driver的組件
-
安裝nvidia container toolkit的組件
-
安裝nvidia devcie plugin的組件
-
安裝nvidia dcgm exporter組件
-
安裝gpu feature discovery組件
本系列文章不打算一上來就開始講NVIDIA GPU Operator，而是先把各個組件的安裝詳細的分析一下，然後手動安裝這些組件，最後再來分析NVIDIA GPU Operator就比較簡單了。
在本篇文章中，我們將介紹NVIDIA GPU Operator安裝GPU Feature Discovery組件。
GPU Feature Discovery介紹
GPU Feature Discovery這個組件主要的作用是給GPU節點打上一些與GPU設備屬性相關的標籤，比如：該節點的GPU驅動是哪個版本，GPU顯存是多大等，其效果是節點擁有很多以"nvidia.com"開頭的標籤。
$ kubectl get nodes cn-beijing.192.168.8.44 -o yaml | grep nvidia.com | grep -v "feature-labels"
nvidia.com/cuda.driver.major: "450"
nvidia.com/cuda.driver.minor: "102"
nvidia.com/cuda.driver.rev: "04"
nvidia.com/cuda.runtime.major: "11"
nvidia.com/cuda.runtime.minor: "0"
nvidia.com/gfd.timestamp: "1616729805"
nvidia.com/gpu.compute.major: "7"
nvidia.com/gpu.compute.minor: "0"
nvidia.com/gpu.count: "1"
nvidia.com/gpu.family: volta
nvidia.com/gpu.machine: Alibaba-Cloud-ECS
nvidia.com/gpu.memory: "16160"
nvidia.com/gpu.present: "true"
nvidia.com/gpu.product: Tesla-V100-SXM2-16GB
nvidia.com/mig.strategy: single
nvidia.com/gpu: "1"
nvidia.com/gpu: "1"打上這些標籤的主要作用是在之後的任務調度中，可以根據這些標籤將任務調度到指定節點上。
集群部署GPU Feature Discovery組件
接下來演示一下如何在集群中部署GPU Feature Discovery組件。
前提條件
-
k8s集群的版本 > 1.8。
-
集群中的GPU節點已經安裝了GPU驅動，如果沒有安裝驅動，請參考本系列文章《NVIDIA GPU Operator分析一：NVIDIA驅動安裝》。
-
集群中的GPU節點已經安裝NVIDIA Container Toolkit，如果沒有安裝，請參考本系列文章《NVIDIA GPU Operator分析二：NVIDIA Container Toolkit安裝》。
-
集群中的GPU節點已經安裝NVIDIA Device Plugin，如果沒有安裝，請參考本系列文章《NVIDIA GPU Operator分析三：NVIDIA Device Plugin安裝》。
安裝步驟
1.下載gpu-operator源碼。
$ git clone -b 1.6.2 https://github.com/NVIDIA/gpu-operator.git
$ cd gpu-operator
$ export GPU_OPERATOR=$(pwd) 2.確認節點已經打了標籤nvidia.com/gpu.present=true。
$ kubectl get nodes -L nvidia.com/gpu.present
NAME STATUS ROLES AGE VERSION GPU.PRESENT
cn-beijing.192.168.8.44 Ready <none> 13d v1.16.9-aliyun.1 true
cn-beijing.192.168.8.45 Ready <none> 13d v1.16.9-aliyun.1 true
cn-beijing.192.168.8.46 Ready <none> 13d v1.16.9-aliyun.1 true
cn-beijing.192.168.9.159 Ready master 13d v1.16.9-aliyun.1
cn-beijing.192.168.9.160 Ready master 13d v1.16.9-aliyun.1
cn-beijing.192.168.9.161 Ready master 13d v1.16.9-aliyun.13.刪除不用的yaml。
$ rm -rf assets/gpu-feature-discovery/*openshift*4.修改assets/gpu-feature-discovery/0500_daemonset.yaml中的鏡像。
-
將container gpu-feature-discovery的鏡像修改為nvcr.io/nvidia/gpu-feature-discovery:v0.4.1。
spec:
serviceAccount: nvidia-gpu-feature-discovery
containers:
- image: "nvcr.io/nvidia/gpu-feature-discovery:v0.4.1"
name: gpu-feature-discovery5.部署gpu-feature-discovery組件。
$ kubectl apply -f assets/gpu-feature-discovery6.查看組件的pod是否處於Running狀態。
$ kubectl get po -n gpu-operator-resources -l app=gpu-feature-discovery
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-49css 1/1 Running 0 96s
gpu-feature-discovery-k4nlp 1/1 Running 0 96s
gpu-feature-discovery-lx46x 1/1 Running 0 96s7.查看其中一個pod的日誌。
$ kubectl logs gpu-feature-discovery-49css -n gpu-operator-resources
gpu-feature-discovery: 2021/03/31 12:58:46 Running gpu-feature-discovery in version v0.4.1
gpu-feature-discovery: 2021/03/31 12:58:46 Loaded configuration:
gpu-feature-discovery: 2021/03/31 12:58:46 Oneshot: false
gpu-feature-discovery: 2021/03/31 12:58:46 FailOnInitError: true
gpu-feature-discovery: 2021/03/31 12:58:46 SleepInterval: 1m0s
gpu-feature-discovery: 2021/03/31 12:58:46 MigStrategy: single
gpu-feature-discovery: 2021/03/31 12:58:46 NoTimestamp: false
gpu-feature-discovery: 2021/03/31 12:58:46 OutputFilePath: /etc/kubernetes/node-feature-discovery/features.d/gfd
gpu-feature-discovery: 2021/03/31 12:58:46 Start running
gpu-feature-discovery: 2021/03/31 12:58:46 Writing labels to output file
gpu-feature-discovery: 2021/03/31 12:58:46 Sleeping for 1m0s
gpu-feature-discovery: 2021/03/31 12:59:46 Writing labels to output file
gpu-feature-discovery: 2021/03/31 12:59:46 Sleeping for 1m0s
gpu-feature-discovery: 2021/03/31 13:00:46 Writing labels to output file
gpu-feature-discovery: 2021/03/31 13:00:46 Sleeping for 1m0s8.查看GPU節點是否被打上以nvidia.com開頭的標籤。
(1)查看集群節點：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
cn-beijing.192.168.8.44 Ready <none> 18d v1.16.9-aliyun.1
cn-beijing.192.168.8.45 Ready <none> 18d v1.16.9-aliyun.1
cn-beijing.192.168.8.46 Ready <none> 18d v1.16.9-aliyun.1
cn-beijing.192.168.9.159 Ready master 19d v1.16.9-aliyun.1
cn-beijing.192.168.9.160 Ready master 19d v1.16.9-aliyun.1
cn-beijing.192.168.9.161 Ready master 19d v1.16.9-aliyun.1(2)以cn-beijing.192.168.8.45為例，查看其標籤：
$ kubectl get nodes cn-beijing.192.168.8.44 -o yaml | grep -v feature-labels | grep nvidia.com
nvidia.com/cuda.driver.major: "450"
nvidia.com/cuda.driver.minor: "102"
nvidia.com/cuda.driver.rev: "04"
nvidia.com/cuda.runtime.major: "11"
nvidia.com/cuda.runtime.minor: "0"
nvidia.com/gfd.timestamp: "1616729805"
nvidia.com/gpu.compute.major: "7"
nvidia.com/gpu.compute.minor: "0"
nvidia.com/gpu.count: "1"
nvidia.com/gpu.family: volta
nvidia.com/gpu.machine: Alibaba-Cloud-ECS
nvidia.com/gpu.memory: "16160"
nvidia.com/gpu.present: "true"
nvidia.com/gpu.product: Tesla-V100-SXM2-16GB
nvidia.com/mig.strategy: single
nvidia.com/gpu: "1"
nvidia.com/gpu: "1"可以看到，節點已經打上了與GPU設備屬性相關的標籤。
總結
本篇文章主要演示瞭如何在集群中手動安裝GPU Feature Discovery組件，至此NVIDIA GPU Operator所涉及的所有組件我們都手動安裝了一遍。在本系列文章的下一篇文章中，我們將從源碼的角度分析NVIDIA GPU Operator是怎麼實現的。