【Kubernetes】使用 Kuboard-Spray 安装 Kubernetes 集群

# 虚拟机启动，安装 Ubuntu 20.04 系统

# 安装 docker
采用的是内网镜像源的安装方式：
```bash
curl -fsSL http://192.168.ip.ip/raw/general/docker/install-docker.sh | DOWNLOAD_URL=http://192.168.ip.ip/docker bash -s
```
安装后，
## 启动 docker 服务
```bash
sudo systemctl daemon-reload
sudo systemctl restart docker
```

# 安装 Kuboard-Spray：图形化界面的离线安装工具
Kuboard-Spray 是一款可以在图形界面引导下完成 Kubernetes 高可用集群离线安装的工具，安装后的集群版本为：
- Kubernetes v1.23.1

执行以下命令：
```bash
docker run -d \ # 在后台运行容器
	--privileged \ # 给予容器特权模式运行
	--restart=unless-stopped \ # 容器退出时自动重启，除非手动停止
	--name=kuboard-spray \ # 设置容器名称为 kuboard-spray
	-p 80:80/tcp \ # 将容器的 80 端口映射到主机的 80 端口
	-v /var/run/docker.sock:/var/run/docker.sock \ # 挂载 Docker socket，允许容器内操作主机 Docker
	-v ~/kuboard-spray-data:/data \ # 将主机的 ~/kuboard-spray-data 目录挂载到容器的 /data
	eipwork/kuboard-spray:latest-amd64 # 使用的镜像名称和标签

```

- KuboardSpray 的信息保存在容器的 `/data` 路径，上面的命令是将其映射到了 `~/kuboard-spray-data` 路径
- 只要此路径的内容不受损坏，重启、升级、重新安装 Kuboard-Spray，或将数据及 Kuboard-Spray 迁移到另外一台机器上，都可以找回原来的信息

## 安装过程中的问题记录
### 目录问题
在执行安装 kuboardspray 命令前，需要创建目录 `~/kuboard-spray-data` 目录
```bash
mkdir ~/kuboard-spray-data
```
### docker 镜像源问题
需要创建 docker 配置文件。不然会出现镜像拉取失败的问题。

```bash
sudo tee /etc/docker/daemon.json <<-'E0F'
{
"insecure-registries": [
"http://192.168.ip.ip:5000",
"http://192.168.ip.ip"
],
"registry-mirrors": [
"http://192.168.ip.ip:5000"
]
}
E0F
```
如果过程中有任何的编辑错误，或者出现各种问题，可以先删除 配置文件
同时命令需要 `sudo` 进行
如果有问题，可以直接删除配置文件
```bash
sudo rm /etc/docker/daemon.json # 或其他文件名称 带路径的
```

# 打开 Kuboard-Spray 界面
在浏览器打开地址 `http://刚刚部署机的 IP`，用户名： `admin` ，默认密码 `Kuboard123` 即可登录。
![Pasted image 20250213142811](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/qv6qhl.webp)

# 加载离线资源包
![Pasted image 20250213152753](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/qviy9i.webp)

选择最新的 spray-v2.21.0c_k8s-v1.26.4_v4.4
## 资源包内容
### kubespray_version
v2.21.0c

### kubernetes
#### 操作系统
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/quxmq2.webp)

#### image_arch
amd64
#### gcr_image_repo
gcr.io
#### kube_image_repo
k8s.gcr.io
#### kube_version
v1.26.4
#### container_manager
container_1.6.19

### etcd
#### etcd_version 
v3.5.6
#### ETCD 部署方式
二进制部署

### 网络插件
#### calico
calico_version v3.24.5
#### flannel
flannel_version v0.20.2
flannel_cni_version v1.2.0

### 依赖组件
#### crun_version
1.4.5
#### krew_version
v0.4.3
#### runc_version
v1.1.4
#### cni_version
v1.2.0
#### crictl_version
v1.26.0
#### nerdctl_version
1.0.0
#### nginx_image_tag
1.23.2
#### coredns_version
v1.9.3
#### dnsautoscaler_version
1.8.5
#### pod_infra_version
3.8

### 可选组件
#### kuboard 默认安装
kuboard_version v3.5.2.4
#### nodelocaldns 默认安装
nodelocaldns_version 1.22.18
#### netchecker 默认安装
netcheck_version v1.2.2
#### metrics_server 默认安装
metrics_server_version v0.6.2

# 离线导入资源包
## 选择镜像
官方给出的有三种镜像可选，因为上文在安装 kuboard-spray 中 有过选择，所以选择 `eipwork/kuboard-spray-resource`
## 下载镜像
因为一直是将 192.168.25.64 当做部署机，所以在该机器上继续操作
```bash
# 1. 执行镜像拉取命令
sudo docker pull eipwork/kuboard-spray-resource:spray-v2.21.0c_k8s_v1.26.4_v4.4-amd64
sudo docker save eipwork/kuboard-spray-resource:spray-v2.21.0c_k8s_v1.26.4_v4.4.-amd64 > kuboard-spray-resource.tar

# 2. 将 kuboard-spray-resource.tar 复制到 kuboard-spray 所在的服务器
sudo scp ./kuboard-spray-resource.tar username@192.168.25.64:~/kuboard-spray-resource.tar

# 3. 在 kuboard-spray 所在的服务器上执行
sudo docker load < ~/kuboard-spray-resource.tar
```

因为此时我使用的是普通账号登录的服务器，所以第二步的命令做了调整：
- 将 原本的 `root@` 更改为登录的用户名 `username@`
- `/root/` 更改为 `~/` （代表的用户主目录
	- `~/` 是 home 目录的简写，等同于 `/home/username/`
- 如果后续步骤需要在 root 目录下操作这个文件，可能需要使用 sudo 命令将文件移动到 root 目录：
```bash
sudo mv ~/kuboard-spray-resource.tar /root/
```

上述执行完后，有如下的输出：
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/relxyg.webp)
- `Loaded image` ：表示镜像加载成功

## 导入到 Kuboard-Spray
### 复制下面的 YAML 内容：
```yaml
downloadFrom: eipwork/kuboard-spray-resource
metadata:
  version: spray-v2.21.0c_k8s-v1.26.4_v4.4-amd64
  type: kubernetes-offline-resource
  kuboard_spray_version:
    min: v1.2.4
  available_at:
    - registry.cn-shanghai.aliyuncs.com/kuboard-spray/kuboard-spray-resource
    - swr.cn-east-2.myhuaweicloud.com/kuboard/kuboard-spray-resource
    - eipwork/kuboard-spray-resource
  issue_date: 2023-4-29
  owner: shaohq@foxmail.com
  can_upgrade_from:
    include:
      - 'spray-v2.21.0[a-c]_k8s-v1.26.[0-3]*_v4.[1-3]-amd64'
      - 'spray-v2.20.0[a-b]_k8s-v1.25.[0-9]*_v3.[0-9]*-amd64'
    exclude: null
  can_replace_to: null
  supported_os:
    - distribution: Ubuntu
      versions:
        - '20.04'
        - '22.04'
    - distribution: Anolis
      versions:
        - '8.4'
        - '8.5'
        - '8.6'
    - distribution: CentOS
      versions:
        - '7.6'
        - '7.8'
        - '7.9'
        - '8'
    - distribution: RedHat
      versions:
        - '7.9'
        - '8.5'
    - distribution: OracleLinux
      versions:
        - '8.5'
        - '8.7'
        - '9.1'
    - distribution: Rocky
      versions:
        - '8.5'
        - '8.7'
        - '9.1'
    - distribution: openEuler
      versions:
        - '20.03'
        - '22.03'
    - distribution: Kylin Linux Advanced Server
      versions:
        - V10
    - distribution: openSUSE Leap
      versions:
        - '15.3'
    - distribution: UnionTech OS Server 20
      versions:
        - '20'
    - distribution: AlmaLinux
      versions:
        - '8.7'
        - '9.1'
  supported_feature:
    eviction_hard: true
data:
  kubespray_version: v2.21.0c
  supported_playbooks:
    install_cluster: pb_cluster.yaml
    remove_node: pb_remove_node.yaml
    add_node: pb_scale.yaml
    sync_nginx_config: pb_sync_nginx_config.yaml
    sync_etcd_address: pb_sync_etcd_address.yaml
    install_addon: pb_install_addon.yaml
    remove_addon: pb_remove_addon.yaml
    cluster_version_containerd: pb_cluster_version_containerd.yaml
    cluster_version_docker: pb_cluster_version_docker.yaml
    upgrade_cluster: pb_upgrade_cluster.yaml
    drain_node: pb_drain_node.yaml
    uncordon_node: pb_uncordon_node.yaml
    cis_scan: true
    renew_cert: pb_renew_cert.yaml
    sync_container_engine_params: pb_sync_container_engine_params.yaml
    backup_etcd: pb_backup_etcd.yaml
    restore_etcd: pb_restore_etcd.yaml
  kubernetes:
    kube_version: v1.26.4
    image_arch: amd64
    gcr_image_repo: gcr.io
    kube_image_repo: k8s.gcr.io
    candidate_admission_plugins: >-
      AlwaysAdmit,AlwaysDeny,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,DefaultIngressClass,DefaultStorageClass,DefaultTolerationSeconds,DenyServiceExternalIPs,EventRateLimit,ExtendedResourceToleration,ImagePolicyWebhook,LimitPodHardAntiAffinityTopology,LimitRanger,MutatingAdmissionWebhook,NamespaceAutoProvision,NamespaceExists,NamespaceLifecycle,NodeRestriction,OwnerReferencesPermissionEnforcement,PersistentVolumeClaimResize,PersistentVolumeLabel,PodNodeSelector,PodSecurity,PodTolerationRestriction,Priority,ResourceQuota,RuntimeClass,SecurityContextDeny,ServiceAccount,StorageObjectInUseProtection,TaintNodesByCondition,ValidatingAdmissionWebhook
    default_enabled_admission_plugins: >-
      CertificateApproval,CertificateSigning,CertificateSubjectRestriction,DefaultIngressClass,DefaultStorageClass,DefaultTolerationSeconds,LimitRanger,MutatingAdmissionWebhook,NamespaceLifecycle,PersistentVolumeClaimResize,Priority,ResourceQuota,RuntimeClass,ServiceAccount,StorageObjectInUseProtection,TaintNodesByCondition,ValidatingAdmissionWebhook
  container_engine:
    - container_manager: containerd
      params:
        containerd_version: 1.6.19
  vars:
    target:
      containerd_version: 1.6.19
      etcd_version: v3.5.6
      calico_version: v3.24.5
      flannel_cni_version: v1.2.0
      kubelet_checksums:
        arm64:
          v1.26.4: a925a5d20d29c362f0c4d60cb005f21d44576837510e0bc65c817961969b4e7e
        amd64:
          v1.26.4: 1e29fe7a097066cfbc1c1d2ab37f8b883c8f3fec414bafe8f2c7b960b0fb60fe
      kubectl_checksums:
        arm64:
          v1.26.4: eea4054825a4c20cc09bc15abcb1354725ad886338e6892141a071caab91d4b6
        amd64:
          v1.26.4: 636ac0eaa467dbceda4b2c4e33662adc9709f5ce40341c9fc1a687fc276ac02d
      kubeadm_checksums:
        arm64:
          v1.26.4: a97052d393e60027c354e97c88493aa14a76c8cfb7418bbdf8425b3711d86e3a
        amd64:
          v1.26.4: aa1a137aa2c3427f199ff652c96b11d6b124358296996eb7b8cbde220607b2fe
      crun_checksums:
        arm64:
          1.4.5: 64a01114060ec12e66b1520c6ee6967410022d1ec73cdc7d14f952343c0769f2
        amd64:
          1.4.5: 84cf20a6060cd53ac21a0590367d1ab65f74baae005c42f2d5bc1af918470455
      runc_checksums:
        arm64:
          v1.1.4: dbb71e737eaef454a406ce21fd021bd8f1b35afb7635016745992bbd7c17a223
        amd64:
          v1.1.4: db772be63147a4e747b4fe286c7c16a2edc4a8458bd3092ea46aaee77750e8ce
      containerd_archive_checksums:
        arm64:
          1.6.19: 25a0dd6cce4e1058824d6dc277fc01dc45da92539ccb39bb6c8a481c24d2476e
        amd64:
          1.6.19: 3262454d9b3581f4d4da0948f77dde1be51cfc42347a1548bc9ab6870b055815..
      nerdctl_archive_checksums:
        arm64:
          1.0.0: 27622c9d95efe6d807d5f3770d24ddd71719c6ae18f76b5fc89663a51bcd6208
        amd64:
          1.0.0: 3e993d714e6b88d1803a58d9ff5a00d121f0544c35efed3a3789e19d6ab36964
      etcd_binary_checksums:
        arm64:
          v3.5.6: 888e25c9c94702ac1254c7655709b44bb3711ebaabd3cb05439f3dd1f2b51a87
        amd64:
          v3.5.6: 4db32e3bc06dd0999e2171f76a87c1cffed8369475ec7aa7abee9023635670fb
      cni_binary_checksums:
        arm64:
          v1.2.0: 525e2b62ba92a1b6f3dc9612449a84aa61652e680f7ebf4eff579795fe464b57
        amd64:
          v1.2.0: f3a841324845ca6bf0d4091b4fc7f97e18a623172158b72fc3fdcdb9d42d2d37
      flannel_cni_binary_checksums:
        arm64:
          v1.2.0: f813ae49b7b84eb95db73f7a3c34d2ee101f8cfc27e3a8054297a36d53308543
        amd64:
          v1.2.0: 63906a5b7dc78fbf1fbd484adbf4931aea5b15546ece3c7202c779ab9ea994a2
      flannel_image_repo: '{{ docker_image_repo }}/flannelcni/flannel'
      flannel_image_tag: '{{ flannel_version }}-{{ image_arch }}'
      flannel_init_image_repo: '{{ docker_image_repo }}/flannelcni/flannel-cni-plugin'
      flannel_init_image_tag: '{{ flannel_cni_version }}-{{ image_arch }}'
      calicoctl_download_url: >-
        https://github.com/projectcalico/calico/releases/download/{{
        calico_ctl_version }}/calicoctl-linux-{{ image_arch }}
      calicoctl_binary_checksums:
        amd64:
          v3.24.5: 01e6c8a2371050f9edd0ade9dcde89da054e84d8e96bd4ba8cf82806c8d3e8e7
        arm64:
          v3.24.5: 2d56b768ed346129b0249261db27d97458cfb35f98bd028a0c817a23180ab2d2
      calico_crds_archive_checksums:
        v3.24.5: 10320b45ebcf4335703d692adacc96cdd3a27de62b4599238604bd7b0bedccc3
      krew_archive_checksums:
        linux:
          arm64:
            v0.4.3: 0994923848882ad0d4825d5af1dc227687a10a02688f785709b03549dd34d71d
          amd64:
            v0.4.3: 5df32eaa0e888a2566439c4ccb2ef3a3e6e89522f2f2126030171e2585585e4f
      crictl_checksums:
        arm64:
          v1.26.0: b632ca705a98edc8ad7806f4279feaff956ac83aa109bba8a85ed81e6b900599
        amd64:
          v1.26.0: cda5e2143bf19f6b548110ffba0fe3565e03e8743fadd625fee3d62fc4134eed
      snapshot_controller_image_tag: v4.2.1
      dns_min_replicas: '{{ [ 2, groups[''kube_control_plane''] | length ] | min }}'
      kuboardspray_extra_downloads:
        kuboard:
          container: true
          file: false
          enabled: '{{ kuboard_enabled | default(false) }}'
          version: '{{ kuboard_version | default(''v3.5.2.4'') }}'
          repo: eipwork/kuboard
          tag: '{{ kuboard_version }}'
          sha256: ''
          groups:
            - kube_control_plane
        netcheck_etcd:
          container: true
          file: false
          enabled: '{{ deploy_netchecker }}'
          version: '{{ netcheck_etcd_image_tag }}'
          dest: >-
            {{ local_release_dir }}/etcd-{{ netcheck_etcd_image_tag }}-linux-{{
            image_arch }}.tar.gz
          repo: '{{ etcd_image_repo }}'
          tag: '{{ netcheck_etcd_image_tag }}'
          sha256: '{{ etcd_digest_checksum|d(None) }}'
          unarchive: false
          owner: root
          mode: '0755'
          groups:
            - k8s_cluster
        coredns:
          enabled: '{{ dns_mode in [''coredns'', ''coredns_dual''] }}'
          container: true
          repo: '{{ coredns_image_repo }}'
          tag: '{{ coredns_image_tag }}'
          sha256: '{{ coredns_digest_checksum|default(None) }}'
          groups:
            - k8s_cluster
  etcd:
    etcd_version: v3.5.6
    etcd_params: null
    etcd_deployment_type:
      - host
  dependency:
    - name: crun
      version: 1.4.5
      target: crun_version
    - name: krew
      version: v0.4.3
      target: krew_version
    - name: runc
      version: v1.1.4
      target: runc_version
    - name: cni-plugins
      version: v1.2.0
      target: cni_version
    - name: crictl
      version: v1.26.0
      target: crictl_version
    - name: nerdctl
      version: 1.0.0
      target: nerdctl_version
    - name: nginx_image
      version: 1.23.2
      target: nginx_image_tag
    - name: coredns
      target: coredns_version
      version: v1.9.3
    - name: cluster-proportional-autoscaler
      target: dnsautoscaler_version
      version: 1.8.5
    - name: pause
      target: pod_infra_version
      version: '3.8'
  network_plugin:
    - name: calico
      params:
        calico_version: v3.24.5
    - name: flannel
      params:
        flannel_version: v0.20.2
        flannel_cni_version: v1.2.0
  addon:
    - name: kuboard
      target: kuboard_enabled
      lifecycle:
        install_by_default: true
        check:
          shell: kubectl get pods -n kuboard -l k8s.kuboard.cn/name=kuboard-v3
          keyword: kuboard-v3
        install_addon_tags:
          - download
          - upgrade
          - kuboard
        remove_addon_tags:
          - upgrade
          - kuboard
        downloads:
          - kuboard
      params_default:
        kuboard_version: v3.5.2.4
        kuboard_port: 80
        kuboard_cluster_name: default
        kuboard_data_dir: /root/kuboard-data
      params: null
    - name: nodelocaldns
      target: enable_nodelocaldns
      lifecycle:
        install_by_default: true
        check:
          shell: kubectl get daemonset -n kube-system nodelocaldns -o json
          keyword: '"k8s-app": "kube-dns"'
        install_addon_tags:
          - download
          - upgrade
          - coredns
          - nodelocaldns
        downloads:
          - nodelocaldns
          - coredns
      params:
        nodelocaldns_version: 1.22.18
        enable_nodelocaldns_secondary: false
    - name: netchecker
      target: deploy_netchecker
      lifecycle:
        install_by_default: true
        check:
          shell: >-
            kubectl get deployment -n {{ netcheck_namespace | default('default')
            }} netchecker-server -o json
          keyword: k8s-netchecker-server
        install_addon_tags:
          - download
          - upgrade
          - netchecker
        remove_addon_tags:
          - upgrade
          - netchecker
        downloads:
          - netcheck_server
          - netcheck_agent
          - netcheck_etcd
      params:
        netcheck_version: v1.2.2
        netcheck_agent_image_repo: '{{ docker_image_repo }}/mirantis/k8s-netchecker-agent'
        netcheck_agent_image_tag: '{{ netcheck_version }}'
        netcheck_server_image_repo: '{{ docker_image_repo }}/mirantis/k8s-netchecker-server'
        netcheck_server_image_tag: '{{ netcheck_version }}'
        netcheck_etcd_image_tag: v3.5.6
    - name: metrics_server
      target: metrics_server_enabled
      lifecycle:
        install_by_default: true
        check:
          shell: kubectl get deployments -n kube-system metrics-server -o json
          keyword: k8s.gcr.io/metrics-server/metrics-server
        install_addon_tags:
          - download
          - upgrade
          - metrics_server
        remove_addon_tags:
          - upgrade
          - metrics_server
        downloads:
          - metrics_server
      params:
        metrics_server_version: v0.6.2

```

### 页面操作
在 Kuboard-Spray 界面中导航到 系统设置 --> 资源包管理，单击 【离线加载资源包】按钮，按照页面提示操作，即可完成资源包的离线导入。
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/sdpg3t.webp)

执行结果：
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/sdzr1y.webp)

列表状态：
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/selgiq.webp)

# 规划并安装集群（集群管理页面）
单击【添加集群安装计划】，定义集群名称，选择刚刚导入的离线资源包。
集群名称不可修改
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/sfn1bp.webp)
单击【确定】，将进入集群规划页面。
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/13/sj5qdl.webp)
在该页面中可以添加每个集群节点的链接参数，并设置节点的角色
需要注意的是，kuboard-spray 所在机器不能当做 K8s 集群的一个节点，因为安装过程中会重启集群节点的容器引擎，这会导致 kuboard-spray 被重启掉。

## 集群规划原则及注意事项
- 最少的节点数量为：1
- ETCD 节点、控制节点的总数量，必须为 `奇数`
- 在 `全局设置` 标签页，可以设置节点的通用链接参数，例如所有的节点都是用相同的 ssh 端口、用户名、密码，则共同的参数只在此处设置即可
- 在节点标签页，如果该节点的角色包含 `etcd` 则必须填写 `ETCD 成员名称` 这个字段
- 如果在 KuboardSpray 所在节点不能直接访问到 Kubernetes 集群的节点，可以设置跳板机参数，使 KuboardSpray 可以通过 ssh 访问集群节点
- 集群安装过程中，除了已经导入的资源以外，还需要使用 yum 或 apt 指令安装一些系统软件，例如 curl， rsync，ipvadm，ethtool 等，此时需要用到操作系统的 apt 软件源 或者 yum 软件源。在 `全局设置` 标签页中，可以引导完成 apt / yum 软件源的设置，可以：
	- 使用节点操作系统已经事先配置的 apt / yum 源
	- 在安装过程中自动配置节点的操作系统使用制定的软件源
- 如果使用 docker 作为集群的容器引擎，还需要在 `全局设置` 标签页制定安装 docker 用的 apt / yum 源，使用 containerd 作为容器引擎的话不需要配置 docker 的apt / yum 源，已包含在 KuboardSpray 离线资源包中。

# 查看所有容器
完成 Kubernetes 集群安装之后，执行以下命令查看所有容器情况
```bash
sudo kubectl get pods -A
```
![image](https://cdn-imagehost.xyzttt.xyz/image/2025/02/24/j3u93d.webp)
会发现已经有一个容器在运行 kuboard-v3，即多集群管理工具（K8s 集群管理界面）

# 打开 Kuboard
IP 是控制节点的 IP
默认用户名：admin
默认密码：Kuboard123

# References
- 
[Kuboard 教程 - 安装高可用的 Kubernetes 集群](https://kuboard.cn/install/install-k8s.html)