sean的农场
large picture of the cover

kylin11安装k8s 1.34 单mater集群（kubeadm联网环境）

1、环境准备

主机名

系统

ip

配置

内核版本

软件版本

master

kylin11

192.168.8.70

4C/4G/50G

6.6

k8s-1.34 containerd-2.1.4 runc-1.3.1 crictl-1.34.0

node01

kylin11

192.168.8.71

4C/4G/50G

6.6

k8s-1.34 containerd-2.1.4 runc-1.3.1 crictl-1.34.0

node02

kylin11

192.168.8.72

4C/4G/50G

6.6

k8s-1.34 containerd-2.1.4 runc-1.3.1 crictl-1.34.0

1.1、配置主机名，ip，host，免密登录

# 设置主机名
hostnamectl set-hostname master
hostnamectl set-hostname node01
hostnamectl set-hostname node02

#关闭DHCP，设置静态ip
vim /etc/sysconfig/network-scripts/ifcfg-ens18
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=eui64
NAME=ens18
UUID=9e635292-87e0-4a97-b5f7-ce2d5151fc97
DEVICE=ens18
ONBOOT=yes
IPADDR=192.168.8.70
PREFIX=24
GATEWAY=192.168.8.1
DNS1=114.114.114.114

# 重启网卡
systemctl restart NetworkManager
nmcli c reload
nmcli c up ens18

# 设置hosts
192.168.8.70 master
192.168.8.71 node01
192.168.8.72 node02

# 在master设置ssh免密登录
 ssh-keygen -t rsa # 三次回车
for i in master node01 node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

1.2、关闭防火墙，swap分区和selinux

# 关闭firewalld
systemctl disable --now firewalld

firewall-cmd --state
not running

# selinux默认已关闭
sestatus
SELinux status:                 disabled

# 如果没有关闭，则修改配置文件
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

#关闭swap分区
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

1.3、配置时间同步

# kylin默认已经开启时间同步
timedatectl 
               Local time: 一 2025-09-29 15:55:53 CST
           Universal time: 一 2025-09-29 07:55:53 UTC
                 RTC time: 一 2025-09-29 07:55:53
                Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no


systemctl status chronyd.service 
Warning: The unit file, source configuration file or drop-ins of chronyd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-09-29 12:02:24 CST; 3h 53min ago
       Docs: man:chronyd(8)
             man:chrony.conf(5)
   Main PID: 923 (chronyd)
      Tasks: 1 (limit: 21400)
     Memory: 4.4M ()
     CGroup: /system.slice/chronyd.service
             └─923 /usr/sbin/chronyd

9月 29 12:02:22 master systemd[1]: Starting NTP client/server...
9月 29 12:02:24 master chronyd[923]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG)
9月 29 12:02:24 master chronyd[923]: Frequency -25.286 +/- 3.260 ppm read from /var/lib/chrony/drift
9月 29 12:02:24 master systemd[1]: Started NTP client/server.
9月 29 12:02:47 master chronyd[923]: Selected source 113.141.164.39 (ntp.ntsc.ac.cn)
9月 29 12:03:52 master chronyd[923]: Selected source 8.149.241.96 (ntp1.aliyun.com)

1.4、配置内核路由转发及网桥过滤

# 修改/etc/sysctl.conf 
net.ipv4.ip_forward = 1
# 生效
sysctl -p

# 添加网桥过滤文件
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# 加载br_netfilter模块,如果不执行此命令不会生效
modprobe br_netfilter
# 查看是否加载
lsmod |grep br_netfilter
br_netfilter           36864  0
bridge                425984  1 br_netfilter

# 重新加载生效
sysctl --system

1.5、安装ipset和ipvsadm

# 安装ipset和ipvsadm
dnf install -y ipset ipvsadm

#配置ipvs需要加载的模块
cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

# 加载生效
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod |grep -e ip_vs -e nf_conntrack

配置完参数后reboot重启服务器

2、组件准备

kylin11默认安装了docker-cli,moby,containerd,我们需要卸载这些组件，并重新安装containerd

dnf list |grep -E 'docker|moby|containerd|runc'

containerd.x86_64                                       1.7.23-1.p02.ky11                               @anaconda      
docker-cli.x86_64                                       1:24.0.9-1.ky11                                 @anaconda      
moby.x86_64                                             24.0.9-1.p01.ky11                               @anaconda      
runc.x86_64                                             1.1.12-1.p03.ky11                               @anaconda      
# 移除组件
dnf remove -y docker* moby* containerd* runc*
Removed:
  containerd-1.7.23-1.p02.ky11.x86_64              docker-cli-1:24.0.9-1.ky11.x86_64              libcgroup-3.1.0-6.ky11.x86_64              moby-24.0.9-1.p01.ky11.x86_64              runc-1.1.12-1.p03.ky11.x86_64              tini-static-0.19.0-1.p01.ky11.x86_64             

# 安装需要用到的软件包
dnf install -y wget

2.1、安装containerd

GitHub仓库地址 https://github.com/containerd/containerd/blob/main/docs/getting-started.md

# 获取压缩包文件
wget https://github.com/containerd/containerd/releases/download/v2.1.4/containerd-2.1.4-linux-amd64.tar.gz

# 解压文件
tar Czxvf /usr/local containerd-2.1.4-linux-amd64.tar.gz 

bin/
bin/containerd-stress
bin/ctr
bin/containerd
bin/containerd-shim-runc-v2

配置systemd启动文件，此处存放路径与官网有差异，systemd优先扫描此路径 /etc/systemd/system/(管理员自定义服务），其次/usr/lib/systemd/system/（包管理器安装的软件存放路径）

curl -o /etc/systemd/system/containerd.service https://raw.githubusercontent.com/containerd/containerd/main/containerd.service

cat /etc/systemd/system/containerd.service 
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target dbus.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

此处注意修改ExecStart= 为实际路径，并添加必要参数

可以输入systemctl show -p FragmentPath containerd

FragmentPath=/etc/systemd/system/containerd.service

查看实际存放路径

设置配置文件toml

mkdir -p /etc/containerd

containerd config default > /etc/containerd/config.toml

修改 containerd 配置文件

# 修改沙箱镜像
 49     [plugins.'io.containerd.cri.v1.images'.pinned_images]
 50       sandbox = 'registry.k8s.io/pause:3.10'
  sandbox = 'registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10.1'
#修改镜像仓库的配置路径
    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/containerd/certs.d'

# 添加SystemdCgroup = true
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
  ...
  [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
    SystemdCgroup = true

关于配置的pause:3.10.1的版本问题，取决于kubeadm config images pull 实际拉取的版本,通过kubeadm config images list 命令查看

配置镜像仓库，2.x版本修改了配置方法。

在 containerd 的 config.toml 中设置 config_path = "/etc/containerd/certs.d" 。在配置路径下创建一个目录树，其中包括 docker.io 作为表示要配置的主机命名空间的目录。然后在 docker.io 中添加一个 hosts.toml 文件来配置主机命名空间。

参见 docs/hosts.md 获取更多信息

$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── docker.io
    └── hosts.toml

$ cat /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]

$ cat /etc/containerd/certs.d/192.168.12.34:5000/hosts.toml
server = "https://192.168.12.34:5000"

[host."https://192.168.12.34:5000"]
  ca = "/path/to/ca.crt"
cat /etc/containerd/certs.d/docker.io/hosts.toml

server = "https://docker.io"

[host."https://docker.shaunyang.site"]
  capabilities = ["pull", "resolve"]
[host."https://docker.m.daocloud.io"]
  capabilities = ["pull", "resolve"]
[host."https://docker.1ms.run"]
  capabilities = ["pull", "resolve"]
[host."https://docker.xuanyuan.me"]
  capabilities = ["pull", "resolve"]

最后 更改完所有配置文件之后启动containerd

systemctl daemon-reload
systemctl enable --now containerd

2.2、安装runc

下载安装

wget https://github.com/opencontainers/runc/releases/download/v1.3.1/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc
runc --version
#有如下输出
runc version 1.3.1
commit: v1.3.1-0-ge6457afc
spec: 1.2.1
go: go1.23.12
libseccomp: 2.5.6

2.3、安装crictl

安装容器运行时接口CRI

wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.34.0/crictl-v1.34.0-linux-amd64.tar.gz
tar Czxvf /usr/local/bin/ crictl-v1.34.0-linux-amd64.tar.gz

配置运行时端点配置

cat > /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: true
pull-image-on-create: false
EOF

3、集群初始化

3.1、安装k8s组件

安装 kubeadm、kubelet 和 kubectl

  • kubeadm：用来初始化集群的指令。

  • kubelet：在集群中的每个节点上用来启动 Pod 和容器等。

  • kubectl：用来与集群通信的命令行工具

添加 Kubernetes 的 yum 仓库。

在仓库定义中的 exclude 参数确保了与 Kubernetes 相关的软件包在运行 yum update 时不会升级，因为升级 Kubernetes 需要遵循特定的过程。

# 此操作会覆盖 /etc/yum.repos.d/kubernetes.repo 中现存的所有配置
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

阿里镜像源

cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

安装并启动

# 清理缓存并重建数据
sudo yum clean all
sudo yum makecache
# 查询可用版本
sudo yum list available --showduplicates | grep kubeadm

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Installed:
  conntrack-tools-1.4.8-2.ky11.x86_64         containernetworking-plugins-1.2.0-4.ky11.x86_64  cri-tools-1.34.0-150500.1.1.x86_64  kubeadm-1.34.1-150500.1.1.x86_64  kubectl-1.34.1-150500.1.1.x86_64  kubelet-1.34.1-150500.1.1.x86_64  libnetfilter_cthelper-1.0.1-1.ky11.x86_64 
  libnetfilter_cttimeout-1.0.1-1.ky11.x86_64  libnetfilter_queue-1.0.5-3.ky11.x86_64

kubelet设置cgroup驱动为systemd

cat <<EOF > /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

kubelet设置开机自启动

sudo systemctl enable --now kubelet

kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环。

3.2、修改集群初始化文件

在主节点上生成配置文件 kubeadm config print init-defaults > kubeadm-config.yaml

  1 apiVersion: kubeadm.k8s.io/v1beta4
  2 bootstrapTokens:
  3 - groups:
  4   - system:bootstrappers:kubeadm:default-node-token
  5   token: abcdef.0123456789abcdef
  6   ttl: 24h0m0s
  7   usages:
  8   - signing
  9   - authentication
 10 kind: InitConfiguration
 11 localAPIEndpoint:
######################################修改################################
 12   advertiseAddress: 192.168.8.70
 13   bindPort: 6443
 14 nodeRegistration:
 15   criSocket: unix:///var/run/containerd/containerd.sock
 16   imagePullPolicy: IfNotPresent
 17   imagePullSerial: true
##################################修改#####################################
 18   name: master
 19   taints: null
 20 timeouts:
 21   controlPlaneComponentHealthCheck: 4m0s
 22   discovery: 5m0s
 23   etcdAPICall: 2m0s
 24   kubeletHealthCheck: 4m0s
 25   kubernetesAPICall: 1m0s
 26   tlsBootstrap: 5m0s
 27   upgradeManifests: 5m0s
 28 ---
 29 apiServer: {}
 30 apiVersion: kubeadm.k8s.io/v1beta4
#############################修改百年令牌期限#########################
 31 caCertificateValidityPeriod: 876000h0m0s
 32 certificateValidityPeriod: 876000h0m0s
 33 certificatesDir: /etc/kubernetes/pki
 34 clusterName: kubernetes
 35 controllerManager: {}
 36 dns: {}
 37 encryptionAlgorithm: RSA-2048
 38 etcd:
 39   local:
 40     dataDir: /var/lib/etcd
################################修改###################################
 41 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
 42 kind: ClusterConfiguration
 43 kubernetesVersion: 1.34.0
 44 networking:
 45   dnsDomain: cluster.local
 46   serviceSubnet: 10.96.0.0/12
##############################修改##################################
      podSubnet: 172.33.0.0/16
 47 proxy: {}
 48 scheduler: {}

3.3、拉取k8s镜像

kubeadm config images pull \
  --image-repository=registry.aliyuncs.com/google_containers  \
  --kubernetes-version=v1.34.1

3.4、初始化集群

kubeadm init --config kubeadm-config.yaml --upload-certs --v=9

3.5、配置kubectl权限

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 验证配置
kubectl config view

3.6、配置kube-proxy使用ipvs模式

# 编辑kube-proxy配置
kubectl edit configmap kube-proxy -n kube-system

# 在config.conf部分修改mode为ipvs
# mode: "ipvs"

# 重启kube-proxy
kubectl rollout restart daemonset kube-proxy -n kube-system

3.7、安装calico网络插件

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/tigera-operator.yaml

wget https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/custom-resources.yaml

vim custom-resources.yaml
# 需要修改成指定的podIP段
      cidr: 172.33.0.0/16

# 验证网络插件状态（所有pod状态为Running）
watch kubectl get pods -n calico-system

3.8、加入工作节点

获取token（在master节点执行）

kubeadm token create --print-join-command

加入集群（在node节点执行）

# 依据实际环境改动
kubeadm join 192.168.8.70:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:6b028dd0c8d1390ea89ed32fa9cdd811accd253c436ccc8fca9642da716dccb8

4、集群验证

1. 检查节点状态

kubectl get nodes
# 预期输出：所有节点状态为Ready

2. 检查系统组件状态

kubectl get pods -n kube-system
# 预期输出：所有pod状态为Running

kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
scheduler            Healthy   ok        
etcd-0               Healthy   ok    

kubectl get svc -A
NAMESPACE          NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
calico-apiserver   calico-api                        ClusterIP   10.98.217.92   <none>        443/TCP                  24m
calico-system      calico-kube-controllers-metrics   ClusterIP   None           <none>        9094/TCP                 9m10s
calico-system      calico-typha                      ClusterIP   10.98.13.246   <none>        5473/TCP                 24m
default            kubernetes                        ClusterIP   10.96.0.1      <none>        443/TCP                  87m
kube-system        kube-dns                          ClusterIP   10.96.0.10     <none>        53/UDP,53/TCP,9153/TCP   87m

3. 验证 dns解析功能

dig -t a www.baidu.com @10.96.0.10

; <<>> DiG 9.18.21 <<>> -t a www.baidu.com @10.96.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50661
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: c595abdf33358469 (echoed)
;; QUESTION SECTION:
;www.baidu.com.			IN	A

;; ANSWER SECTION:
www.baidu.com.		30	IN	CNAME	www.a.shifen.com.
www.a.shifen.com.	30	IN	A	220.181.111.1
www.a.shifen.com.	30	IN	A	220.181.111.232

;; Query time: 33 msec
;; SERVER: 10.96.0.10#53(10.96.0.10) (UDP)
;; WHEN: Tue Sep 30 18:22:47 CST 2025
;; MSG SIZE  rcvd: 161

4. 验证 IPVS 模式是否生效

# 查看kube-proxy日志确认IPVS模式
kubectl logs -n kube-system $(kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].metadata.name}') | grep "Using ipvs Proxier"

# 查看IPVS规则
ipvsadm -Ln
# 预期输出：包含K8s服务的IPVS规则

curl 127.0.0.1:10249/proxyMode
# 预期输出：ipvs

5. 验证集群功能

# 创建测试部署
kubectl create deployment nginx --image=nginx:1.23

# 暴露服务
kubectl expose deployment nginx --port=80 --type=NodePort

# 查看服务
kubectl get svc nginx

# 访问测试（替换为实际NodePort）
curl http://192.168.8.70:3xxxx

site logo

© 2025 sean的农场