其他分享
首页 > 其他分享> > 记一次k8s问题处理 => 6443端口没有监听 => 证书过期处理

记一次k8s问题处理 => 6443端口没有监听 => 证书过期处理

作者:互联网

原始的问题是这样:

The connection to the server 192.168.122.200:6443 was refused - did you specify the right host or port?

lsof -i :6443

端口没有监听 

master 的kubelet.service 是active的,docker容器api_server是down的

node的 kubelet.service 是 dead, docker容器都是down的

journalctl -xefu kubelet 看到的日志都是连不上6443端口

后面找到容器kube-apiserver查看日志:

 

 

 似乎是证书过期了。。。

以下是解决步骤:

一. 检查证书是否过期。

可以通过下面两种方式检查 Kubernetes 的证书是否过期。

1. kubeadm 命令查看

可以通过 kubeadm alpha certs check-expiration 命令查看相关证书是否过期。

[root@master ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 08, 2023 14:11 UTC   364d                                    no      
apiserver                  Jul 08, 2023 14:11 UTC   364d            ca                      no      
apiserver-etcd-client      Jul 08, 2023 14:11 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Jul 08, 2023 14:11 UTC   364d            ca                      no      
controller-manager.conf    Jul 08, 2023 14:11 UTC   364d                                    no      
etcd-healthcheck-client    Jul 08, 2023 14:11 UTC   364d            etcd-ca                 no      
etcd-peer                  Jul 08, 2023 14:11 UTC   364d            etcd-ca                 no      
etcd-server                Jul 08, 2023 14:11 UTC   364d            etcd-ca                 no      
front-proxy-client         Jul 08, 2023 14:11 UTC   364d            front-proxy-ca          no      
scheduler.conf             Jul 08, 2023 14:11 UTC   364d                                    no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 02, 2031 14:04 UTC   8y              no      
etcd-ca                 Jul 02, 2031 14:04 UTC   8y              no      
front-proxy-ca          Jul 02, 2031 14:04 UTC   8y              no      

2. openssl 命令查看

版本过低无法使用 kubeadm 命令时,可以通过 openssl 查看对应证书是否过期。

[root@master ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Jul  4 14:04:27 2021 GMT
            Not After : Jul  8 14:11:17 2023 GMT

二. 自动更新证书

Kubenetes 在升级控制面板相关组件时会主动更新证书,因此如果保证 Kubernetes 能够定期(一年以内)升级的话,证书会自动更新。

三. 手动更新证书

1. 证书备份

cp -rp /etc/kubernetes /etc/kubernetes.bak

2. 删除旧的证书

将 /etc/kubernetes/pki 下要重新生成的证书删除

sudo rm -rf /etc/kubernetes/pki/apiserver.key

3. 重新生成证书

主要通过 kubeadm alpha certs renew 命令生成,命令简介如下

kubeadm alpha certs renew                                                                                         
Usage:
  kubeadm alpha certs renew [flags]
  kubeadm alpha certs renew [command]

Available Commands:
  all                      renew all available certificates
  apiserver                Generates the certificate for serving the Kubernetes API
  apiserver-etcd-client    Generates the client apiserver uses to access etcd
  apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet
  etcd-healthcheck-client  Generates the client certificate for liveness probes to healtcheck etcd
  etcd-peer                Generates the credentials for etcd nodes to communicate with each other
  etcd-server              Generates the certificate for serving etcd
  front-proxy-client       Generates the client for the front proxy

重新生成所有证书

[root@master ~]# kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

W0708 11:36:20.404836    6022 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

4. 重新生成配置文件

备份旧的配置

mv /etc/kubernetes/*.conf /tmp/

生成新的配置

主要通过 kubeadm init phase kubeconfig 命令执行:

kubeadm init phase kubeconfig -h                                                                       
Usage:
  kubeadm init phase kubeconfig [flags]
  kubeadm init phase kubeconfig [command]

Available Commands:
  admin              Generates a kubeconfig file for the admin to use and for kubeadm itself
  all                Generates all kubeconfig files
  controller-manager Generates a kubeconfig file for the controller manager to use
  kubelet            Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
  scheduler          Generates a kubeconfig file for the scheduler to use

5. 后续操作

完成证书和配置文件的更新后,需要进行一系列后续操作保证更新生效,主要包括重启 kubelet、更新管理配置。

systemctl restart kubelet

将新生成的 admin.conf 文件拷贝,替换 ~/.kube 目录下的 config 文件。

cp /etc/kubernetes/admin.conf ~/.kube/config

至此master节点就恢复好了

 

恢复node节点

关于token失效, 重新生成

master集群初始化后,token24小时后就会失效,如果到了token失效时间,node再加入集群,需要重新生产token:

[root@master ~]# kubeadm token list
[root@master ~]# 

token没有了

重新生产token

[root@master ~]# kubeadm token create
W0708 11:48:22.716513   15630 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
aju641.rs9sumbg24v80hld
获取--discovery-token-ca-cert-hash值
[root@master ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //'
bb184b9cab8543539620b7736017514058ffe925ea04a8bed4c38465aaa004ae
加入集群命令--在node节点执行




 

 

 

https://blog.csdn.net/a1308422754/article/details/107157009/

https://blog.csdn.net/Ahri_J/article/details/107466921

标签:14,certificate,处理,Jul,kubelet,6443,etcd,kubeadm,k8s
来源: https://www.cnblogs.com/jay763190097/p/16459982.html