Failed to list *v1.CronJob: the server could not find the requested resource
作者:互联网
基础环境
kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-12T12:40:09Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.15-eks-84b4fe6", GitCommit:"c813d094d1e9fa409079753049cfaa9a67524ed2", GitTreeState:"clean", BuildDate:"2022-06-09T18:20:31Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
Prometheus 告警信息
{container="kube-rbac-proxy-self", endpoint="https-self", instance="10.20.108.17:9443", job="kube-state-metrics", namespace="monitoring", pod="kube-state-metrics-695cc54b57-5nkv2", resource="*v1.CronJob", result="error", service="kube-state-metrics"}
0.00792311111111111
{container="kube-rbac-proxy-self", endpoint="https-self", instance="10.20.108.17:9443", job="kube-state-metrics", namespace="monitoring", pod="kube-state-metrics-695cc54b57-5nkv2", resource="*v1.PodDisruptionBudget", result="error", service="kube-state-metrics"}
0.00792311111111111
问题分析
直接查看 k8s 中 kube-state-metrics pod 的报错;
报错如下:
W0815 04:57:34.763696 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1.PodDisruptionBudget: the server could not find the requested resource
E0815 04:57:34.763739 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: the server could not find the requested resource
W0815 04:57:42.762271 1 refl
问题排查
网上找了不少文档,google了不少材料,发现一个比较有价值的链接:
https://github.com/kubernetes/kube-state-metrics/issues/295
https://github.com/kubernetes/kube-state-metrics/issues/295
按照文档中所说,修改了 sa 权限,然后指定 监控指标。(去除 告警的指标)
但是还是不好使
问题解决
这套监控使用 https://github.com/prometheus-operator/prometheus-operator 来部署的;
安装完后,并没有做定制化的修改;
经最后查看,是 kubeStateMetrics 的版本 和 k8s 版本不兼容,导致有些功能不能支持;
对应的版本信息如下:
线上环境使用的版本如下:
最后修改 kube-state-metrics 的版本为 v2.3.0,然后重新部署,问题解决;
标签:requested,CronJob,resource,metrics,v1,state,go,kube 来源: https://www.cnblogs.com/Star-Haitian/p/16587928.html