程序人生网站首页 程序人生

62、 Helm 及其它功能性组件 - prometheus、资源限制（2

发布时间：2023-02-01 11:20:00编辑：雪饮阅读（）

Step1

自上篇

61、 Helm 及其它功能性组件 - prometheus、资源限制（1] (gaojiupan.cn)

后，其实我们应该拿到prometheus程序应该不建议之间从github拿到，因为github默认是获取最新的，和老师当时的环境是有很多变化的。

我怀疑老师的应该就是如下面这个路径的

D:\BaiduNetdiskDownload\10、Kubernetes - Helm 及其它功能性组件\鸿鹄论坛_10、Kubernetes - Helm 及其它功能性组件\2、资料\镜像文件.zip\Images\prometheus-operator\

kube-prometheus.git.tar.gz

但是解压出来的

/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/grafana-service.yaml

/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-service.yaml

/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/alertmanager-service.yaml

这三个文件配置的修改还和上篇一样

Step2

然后安装prometheus如

kubectl apply -f /usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/

该命令需要执行多次，待一些错误消失，因为有点像是交叉编译，相互依赖的。

[root@k8s-master01 prometheus]# kubectl apply -f /usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/

namespace/monitoring created

customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created

customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created

customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created

customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created

customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created

clusterrole.rbac.authorization.k8s.io/prometheus-operator created

clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created

deployment.apps/prometheus-operator created

service/prometheus-operator created

serviceaccount/prometheus-operator created

secret/alertmanager-main created

service/alertmanager-main created

serviceaccount/alertmanager-main created

secret/grafana-datasources created

configmap/grafana-dashboard-apiserver created

configmap/grafana-dashboard-controller-manager created

configmap/grafana-dashboard-k8s-cluster-rsrc-use created

configmap/grafana-dashboard-k8s-node-rsrc-use created

configmap/grafana-dashboard-k8s-resources-cluster created

configmap/grafana-dashboard-k8s-resources-namespace created

configmap/grafana-dashboard-k8s-resources-pod created

configmap/grafana-dashboard-k8s-resources-workload created

configmap/grafana-dashboard-k8s-resources-workloads-namespace created

configmap/grafana-dashboard-kubelet created

configmap/grafana-dashboard-nodes created

configmap/grafana-dashboard-persistentvolumesusage created

configmap/grafana-dashboard-pods created

configmap/grafana-dashboard-prometheus-remote-write created

configmap/grafana-dashboard-prometheus created

configmap/grafana-dashboard-proxy created

configmap/grafana-dashboard-scheduler created

configmap/grafana-dashboard-statefulset created

configmap/grafana-dashboards created

deployment.apps/grafana created

service/grafana created

serviceaccount/grafana created

clusterrole.rbac.authorization.k8s.io/kube-state-metrics created

clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created

deployment.apps/kube-state-metrics created

role.rbac.authorization.k8s.io/kube-state-metrics created

rolebinding.rbac.authorization.k8s.io/kube-state-metrics created

service/kube-state-metrics created

serviceaccount/kube-state-metrics created

clusterrole.rbac.authorization.k8s.io/node-exporter created

clusterrolebinding.rbac.authorization.k8s.io/node-exporter created

daemonset.apps/node-exporter created

service/node-exporter created

serviceaccount/node-exporter created

apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

clusterrole.rbac.authorization.k8s.io/prometheus-adapter created

clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created

clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created

clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created

clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created

configmap/adapter-config created

deployment.apps/prometheus-adapter created

rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created

service/prometheus-adapter created

serviceaccount/prometheus-adapter created

clusterrole.rbac.authorization.k8s.io/prometheus-k8s created

clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created

rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created

rolebinding.rbac.authorization.k8s.io/prometheus-k8s created

role.rbac.authorization.k8s.io/prometheus-k8s-config created

role.rbac.authorization.k8s.io/prometheus-k8s created

service/prometheus-k8s created

serviceaccount/prometheus-k8s created

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/0prometheus-operator-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/alertmanager-alertmanager.yaml": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/alertmanager-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/grafana-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/kube-state-metrics-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/node-exporter-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-prometheus.yaml": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-rules.yaml": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-serviceMonitorApiserver.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-serviceMonitorCoreDNS.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-serviceMonitorKubeControllerManager.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-serviceMonitorKubeScheduler.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

unable to recognize "/usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/prometheus-serviceMonitorKubelet.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

[root@k8s-master01 prometheus]# kubectl apply -f /usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/

namespace/monitoring unchanged

customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com unchanged

customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com unchanged

customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com unchanged

customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com unchanged

customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com unchanged

clusterrole.rbac.authorization.k8s.io/prometheus-operator unchanged

clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator unchanged

deployment.apps/prometheus-operator unchanged

service/prometheus-operator unchanged

serviceaccount/prometheus-operator unchanged

servicemonitor.monitoring.coreos.com/prometheus-operator created

alertmanager.monitoring.coreos.com/main created

secret/alertmanager-main unchanged

service/alertmanager-main unchanged

serviceaccount/alertmanager-main unchanged

servicemonitor.monitoring.coreos.com/alertmanager created

secret/grafana-datasources unchanged

configmap/grafana-dashboard-apiserver unchanged

configmap/grafana-dashboard-controller-manager unchanged

configmap/grafana-dashboard-k8s-cluster-rsrc-use unchanged

configmap/grafana-dashboard-k8s-node-rsrc-use unchanged

configmap/grafana-dashboard-k8s-resources-cluster unchanged

configmap/grafana-dashboard-k8s-resources-namespace unchanged

configmap/grafana-dashboard-k8s-resources-pod unchanged

configmap/grafana-dashboard-k8s-resources-workload unchanged

configmap/grafana-dashboard-k8s-resources-workloads-namespace unchanged

configmap/grafana-dashboard-kubelet unchanged

configmap/grafana-dashboard-nodes unchanged

configmap/grafana-dashboard-persistentvolumesusage unchanged

configmap/grafana-dashboard-pods unchanged

configmap/grafana-dashboard-prometheus-remote-write unchanged

configmap/grafana-dashboard-prometheus unchanged

configmap/grafana-dashboard-proxy unchanged

configmap/grafana-dashboard-scheduler unchanged

configmap/grafana-dashboard-statefulset unchanged

configmap/grafana-dashboards unchanged

deployment.apps/grafana configured

service/grafana unchanged

serviceaccount/grafana unchanged

servicemonitor.monitoring.coreos.com/grafana created

clusterrole.rbac.authorization.k8s.io/kube-state-metrics unchanged

clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged

deployment.apps/kube-state-metrics unchanged

role.rbac.authorization.k8s.io/kube-state-metrics unchanged

rolebinding.rbac.authorization.k8s.io/kube-state-metrics unchanged

service/kube-state-metrics unchanged

serviceaccount/kube-state-metrics unchanged

servicemonitor.monitoring.coreos.com/kube-state-metrics created

clusterrole.rbac.authorization.k8s.io/node-exporter unchanged

clusterrolebinding.rbac.authorization.k8s.io/node-exporter unchanged

daemonset.apps/node-exporter configured

service/node-exporter unchanged

serviceaccount/node-exporter unchanged

servicemonitor.monitoring.coreos.com/node-exporter created

apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged

clusterrole.rbac.authorization.k8s.io/prometheus-adapter unchanged

clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged

clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter unchanged

clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator unchanged

clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources unchanged

configmap/adapter-config unchanged

deployment.apps/prometheus-adapter configured

rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader unchanged

service/prometheus-adapter unchanged

serviceaccount/prometheus-adapter unchanged

clusterrole.rbac.authorization.k8s.io/prometheus-k8s unchanged

clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged

prometheus.monitoring.coreos.com/k8s created

rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config unchanged

rolebinding.rbac.authorization.k8s.io/prometheus-k8s unchanged

role.rbac.authorization.k8s.io/prometheus-k8s-config unchanged

role.rbac.authorization.k8s.io/prometheus-k8s unchanged

prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created

service/prometheus-k8s unchanged

serviceaccount/prometheus-k8s unchanged

servicemonitor.monitoring.coreos.com/prometheus created

servicemonitor.monitoring.coreos.com/kube-apiserver created

servicemonitor.monitoring.coreos.com/coredns created

servicemonitor.monitoring.coreos.com/kube-controller-manager created

servicemonitor.monitoring.coreos.com/kube-scheduler created

servicemonitor.monitoring.coreos.com/kubelet created

Step3

然后我们需要建立一个metrics的yaml模板

[root@k8s-master01 prometheus]# cat metrics.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

labels:

k8s-app: metrics-server

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

labels:

k8s-app: metrics-server

rbac.authorization.k8s.io/aggregate-to-admin: "true"

rbac.authorization.k8s.io/aggregate-to-edit: "true"

rbac.authorization.k8s.io/aggregate-to-view: "true"

rules:

- apiGroups:

- metrics.k8s.io

resources:

- pods

- nodes

verbs:

- get

- list

- watch

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

labels:

k8s-app: metrics-server

rules:

- apiGroups:

- ""

resources:

- pods

- nodes

- nodes/stats

- namespaces

- configmaps

verbs:

- get

- list

- watch

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

labels:

k8s-app: metrics-server

namespace: kube-system

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: Role

subjects:

- kind: ServiceAccount

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

k8s-app: metrics-server

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- kind: ServiceAccount

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

labels:

k8s-app: metrics-server

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- kind: ServiceAccount

namespace: kube-system

---

apiVersion: v1

kind: Service

metadata:

labels:

k8s-app: metrics-server

namespace: kube-system

spec:

ports:

- name: https

port: 443

protocol: TCP

targetPort: https

selector:

k8s-app: metrics-server

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

k8s-app: metrics-server

namespace: kube-system

spec:

selector:

matchLabels:

k8s-app: metrics-server

strategy:

rollingUpdate:

maxUnavailable: 0

template:

metadata:

labels:

k8s-app: metrics-server

spec:

hostNetwork: true

containers:

- args:

- --cert-dir=/tmp

- --secure-port=4443

- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

- --kubelet-use-node-status-port

- --metric-resolution=15s

- --logtostderr

- --kubelet-insecure-tls

image: k8s.gcr.io/metrics-server/metrics-server:v0.5.2

imagePullPolicy: IfNotPresent

livenessProbe:

failureThreshold: 3

httpGet:

path: /livez

port: https

scheme: HTTPS

periodSeconds: 10

ports:

- containerPort: 4443

protocol: TCP

readinessProbe:

failureThreshold: 3

httpGet:

path: /readyz

port: https

scheme: HTTPS

initialDelaySeconds: 20

periodSeconds: 10

resources:

requests:

cpu: 100m

memory: 200Mi

securityContext:

readOnlyRootFilesystem: true

runAsNonRoot: true

runAsUser: 1000

volumeMounts:

- mountPath: /tmp

nodeSelector:

kubernetes.io/os: linux

priorityClassName: system-cluster-critical

serviceAccountName: metrics-server

volumes:

- emptyDir: {}

---

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

labels:

k8s-app: metrics-server

spec:

group: metrics.k8s.io

groupPriorityMinimum: 100

insecureSkipTLSVerify: true

service:

namespace: kube-system

version: v1beta1

versionPriority: 100

标红的这几处是值得注意的地方

然后创建

[root@k8s-master01 prometheus]# kubectl create -f metrics.yaml

serviceaccount/metrics-server created

clusterrole.rbac.authorization.k8s.io/system:metrics-server created

rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created

clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created

clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

service/metrics-server created

deployment.apps/metrics-server created

Error from server (AlreadyExists): error when creating "metrics.yaml": clusterroles.rbac.authorization.k8s.io "system:aggregated-metrics-reader" already exists

Error from server (AlreadyExists): error when creating "metrics.yaml": apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io" already exists

若有已存在的这种错误可不予理会

Step4

接下来需要各个节点用docker导入所依赖的镜像

k8s.gcr.io/metrics-server/metrics-server:v0.5.2

然后这三个还是老出问题。。。

[root@k8s-master01 prometheus]# kubectl get pod -n monitoring -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

grafana-7dc5f8f9f6-d7rzm 1/1 Running 0 37m 10.224.2.19 k8s-node01 <none> <none>

kube-state-metrics-5964dbdd99-wz6lh 3/4 CrashLoopBackOff 2 2m10s 10.224.1.252 k8s-node02 <none> <none>

node-exporter-gm84j 2/2 Running 0 37m 192.168.66.10 k8s-master01 <none> <none>

node-exporter-pbngc 2/2 Running 0 37m 192.168.66.21 k8s-node02 <none> <none>

node-exporter-wzw5x 2/2 Running 0 37m 192.168.66.20 k8s-node01 <none> <none>

prometheus-adapter-668748ddbd-x7964 0/1 CrashLoopBackOff 11 37m 10.224.2.20 k8s-node01 <none> <none>

prometheus-operator-7447bf4dcb-dgsm9 0/1 CrashLoopBackOff 11 37m 10.224.1.250 k8s-node02 <none> <none>

所以我修改到master节点固定调度上哈

kubectl edit deployment kube-state-metrics -n monitoring

kubectl edit deployment prometheus-adapter -n monitoring

kubectl edit deployment prometheus-operator -n monitoring

spec=>template=>spec:

nodeName: k8s-master01

都running了。。。

[root@k8s-master01 prometheus]# kubectl get pod -n monitoring -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

alertmanager-main-0 1/2 Running 0 26s 10.224.2.22 k8s-node01 <none> <none>

grafana-7dc5f8f9f6-d7rzm 1/1 Running 0 44m 10.224.2.19 k8s-node01 <none> <none>

kube-state-metrics-6b976749f9-s4nnp 4/4 Running 0 94s 10.224.0.23 k8s-master01 <none> <none>

node-exporter-gm84j 2/2 Running 0 44m 192.168.66.10 k8s-master01 <none> <none>

node-exporter-pbngc 2/2 Running 0 44m 192.168.66.21 k8s-node02 <none> <none>

node-exporter-wzw5x 2/2 Running 0 44m 192.168.66.20 k8s-node01 <none> <none>

prometheus-adapter-6998484678-9trn4 1/1 Running 0 68s 10.224.0.24 k8s-master01 <none> <none>

prometheus-k8s-0 3/3 Running 0 16s 10.224.1.253 k8s-node02 <none> <none>

prometheus-k8s-1 3/3 Running 1 16s 10.224.2.23 k8s-node01 <none> <none>

prometheus-operator-5dc9567958-x8996 1/1 Running 0 30s 10.224.0.25 k8s-master01 <none> <none>

虽然我这个方法不太好，但是我觉得这种应该都是部署master算是正常的吧

Step5

稍待片刻就有节点数据了

[root@k8s-master01 prometheus]# kubectl top node

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%

k8s-master01 90m 2% 1099Mi 28%

k8s-node01 41m 1% 625Mi 16%

k8s-node02 36m 0% 599Mi 15%

但pod数据为空。。。

[root@k8s-master01 prometheus]# kubectl top pod

[root@k8s-master01 prometheus]#

但这样却能看到

[root@k8s-master01 prometheus]# kubectl top pod --all-namespaces

NAMESPACE NAME CPU(cores) MEMORY(bytes)

kube-system coredns-5c98db65d4-d78lt 3m 15Mi

kube-system coredns-5c98db65d4-smd45 3m 13Mi

kube-system etcd-k8s-master01 11m 86Mi

kube-system kube-apiserver-k8s-master01 17m 227Mi

kube-system kube-controller-manager-k8s-master01 14m 61Mi

kube-system kube-flannel-ds-amd64-5wzvq 2m 14Mi

kube-system kube-flannel-ds-amd64-9rxst 2m 14Mi

kube-system kube-flannel-ds-amd64-jfgkk 2m 16Mi

kube-system kube-proxy-fdxwt 6m 23Mi

kube-system kube-proxy-stpht 5m 22Mi

kube-system kube-proxy-zq4n4 4m 23Mi

kube-system kube-scheduler-k8s-master01 1m 15Mi

kube-system metrics-server-d4cd976cb-dvjwd 3m 24Mi

monitoring alertmanager-main-0 2m 18Mi

monitoring alertmanager-main-1 2m 20Mi

monitoring alertmanager-main-2 2m 18Mi

monitoring grafana-7dc5f8f9f6-d7rzm 2m 40Mi

monitoring kube-state-metrics-6b976749f9-s4nnp 1m 51Mi

monitoring node-exporter-gm84j 1m 31Mi

monitoring node-exporter-pbngc 1m 15Mi

monitoring node-exporter-wzw5x 1m 16Mi

monitoring prometheus-adapter-6998484678-9trn4 1m 11Mi

monitoring prometheus-k8s-0 4m 40Mi

monitoring prometheus-k8s-1 5m 39Mi

monitoring prometheus-operator-5dc9567958-x8996 1m 23Mi

或者只看monitoring的

[root@k8s-master01 prometheus]# kubectl top pod -n monitoring

NAME CPU(cores) MEMORY(bytes)

alertmanager-main-0 2m 18Mi

alertmanager-main-1 2m 20Mi

alertmanager-main-2 2m 18Mi

grafana-7dc5f8f9f6-d7rzm 3m 40Mi

kube-state-metrics-6b976749f9-s4nnp 1m 51Mi

node-exporter-gm84j 0m 31Mi

node-exporter-pbngc 1m 15Mi

node-exporter-wzw5x 1m 16Mi

prometheus-adapter-6998484678-9trn4 1m 11Mi

prometheus-k8s-0 3m 40Mi

prometheus-k8s-1 5m 38Mi

prometheus-operator-5dc9567958-x8996 1m 23Mi

看来是名称空间之间的问题，那么就是说我的环境里面可能和老师的某些个软件或插件什么版本不同吧，某些特性，特别像是拿yaml里面的配置变了之类的吧。

Step6

那么此时可以通过master节点访问prometheus的web端了

http://192.168.66.10:30200/

奇怪的现象是targets这里没有内容

我的猜测是因为上面那三个插件部署到了master节点的原因吧。暂时没有比较好的解决方案，暂时先放放。

但我在公司的环境时候是这样的

有健康的和不健康的，我当初怀疑是由于在master上，但是也刚开始不在master上的，可能是有之前收集的数据的残留吧。

。。。

据说全部为up状态才算是真正的成功了。

Step7

关于上面的没有targets的问题，好像是时间同步问题

各节点执行如

systemctl enable ntpd

ntpdate time1.aliyun.com

并且访问prometheus的浏览器所在宿主机时间也同步

然后原来的流程重新走下。。。

kubectl delete -f /usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/

耐心等待monitoring名称空间消失

kubectl get ns

Prometheus重建

kubectl apply -f /usr/local/install-k8s/plugin/prometheus/kube-prometheus/manifests/

迁移到master

kubectl edit deployment kube-state-metrics -n monitoring

kubectl edit deployment prometheus-adapter -n monitoring

kubectl edit deployment prometheus-operator -n monitoring

spec=>template=>spec:

nodeName: k8s-master01

上面我们建立的metrics重建

kubectl delete -f metrics.yaml

kubectl create -f metrics.yaml

然而。。。没有用

先不管了。

Step8

应该达到的效果

回到prometheus首页一个查询语句

sum by (pod_name)( rate(container_cpu_usage_seconds_total{image!="", pod_name!=""}[1m] ))

点击graph看到图表

Step9

进入grafana

http://192.168.66.10:30100/login

账号与密码默认都是admin

没有修改过默认密码就会提示建议修改下密码

Step10

默认已经添加有数据源

进入默认的prometheus数据源

测试这个默认数据源

默认数据源一般是没有问题的

Step11

默认数据源中有包含几个模板可以导入，这里都导入下

直接点击home不用下拉选择

进入界面后选择查看节点的信息

奇怪的是只能选择master节点

直点不下拉

选择查看pod

可以看到pod的情况

Step12

各节点导入hpa高可用docker示例镜像

D:\BaiduNetdiskDownload\10、Kubernetes - Helm 及其它功能性组件\鸿鹄论坛_10、Kubernetes - Helm 及其它功能性组件\2、资料\镜像文件.zip\Images\metrics\hpa-example.tar

Master创建hpa测试用例

[root@k8s-master01 ~]# kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80

kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.

service/php-apache created

deployment.apps/php-apache created

这是一个php编写的测试用例，很耗性能的程序。

可能是上面hpa-example.tar里面或者这个pod（倾向后者）有latest的镜像依赖，所以我这里还要导入一个镜像

gcr.io/google_containers/hpa-example

否则php-apache这个pod出现ErrImagePull异常

导入镜像后还是出现ErrImagePull异常

编辑deployment将镜像拉取策略imagePullPolicy从Always修改为IfNotPresent

然后就ok了

[root@k8s-master01 ~]# kubectl get pod

NAME READY STATUS RESTARTS AGE

php-apache-799f99c985-4txtd 1/1 Running 0 6s

[root@k8s-master01 ~]# kubectl get pod

NAME READY STATUS RESTARTS AGE

php-apache-799f99c985-4txtd 1/1 Running 0 12s

终于解决了之前kubectl top pod没有数据的情况

[root@k8s-master01 ~]# kubectl top pod

NAME CPU(cores) MEMORY(bytes)

php-apache-799f99c985-4txtd 1m 10Mi

[root@k8s-master01 ~]# kubectl top pod php-apache-799f99c985-4txtd

NAME CPU(cores) MEMORY(bytes)

php-apache-799f99c985-4txtd 1m 10Mi

可能老师那个有数据是因为之前有hpa类似pod残留吧

或者就是在默认的名称空间中有pod吧，昨天我好像是清理了一大堆pod。。。

Step13

新的问题，hpa没有资源

[root@k8s-master01 ~]# kubectl get hpa

No resources found.

[root@k8s-master01 ~]# kubectl get hpa --all-namespaces

No resources found.

[root@k8s-master01 ~]# kubectl get rc

No resources found.

[root@k8s-master01 ~]# kubectl get rc --all-namespaces

No resources found.

可能原因是rc没有资源

增加负载

[root@k8s-master01 ~]# kubectl run -i --tty load-generator --image=busybox /bin/sh

kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.

If you don't see a command prompt, try pressing enter.

/ #

不断请求刚才这个php-apache的pod，以消耗性能

If you don't see a command prompt, try pressing enter.

/ #

/ # while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done

OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!

另开会话可以看到Hpa仍旧没有资源

[root@k8s-master01 ~]# kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

load-generator-7d549cd44-zxxms 1/1 Running 0 6m1s 10.224.2.69 k8s-node02 <none> <none>

php-apache-799f99c985-4txtd 1/1 Running 0 29m 10.224.2.68 k8s-node02 <none> <none>

[root@k8s-master01 ~]# kubectl get hpa

No resources found.

难道是因为节点都相同？

所以如这个暂时也不会出来了

消耗服务的脚本不断执行，按理来说pod会不断创建新的副本以满足所需。但实际上我这里看不出这种情况。

[root@k8s-master01 ~]# kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

load-generator-7d549cd44-zxxms 1/1 Running 0 12m 10.224.2.69 k8s-node02 <none> <none>

php-apache-799f99c985-4txtd 1/1 Running 0 36m 10.224.2.68 k8s-node02 <none> <none>
按老师的实验应该是等到10个pod副本（含pod本体），老师实际最后好像出了9个，但我这里一直是没有变化。

可能是忘了创建hpa控制器。。。

[root@k8s-master01 ~]# kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

horizontalpodautoscaler.autoscaling/php-apache autoscaled

终于看到了变化

[root@k8s-master01 ~]# kubectl get pod -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

load-generator-7d549cd44-zxxms 1/1 Running 0 18m 10.224.2.69 k8s-node02 <none> <none>

php-apache-799f99c985-4txtd 1/1 Running 0 41m 10.224.2.68 k8s-node02 <none> <none>

php-apache-799f99c985-7tzgv 1/1 Running 0 8s 10.224.2.72 k8s-node02 <none> <none>

php-apache-799f99c985-97l4d 1/1 Running 0 24s 10.224.2.70 k8s-node02 <none> <none>

php-apache-799f99c985-c9blg 0/1 ContainerCreating 0 8s <none> k8s-node01 <none> <none>

php-apache-799f99c985-gx7mk 1/1 Running 0 8s 10.224.2.71 k8s-node02 <none> <none>

php-apache-799f99c985-kk2ch 0/1 ContainerCreating 0 24s <none> k8s-node01 <none> <none>

php-apache-799f99c985-mpls4 0/1 ContainerCreating 0 24s <none> k8s-master01 <none> <none>

php-apache-799f99c985-rrhkg 0/1 ContainerCreating 0 8s <none> k8s-master01 <none> <none>

hpa也有数据了

[root@k8s-master01 ~]# kubectl get hpa

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

php-apache Deployment/php-apache 113%/50% 1 10 10 65s

[root@k8s-master01 ~]# kubectl get hpa

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

php-apache Deployment/php-apache 113%/50% 1 10 10 73s

关键字词：prometheus

上一篇：60、 Helm 及其它功能性组件 - Dashboard](dial tcp 10.96.0.1443 io timeout问题解决-master方案)

下一篇：63、Kubernetes - Helm 及其它功能性组件 - prometheus、资源限制（3]

您当前的位置： 首页 > 慢生活 > 程序人生 网站首页程序人生

62、 Helm 及其它功能性组件 - prometheus、资源限制（2

相关文章

您当前的位置：首页 > 慢生活 > 程序人生网站首页程序人生