Use the built-in cAdvisor monitoring container
CAdvisor is already built into the kubelet component, so we do not need to install it separately. The data path of cAdvisor is / api/v1/nodes/<node>/proxy/metrics
1. Add job s, update prometheus configuration

- job_name: 'kubernetes-cadvisor'
  kubernetes_sd_configs:
  - role: node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

$ kubectl apply -f prometheus-cm.yaml
$ kubectl get svc -n kube-ops |grep prometheus
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
$ curl -X POST "http://10.102.197.83:9090/-/reload "#Make Configuration Effective

Monitor apiserver
1. Add job s, update prometheus configuration

- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

$ kubectl apply -f prometheus-cm.yaml
$ kubectl get svc -n kube-ops |grep prometheus
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
$ curl -X POST "http://10.102.197.83:9090/-/reload "#Make Configuration Effective

Configure Automatic Discovery and Monitoring for Common SVCS
1. Add job s, update prometheus configuration

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

To automatically discover services in the cluster, we need to add the declaration prometheus.io/scrape=true in the annotation area of the Service
$ kubectl apply -f prometheus-cm.yaml
$ kubectl get svc -n kube-ops |grep prometheus
prometheus NodePort 10.102.197.83 <none> 9090:32619/TCP
$ curl -X POST "http://10.102.197.83:9090/-/reload "#Make Configuration Effective
2. Modify the svc of redis to dynamically discover and monitor (based on static discovery)
New annotations

kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: kube-ops
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
spec:
  selector:
    app: redis
  ports:
  - name: redis
    port: 6379
    targetPort: 6379
  - name: prom
    port: 9121
    targetPort: 9121

Add prometheus.io/scrape=true to the previously created redis Service
Since the metrics interface of the redis service is on the redis-exporter service 9121, we also need to add an annotations such as prometheus.io/port=9121
$ kubectl apply -f prome-redis.yaml
3. Modify trafik's svc to dynamically discover and monitor (based on static discovery)

apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
  annotations:
    prometheus.io/scrape: "true"        #Newly added
    prometheus.io/port: "8080"        #Newly added
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

Later we have a new service, and if the service itself provides the / metrics interface, we don't need to configure it statically at all
When service dynamic discovery is turned on, the services that are automatically discovered and monitored by default are as follows:prometheus itself, kube-dns
4. Automatically discover kube-state-metrics
Implement state monitoring of various resource objects such as Pod, DaemonSet, Deployment, Job, CronJob on the Kubernetes cluster
$ git clone https://github.com/kubernetes/kube-state-metrics.git
$ cd kube-state-metrics/kubernetes
$ kubectl apply -f .
After deploying kube-state-metrics to Kubernetes, you will find that Prometheus in the Kubernetes cluster automatically discovers kube-state-metrics under the job kubernetes-service-endpoints and starts pulling metrics because the manifest definition file kube-state-metri for deploying kube-state-metricsCs-Service.yaml's definition of Service includes an annotation such as prometheus.io/scrape:'true'