Prometheus

Удаление метрик

Для управления добавить при запуске:

--web.enable-admin-api

Удалить все метрики:

curl -X POST -g 'http://172.0.0.1:9090/api/v2/admin/tsdb/delete_series?match[]={__name__=~".+"}'
curl -X POST -g 'http://172.0.0.1:9090/api/v2/admin/tsdb/clean_tombstones'

Удалить все временные сегменты WAL:

rm -r /prometheus/wal/checkpoint.*.tmp

Удалить весь WAL:

rm -r /prometheus/wal/*

https://www.shellhacks.com/prometheus-delete-time-series-metrics/
https://help.sap.com/viewer/3f4043064eed446a895bc8ba7e61dc83/2.7.latest/en-US/8c69c688050845f9a0f15066ceb6170d.html - Handle Corrupt Prometheus Write-Ahead Log (WAL)
https://help.sap.com/viewer/3f4043064eed446a895bc8ba7e61dc83/2.7.latest/en-US/a432865bd9164d768bf0e9bf0d13f399.html - Change the Prometheus Retention Size and Time

K8s

prometheus — определяет инсталляцию (кластер) Prometheus;
servicemonitor — определяет, как мониторить набор сервисов (т.е. собирать их метрики);

Показать ресурс prometheus в yaml формате:

kubectl get prometheus cluster-monitoring-prometh-prometheus -o=yaml

Включить администрирование через api:

kubectl get prometheus cluster-monitoring-prometh-prometheus -o=yaml | sed 's/enableAdminAPI: false/enableAdminAPI: true/' | kubectl replace -f -

Пример ресурса

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  creationTimestamp: "2020-09-18T10:34:29Z"
  generation: 3
  labels:
    app: prometheus-operator-prometheus
    chart: prometheus-operator-8.15.14
    heritage: Helm
    release: cluster-monitoring
  name: cluster-monitoring-prometh-prometheus
  namespace: monitoring
  resourceVersion: "84353988"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/cluster-monitoring-prometh-prometheus
  uid: ba19030f-b46b-4aa6-9720-c8888324c70f
spec:
  alerting:
    alertmanagers: []
  baseImage: quay.io/prometheus/prometheus
  enableAdminAPI: true
  externalLabels:
    cluster: dvs-checkpoint
    datacenter: dvs-checkpoint
  externalUrl: http://cluster-monitoring-prometh-prometheus.monitoring:9090
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      release: cluster-monitoring
  portName: web
  remoteWrite:
  - queueConfig:
      batchSendDeadline: 15s
      capacity: 100000
      maxBackoff: 500ms
      maxSamplesPerSend: 20000
      maxShards: 10
      minBackoff: 30ms
      minShards: 1
    remoteTimeout: 30s
    url: http://remote.host.address:8428/api/v1/write
  replicas: 1
  retention: 5d
  routePrefix: /
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      app: prometheus-operator
      release: cluster-monitoring
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: cluster-monitoring-prometh-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 30Gi
        storageClassName: default
  version: v2.18.1

Показать custom resource (kind:) ServiceMonitor:

kubectl get servicemonitors.monitoring.coreos.com/cluster-monitoring-prometh-kubelet -o yaml

Показать аргументы запуска контейнера prometheus в pod prometheus-cluster-monitoring-prometh-prometheus-0:

kubectl get pod prometheus-cluster-monitoring-prometh-prometheus-0 -o=json | jq -r '.spec.containers[] | select( ."name" == "prometheus" ).args'

Послать команду в контейнер prometheus pod'а prometheus-prometheus-operator-prometheus-0:

kubectl -n monitoring exec prometheus-cluster-monitoring-prometh-prometheus-0 -c prometheus -- \
  sh -c 'prometheus --version'

Ссылки

https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md - API Docs
https://github.com/prometheus-community/helm-charts/tree/main/charts - Helm charts
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack - collection of Kubernetes manifests