Observability

Post Installation Dependencies

The following third-party components are required in the same Kubernetes cluster where Container Storage Modules Observability has been deployed:

• Prometheus
• Grafana
• Other Deployment Methods

There are various ways to deploy these components. We recommend following the Helm deployments according to the specifications defined below.

Tip: Container Storage Modules Observability must be deployed first. Once the module has been deployed, you can proceed to deploying/configuring Prometheus and Grafana.

Prometheus

Prometheus and Container Storage Modules Observability services run on the same Kubernetes cluster, with Container Storage Modules sending metrics to the OpenTelemetry Collector, which Prometheus then scrapes for data.

Note: It is the user’s responsibility to provide persistent storage for Prometheus if they want to preserve historical data.

Prometheus Helm Deployment

Here’s a minimal Prometheus configuration using insecure skip verify; for proper TLS, add a ca_file signed by the same CA as the Container Storage Modules Observability certificate. More details about Prometheus configuration, see Prometheus configuration.

Note: Replace OTEL-COLLECTOR-NAMESPACE with the namespace where otel-collector service is running in prometheus-values.yaml.

1. Create a values file named prometheus-values.yaml.

   # prometheus-values.yaml
   alertmanager:
     enabled: false
   nodeExporter:
     enabled: false
   pushgateway:
     enabled: false
   kube-state-metrics:
     enabled: false
   configmapReload:
     prometheus:
       enabled: false
   server:
     enabled: true
     image:
       repository: quay.io/prometheus/prometheus
       tag: v2.34.0
       pullPolicy: IfNotPresent
     persistentVolume:
       enabled: false
     service:
       type: NodePort
       servicePort: 9090
   extraScrapeConfigs: |
     - job_name: 'karavi-metrics-[CSI-DRIVER]'
       scrape_interval: 5s
       scheme: https
       tls_config:
         insecure_skip_verify: true
       metrics_path: /metrics
       kubernetes_sd_configs:
         - role: endpoints
           namespaces:
             names:
               - [OTEL-COLLECTOR-NAMESPACE]
       relabel_configs:
         - source_labels:
             - __meta_kubernetes_service_label_app_kubernetes_io_instance
             - __meta_kubernetes_service_label_app_kubernetes_io_name
           action: keep
           regex: karavi-observability;otel-collector
         - source_labels: [__meta_kubernetes_endpoint_port_name]
           action: keep
           regex: exporter-https
         - source_labels: [__address__]
           target_label: __address__
           regex: (.+):\d+
           replacement: ${1}:8443
   

   To enable scraping of Kubernetes object state metrics, set kube-state-metrics.enabled to true in the prometheus-values.yaml configuration file.

   To scrape the KubeVirt metrics, include the following scrape config to the prometheus-values.yaml configuration file.

      - job_name: 'kubevirt-metrics'
        scrape_interval: 10s
        # Kubernetes service discovery to find the kubevirt-prometheus-metrics service
        kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              # Ensure this is the correct namespace where KubeVirt is installed
              names: [KUBEVIRT-NAMESPACE]
   
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            regex: 'kubevirt-prometheus-metrics'
            action: keep
   
          - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_service_port_name]
            regex: '(.+);https-metrics'
            target_label: '__address__'
            replacement: '$1:8443'
   
        scheme: https
        tls_config:
          insecure_skip_verify: true
   

   To scrape the KubeVirt metrics, include the following scrape config to the prometheus-values.yaml configuration file.

      - job_name: 'kubevirt-metrics'
        scrape_interval: 10s
        # Kubernetes service discovery to find the kubevirt-prometheus-metrics service
        kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              # Ensure this is the correct namespace where KubeVirt is installed
              names: [KUBEVIRT-NAMESPACE]
   
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            regex: 'kubevirt-prometheus-metrics'
            action: keep
   
          - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_service_port_name]
            regex: '(.+);https-metrics'
            target_label: '__address__'
            replacement: '$1:8443'
   
        scheme: https
        tls_config:
          insecure_skip_verify: true
   

   To scrape the KubeVirt metrics, include the following scrape config to the prometheus-values.yaml configuration file.

      - job_name: 'kubevirt-metrics'
        scrape_interval: 10s
        # Kubernetes service discovery to find the kubevirt-prometheus-metrics service
        kubernetes_sd_configs:
          - role: endpoints
            namespaces:
              # Ensure this is the correct namespace where KubeVirt is installed
              names: [KUBEVIRT-NAMESPACE]
   
        relabel_configs:
          - source_labels: [__meta_kubernetes_service_name]
            regex: 'kubevirt-prometheus-metrics'
            action: keep
   
          - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_service_port_name]
            regex: '(.+);https-metrics'
            target_label: '__address__'
            replacement: '$1:8443'
   
        scheme: https
        tls_config:
          insecure_skip_verify: true
   

   To scrape powerstore metrics using Powerstore Metrics Exporter, add the following configurations to prometheus-values.yaml. More details can be found here

        - job_name: powerstore_10.0.0.1_cluster
          honor_timestamps: true
          scrape_interval: 15m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/cluster
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_port
          honor_timestamps: true
          scrape_interval: 5m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/port
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_file
          honor_timestamps: true
          scrape_interval: 5m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/file
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_hardware
          honor_timestamps: true
          scrape_interval: 10m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/hardware
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_volume
          honor_timestamps: true
          scrape_interval: 5m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/volume
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_appliance
          honor_timestamps: true
          scrape_interval: 5m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/appliance
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_nas
          honor_timestamps: true
          scrape_interval: 5m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/nas
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_volumeGroup
          honor_timestamps: true
          scrape_interval: 5m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/volumeGroup
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
        - job_name: powerstore_10.0.0.1_capacity
          honor_timestamps: true
          scrape_interval: 15m
          scrape_timeout: 3m
          metrics_path: /metrics/10.0.0.1/capacity
          scheme: http
          follow_redirects: true
          static_configs:
            - targets:
                - 127.0.0.1:9010
   

   Replace 10.0.0.1 with the powerstore array IP.

2. If using Rancher, create a ServiceMonitor.

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: otel-collector
     namespace: powerflex
   spec:
     endpoints:
     - path: /metrics
       port: exporter-https
       scheme: https
       tlsConfig:
         insecureSkipVerify: true
     selector:
       matchLabels:
         app.kubernetes.io/instance: karavi-observability
         app.kubernetes.io/name: otel-collector
   

3. Add the Prometheus Helm chart repository.

   On your terminal, run each of the commands below:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo add stable https://charts.helm.sh/stable
   helm repo update
   

4. Install the Helm chart.

   On your terminal, run the command below:

   helm install prometheus prometheus-community/prometheus -n [CSM_NAMESPACE] -f prometheus-values.yaml
   

Grafana

The Grafana dashboards require Grafana to be deployed in the same Kubernetes cluster as Container Storage Modules Observability. Below are the configuration details required to properly set up Grafana to work with Container Storage Modules Observability.

Grafana must be configured with the following data sources/plugins:

Settings for the Grafana Prometheus data source:

Grafana Helm Deployment

Below are the steps to deploy a new Grafana instance into your Kubernetes cluster:

1. Create a ConfigMap.

   When using a network that requires a decryption certificate, the Grafana server MUST be configured with the necessary certificate. If no certificate is required, skip to step 2.

   • Create a Config file named grafana-configmap.yaml The file should look like this:

   # grafana-configmap.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: certs-configmap
     namespace: [CSM_NAMESPACE]
     labels:
       certs-configmap: "1"
   data:
     ca-certificates.crt: |-
       -----BEGIN CERTIFICATE-----
     ReplaceMeWithActualCaCERT=
       -----END CERTIFICATE-----
   

   NOTE: you need an actual CA Cert for it to work

   On your terminal, run the commands below:

   kubectl create -f grafana-configmap.yaml
   

2. Create a values file.

   Create a Config file named grafana-values.yaml The file should look like this:

   # grafana-values.yaml 
   image:
     repository: grafana/grafana
     tag: 11.5.2
     sha: ""
     pullPolicy: IfNotPresent
   service:
     type: NodePort
   
   ## Administrator credentials when not using an existing Secret
   adminUser: admin
   adminPassword: admin
   
   ## Configure grafana datasources
   ## ref: http://docs.grafana.org/administration/provisioning/#datasources
   ##
   datasources:
     datasources.yaml:
       apiVersion: 1
       datasources:
       - name: Prometheus
         type: prometheus
         access: proxy
         url: 'http://prometheus-server:9090'
         isDefault: null
         version: 1
         editable: true
   testFramework:
     enabled: false
   sidecar:
     datasources:
       enabled: true
     dashboards:
       enabled: true
   
   ## Additional grafana server ConfigMap mounts
   ## Defines additional mounts with ConfigMap. ConfigMap must be manually created in the namespace.
   extraConfigmapMounts: [] # If you created a ConfigMap on the previous step, delete [] and uncomment the lines below 
   #   - name: certs-configmap
   #     mountPath: /etc/ssl/certs/ca-certificates.crt
   #     subPath: ca-certificates.crt
   #     configMap: certs-configmap
   #     readOnly: true
   

3. Add the Grafana Helm chart repository.

   On your terminal, run each of the commands below:

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   

4. Install the Helm chart.

   On your terminal, run the commands below:

   helm install grafana grafana/grafana -n [CSM_NAMESPACE] -f grafana-values.yaml
   

Other Deployment Methods

• Grafana Labs Operator Deployment

• Rancher Monitoring and Alerting Deployment

Importing Container Storage Modules for Observability Dashboards

Once Grafana is properly configured, you can import the pre-built observability dashboards. Log into Grafana and click the + icon in the side menu. Then click Import. From here you can upload the JSON files or paste the JSON text directly into the text area. Below are the locations of the dashboards that can be imported:

kubevirt_vmi_info * on(vmi_pod, namespace) group_left(persistentvolumeclaim) label_replace(kube_pod_spec_volumes_persistentvolumeclaims_info, "vmi_pod", "$1", "pod", "(.*)")

kubevirt_vmi_info * on(vmi_pod, namespace) group_left(persistentvolumeclaim) label_replace(kube_pod_spec_volumes_persistentvolumeclaims_info, "vmi_pod", "$1", "pod", "(.*)")

kubevirt_vmi_info * on(vmi_pod, namespace) group_left(persistentvolumeclaim) label_replace(kube_pod_spec_volumes_persistentvolumeclaims_info, "vmi_pod", "$1", "pod", "(.*)")

Dynamic Configuration

Some parameters can be configured/updated during runtime without restarting the Container Storage Modules for Observability services. These parameters will be stored in ConfigMaps that can be updated on the Kubernetes cluster. This will automatically change the settings on the services.

To update any of these settings, run the following command on the Kubernetes cluster then save the updated ConfigMap data.

kubectl edit configmap [CONFIG_MAP_NAME] -n [CSM_NAMESPACE]

Tracing

Container Storage Modules Observability is instrumented to report trace data to Zipkin. This helps gather timing data needed to troubleshoot latency problems with Container Storage Modules Observability. Follow the instructions below to enable the reporting of trace data:

1. Deploy a Zipkin instance in the CSM namespace and expose the service as NodePort for external access.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: zipkin
     labels:
       app.kubernetes.io/name: zipkin
       app.kubernetes.io/instance: zipkin-instance
       app.kubernetes.io/managed-by: zipkin-service
   spec:
     replicas: 1
     selector:
       matchLabels:
         app.kubernetes.io/name: zipkin
         app.kubernetes.io/instance: zipkin-instance
     template:
       metadata:
         labels:
           app.kubernetes.io/name: zipkin
           app.kubernetes.io/instance: zipkin-instance
       spec:
         containers:
           - name: zipkin
             image: "openzipkin/zipkin"
             imagePullPolicy: IfNotPresent
             env:
             - name: "STORAGE_TYPE"
               value: "mem"
             - name: "TRANSPORT_TYPE"
               value: "http"
   
   ---
   
   apiVersion: v1
   kind: Service
   metadata:
     name: zipkin
     labels:
       app.kubernetes.io/name: zipkin
       app.kubernetes.io/instance: zipkin-instance
       app.kubernetes.io/managed-by: zipkin-service
   spec:
     ports:
       - port: 9411
         targetPort: 9411
         protocol: TCP
     type: "NodePort"
     selector:
       app.kubernetes.io/name: zipkin
       app.kubernetes.io/instance: zipkin-instance
   

2. Add the Zipkin URI to the Container Storage Modules Observability ConfigMaps. Based on the manifest above, Zipkin will be running on port 9411.

   Note: Zipkin tracing is currently not supported for the collection of PowerFlex metrics.

   Update the ConfigMaps from the Dynamic Configuration. Here is an example updating the karavi-metrics-powerstore-configmap based on the deployment manifest above.

   kubectl edit configmap/karavi-metrics-powerstore-configmap -n [CSM_NAMESPACE]
   

   Update the ZIPKIN_URI and ZIPKIN_PROBABILITY values and save the ConfigMap.

   ZIPKIN_URI: "http://zipkin:9411/api/v2/spans"
   ZIPKIN_SERVICE_NAME: "metrics-powerstore"
   ZIPKIN_PROBABILITY: "1.0"
   

   Once the ConfigMaps are updated, the changes will automatically be applied and tracing can be seen by accessing Zipkin on the exposed port.

Updating Storage System Credentials

If storage system credentials are updated in the CSI Driver, update Container Storage Modules Observability with the new credentials

When Container Storage Modules for Observability uses the Authorization module

All storage system requests by Container Storage Modules Observability will go through the Authorization module. Perform the following steps:

Update the Authorization Module Token

Update Storage Systems

If the list of storage systems managed by a Dell CSI Driver have changed, the following steps can be performed to update Container Storage Modules Observability to reference the updated systems:

When Container Storage Modules for Observability does not use the Authorization module

In this case all storage system requests made by Container Storage Modules Observability will not be routed through the Authorization module. The following must be performed: