Install and Configure Grafana on Kubernetes

Posted on by

We are going to deploy Grafana to visualise Prometheus monitoring data.

Pre-requisites

We are using our Kubernetes homelab to deploy Grafana.

A working NFS server is required to create persistent volumes. Note that NFS server configuration is not covered in this article, but the way we set it up can be found here.

Our NFS server IP address is 10.11.1.20, and we have the following export configured for Grafana:

/mnt/storage-k8s/nfs/grafana

The owner:group of the NFS folder is set to 472:472, because of Grafana deployment runAsUser: 472.

Download Files from GitHub

Configuration files used in this article are hosted on GitHub. Clone the following repository:

$ git clone https://github.com/lisenet/kubernetes-homelab.git
$ cd ./kubernetes-homelab/

TLDR; Install and Configure Grafana: All in One Go

Create a monitoring namespace:

$ kubectl create ns monitoring

Create everything with a single command:

$ kubectl apply -f ./kubernetes/grafana/

Note to self: this can be a Helm chart.

Install and Configure Grafana: Step by Step

Step by step instructions. Note that this homelab project is under development, therefore please refer to GitHub for any source code changes.

Create a Namespace

Create a monitoring namespace:

$ kubectl create ns monitoring

Create Config Maps

Configure Grafana to use Prometheus as a data source.

$ kubectl apply -f ./kubernetes/grafana/grafana-config-map-datasource.yml

This is what the code looks like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "Prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus-service.monitoring.svc:9090",
                "version": 1
            }
        ]
    }

We can also use a config map to define our initial Grafana configuration like TLS certificates and logging options. Create a config map that we will later use to populate a custom grafana.ini file:

$ kubectl apply -f ./kubernetes/grafana/grafana-config-map-ini.yml

This is what the code looks like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-ini
  namespace: monitoring
data:
  grafana.ini: |
    [server]
      protocol = https
      http_port = 3000
      cert_file = /etc/grafana/certs/tls.crt
      cert_key  = /etc/grafana/certs/tls.key
    [analytics]
      reporting_enabled = false
      check_for_updates = true
    [log]
      mode = console
      level = info
    [paths]
      data         = /var/lib/grafana/data
      logs         = /var/log/grafana
      plugins      = /var/lib/grafana/plugins
      provisioning = /etc/grafana/provisioning

Configure Grafana dashboard providers.

$ kubectl apply -f ./kubernetes/grafana/grafana-config-map-dashboard-providers.yml

This is what the code looks like:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-providers
  namespace: monitoring
  labels:
    app: grafana
data:
  dashboardproviders.yaml: |
    apiVersion: 1
    providers:
    - name: k8s-cluster-summary
      folder: homelab
      options:
        path: /var/lib/grafana/dashboards/k8s-cluster-summary
      orgId: 1
      type: file
      disableDeletion: false
    - name: node-exporter-full
      folder: homelab
      options:
        path: /var/lib/grafana/dashboards/node-exporter-full
      orgId: 1
      type: file
      disableDeletion: false

Configure Grafana dashboards for K8s cluster and node exporter:

$ kubectl apply -f ./kubernetes/grafana/grafana-config-map-dashboards-k8s-cluster.yml
$ kubectl apply -f ./kubernetes/grafana/grafana-config-map-dashboards-node-exporter.yml

Create a TLS Secret

Create a secret to store our TLS certificates.

$ kubectl apply -f ./kubernetes/grafana/grafana-secret-tls.yml

This is what the code looks like:

---
apiVersion: v1
kind: Secret
metadata:
  name: grafana-tls-cert
  namespace: monitoring
type: kubernetes.io/tls
data:
  tls.crt: |
    [... your TLS certificate ...
  tls.key: |
    [... your TLS private key ...]

Create a Persistent Volume

We want to keep grafana configuration data and store it on a persistent volume.

$ kubectl apply -f ./kubernetes/grafana/grafana-pv.yml

This is what the code looks like:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv-grafana
  namespace: monitoring
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /mnt/storage-k8s/nfs/grafana
    server: 10.11.1.20

Create a Persistent Volume Claim

Allow grafana to request persistent storage.

$ kubectl apply -f ./kubernetes/grafana/grafana-pvc.yml

This is what the code looks like:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc-grafana
  namespace: monitoring
spec:
  storageClassName: nfs
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Create a Deployment Configuration

$ kubectl apply -f ./kubernetes/grafana/grafana-deployment.yml

This is what the code looks like:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        runAsGroup: 472
        runAsNonRoot: true
        runAsUser: 472
      containers:
      - name: grafana
        image: grafana/grafana:7.3.7
        imagePullPolicy: IfNotPresent
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "2Gi"
            cpu: "1000m"
          requests: 
            memory: "1Gi"
            cpu: "500m"
        volumeMounts:
          - name: grafana-config
            mountPath: /etc/grafana/grafana.ini
            subPath: grafana.ini

          - name: grafana-storage
            mountPath: /var/lib/grafana

          - name: grafana-certs
            mountPath: /etc/grafana/certs/
            readOnly: true

          - name: grafana-datasources
            mountPath: /etc/grafana/provisioning/datasources/prometheus.yaml
            subPath: prometheus.yaml

          - name: grafana-dashboard-providers
            mountPath: /etc/grafana/provisioning/dashboards/dashboardproviders.yaml
            subPath: dashboardproviders.yaml

          - name: dashboards-k8s-cluster-summary
            mountPath: /var/lib/grafana/dashboards/k8s-cluster-summary

          - name: dashboards-node-exporter-full
            mountPath: /var/lib/grafana/dashboards/node-exporter-full

      # See https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
      #initContainers:
      #  - name: fix-nfs-permissions
      #    image: busybox
      #    command: ["sh", "-c", "chown -R 472:472 /var/lib/grafana"]
      #    securityContext:
      #      runAsUser: 0
      #      runAsNonRoot: false
      #    volumeMounts:
      #      - name: grafana-storage
      #        mountPath: /var/lib/grafana/

      restartPolicy: Always
      terminationGracePeriodSeconds: 60
      volumes:
        - name: grafana-config
          configMap:
            name: grafana-ini

        - name: grafana-storage
          persistentVolumeClaim:
            claimName: nfs-pvc-grafana

        - name: grafana-certs
          secret:
            secretName: grafana-tls-cert
            items:
            - key: tls.crt
              path: tls.crt
            - key: tls.key
              path: tls.key

        - name: grafana-datasources
          configMap:
            name: grafana-datasources

        - name: grafana-dashboard-providers
          configMap:
            name: grafana-dashboard-providers

        - name: dashboards-k8s-cluster-summary
          configMap:
            name: grafana-dashboards-k8s-cluster-summary

        - name: dashboards-node-exporter-full
          configMap:
            name: grafana-dashboards-node-exporter-full

Create a Service

$ kubectl apply -f ./kubernetes/grafana/grafana-service.yml

This is what the code looks like:

---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port:   '3000'
  labels:
    app: grafana
spec:
  selector: 
    app: grafana
  type: NodePort  
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32000

Check Monitoring Namespace

$ kubectl -n monitoring get all -l app=grafana
NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-6fff94cd6b-d8gmn   1/1     Running   0          6m47s

NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/grafana   NodePort   10.110.20.247   none          3000:32000/TCP   6d4h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           6d4h

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-6fff94cd6b   1         1         1       6d4h

Create a Dashboard to Monitor Kubernetes Deployments

We can access grafana dashboard by using its service node port 32000. Default credentials are admin/admin.

We should have a couple of dashboards installed already, Kubernetes Cluster Summary and Node Exporter Full. Optionally, install a dashboard to monitor Kubernetes deployments: https://grafana.com/grafana/dashboards/8588

The end result should look something like this:

What’s Next?

We will be configuring monitoring and adding Grafana dashboards for our homelab services: Etcd, Bind DNS, HAProxy, Linux servers etc.

Leave a Reply

Your email address will not be published. Required fields are marked *