Prometheus and Grafana are tools that are commonly used for monitoring and alerting.

Prometheus is an open-source system monitoring and alerting toolkit. It can collect metrics from various sources, including containers, servers, and applications, and store them in a time-series database. Prometheus provides a flexible query language, called PromQL, that allows you to retrieve and analyze data. It also includes a web interface and an API for interacting with the data.

Grafana is an open-source data visualization and monitoring platform. It allows you to create visualizations of your data, including graphs, gauges, and maps, and to set up alerts based on certain thresholds. Grafana can connect to a variety of data sources, including Prometheus, and provides a way to build dashboards to monitor your systems and applications.

Together, Prometheus and Grafana can be used to monitor the performance and availability of your infrastructure and applications and to alert you when there are problems. They are widely used in production environments to ensure that systems are running smoothly and to identify and resolve issues quickly.

In this article, I have shown how you can easily set up Prometheus and Grafana in AWS EKS.

Steps to Follow

• Setup eksctl using the instructions given on the official website eksctl.io
• Create a cluster by running the following command

eksctl create cluster 

Create IAM OIDC Provider

oidc_id=$(aws eks describe-cluster --name xxxx --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

eksctl utils associate-iam-oidc-provider --cluster xxxx --approve

Set Up EBS CSI addon for EKS

Add IAM Role using eksctl

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

Then add EBS CSI to eks by running the following command

eksctl create addon --name aws-ebs-csi-driver --cluster my-cluster --service-account-role-arn arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole --force

Install helm

Helm is a package manager for Kubernetes, an open-source container orchestration platform. Helm helps you manage Kubernetes applications by making it easy to install, update, and delete them.

To install helm on EKS, run the following commands

sudo yum install openssl -y
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

Once you install Helm on EKS then add Prometheus and Grafana repo by running the command

# add prometheus Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# add grafana Helm repo
helm repo add grafana https://grafana.github.io/helm-charts

Install Prometheus and grafana using helm

Install Helm by running the below command

kubectl create namespace prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm install prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"

To check if the installation went well or not, then please run this command

kubectl get all -n prometheus

If Prometheus installation went well then run this command to port forward and view this locally

kubectl port-forward -n prometheus deploy/prometheus-server 8080:9090

To install Grafana you need to add this yaml file first

mkdir ${HOME}/environment/grafana

cat << EoF > ${HOME}/environment/grafana/grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true
EoF

Then run this command to install Grafana

kubectl create namespace grafana

helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='EKS!sAWSome' \
    --values ${HOME}/environment/grafana/grafana.yaml \
    --set service.type=LoadBalancer

This command will create the Grafana service with an external load balancer to get the public view.

To get the external load balancer URL, run the following command

export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "http://$ELB"

I have created a detailed video on this topic which you can view here.