Skip to content

CloudWatch

Although the Kubernetes Management Operator on managed EKS clusters provides integrated logging capabilities, organizations may have standardized on AWS CloudWatch for their application log aggregation and/or cluster monitoring .

CloudWatch Container Insights provides a way to collect, aggregate, correlate, and summarize metrics and logs from containers running on ECS, EKS, and Linux K8s platforms running on Amazon EC2. This recipe describes how customers can standardize the configuration, deployment, and lifecycle management of CloudWatch Container Insights across their fleet of clusters.

What Will You Do

In this exercise,

  • You will create a customized "cloudwatch-logging" addon utilizing Fluent Bit
  • You will use the addon in a custom cluster blueprint
  • You will then apply this cluster blueprint to a managed cluster

Important

This recipe describes the steps to create and use a custom cluster blueprint using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.

Assumptions

  • You have already provisioned or imported one or more Kubernetes clusters using the controller.
  • You have attached the IAM policy CloudWatchAgentServerPolicy to the IAM role attached to your worker nodes.

Step 1: Download Fluent Bit K8s YAML manifest

Download the latest fluent-bit configuration for CloudWatch Container Insights from the official repository.

wget -O cloudwatch-logging-fluent-bit.yaml https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml

You will need to add a ConfigMap to the file named "cloudwatch-logging-fluent-bit.yaml" and update the highlighted values to match your cluster configuration. Append the ConfigMap below to the file named "cloudwatch-logging-fluent-bit.yaml"

---
apiVersion: v1
data:
  cluster.name: my-cluster-name
  logs.region: us-west-1
  http.port: "2020"
  http.server: "On"
  read.head: "Off"
  read.tail: "On"
kind: ConfigMap
metadata:
  name: fluent-bit-cluster-info
  namespace: amazon-cloudwatch

Step 2: Create Addon

  • Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
  • Under Infrastructure, select "Namespaces" and create a new namespace called "amazon-cloudwatch", Set the Pod Security Policy to "rafay-privileged-psp"
  • Select "Addons" and "Create" a new Addon called "cloudwatch-logging"
  • Ensure that you select "K8s YAML" for the type, "Upload files manually" for the Artifact Sync, and set the namespace as "amazon-cloudwatch"
  • Click on "+New Version"
  • Enter "v2.10.0" for the Version Name and "UPLOAD" the file created in step 2.
  • Select "Save Changes"
  • Once the addon is created, publish it, and optionally provide a version so that it can be tracked.

Create Addon

Step 3: Create Blueprint

Now, we are ready to assemble a custom cluster blueprint using the newly created CloudWatch addon. We can add additional addons to the blueprint at the same time.

  • Under Infrastructure, select "Blueprints"
  • Create a new blueprint and give it a name such as "standard-blueprint"
  • Set the Version Name
  • Set the PSP Policy Type to "cluster-scoped"
  • Select the ""cloudwatch-logging" addon
  • Disable Managed System Add-On "Log Aggregation"
  • Select "Save Changes"

Create Blueprint

Step 4: Apply Blueprint

Now, we are ready to apply this custom blueprint to a cluster.

  • Click on Options for the target Cluster in the Web Console
  • Select "Update Blueprint" and select the "standard-blueprint" blueprint we created from the list

Create Blueprint

  • Click on "Save and Publish".

This will start the deployment of the addons configured in the "standard-blueprint" blueprint to the targeted cluster. The blueprint sync process can take a few minutes. Once complete, the cluster will display the current cluster blueprint details and whether the sync was successful or not.

Step 5: Verify Blueprint

Users can optionally verify whether the required resources for the custom blueprint were created on the cluster. Click on the Kubectl button on the cluster to open a virtual terminal

First, we will verify if the "amazon-cloudwatch" namespace has been created

kubectl get ns amazon-cloudwatch

NAME                STATUS   AGE
amazon-cloudwatch   Active   6m17s

Next, we will verify that the required pods were created in the "amazon-cloudwatch" namespace. You should see something like the example below.

kubectl get pod -n amazon-cloudwatch

NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-sp6nt   1/1     Running   0          5m2s
fluent-bit-t4hhm   1/1     Running   0          5m2s

Step 6: View Logs in CloudWatch

You can now verify in CloudWatch that the logs are being collected from the cluster. The following three Log groups will be created and populated with their appropriate logs.

  • /aws/containerinsights/my-cluster-name/applicaiton
  • /aws/containerinsights/my-cluster-name/dataplane
  • /aws/containerinsights/my-cluster-name/host

View CloudWatch

Recap

Congratulations! You have successfully created a custom cluster blueprint with the "cloudwatch-logging" addon and applied it to a cluster. You can now use this blueprint on as many clusters as you require.