Although the Kubernetes Management Operator on managed EKS clusters provides integrated logging capabilities, organizations may have standardized on AWS CloudWatch for their application log aggregation and/or cluster monitoring .
CloudWatch Container Insights provides a way to collect, aggregate, correlate, and summarize metrics and logs from containers running on ECS, EKS, and Linux K8s platforms running on Amazon EC2. This recipe describes how customers can standardize the configuration, deployment, and lifecycle management of CloudWatch Container Insights across their fleet of clusters.
What Will You Do¶
In this exercise,
- You will create a customized "cloudwatch-logging" addon utilizing Fluent Bit
- You will use the addon in a custom cluster blueprint
- You will then apply this cluster blueprint to a managed cluster
This recipe describes the steps to create and use a custom cluster blueprint using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.
- You have already provisioned or imported one or more Kubernetes clusters using the controller.
- You have attached the IAM policy CloudWatchAgentServerPolicy to the IAM role attached to your worker nodes.
Step 1: Download Fluent Bit K8s YAML manifest¶
Download the latest fluent-bit configuration for CloudWatch Container Insights from the official repository.
wget -O cloudwatch-logging-fluent-bit.yaml https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
You will need to add a ConfigMap to the file named "cloudwatch-logging-fluent-bit.yaml" and update the highlighted values to match your cluster configuration. Append the ConfigMap below to the file named "cloudwatch-logging-fluent-bit.yaml"
--- apiVersion: v1 data: cluster.name: my-cluster-name logs.region: us-west-1 http.port: "2020" http.server: "On" read.head: "Off" read.tail: "On" kind: ConfigMap metadata: name: fluent-bit-cluster-info namespace: amazon-cloudwatch
Step 2: Create Addon¶
- Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
- Under Infrastructure, select "Namespaces" and create a new namespace called "amazon-cloudwatch", Set the Pod Security Policy to "rafay-privileged-psp"
- Select "Addons" and "Create" a new Addon called "cloudwatch-logging"
- Ensure that you select "K8s YAML" for the type, "Upload files manually" for the Artifact Sync, and set the namespace as "amazon-cloudwatch"
- Click on "+New Version"
- Enter "v2.10.0" for the Version Name and "UPLOAD" the file created in step 2.
- Select "Save Changes"
- Once the addon is created, publish it, and optionally provide a version so that it can be tracked.
Step 3: Create Blueprint¶
Now, we are ready to assemble a custom cluster blueprint using the newly created CloudWatch addon. We can add additional addons to the blueprint at the same time.
- Under Infrastructure, select "Blueprints"
- Create a new blueprint and give it a name such as "standard-blueprint"
- Set the Version Name
- Set the PSP Policy Type to "cluster-scoped"
- Select the ""cloudwatch-logging" addon
- Disable Managed System Add-On "Log Aggregation"
- Select "Save Changes"
Step 4: Apply Blueprint¶
Now, we are ready to apply this custom blueprint to a cluster.
- Click on Options for the target Cluster in the Web Console
- Select "Update Blueprint" and select the "standard-blueprint" blueprint we created from the list
- Click on "Save and Publish".
This will start the deployment of the addons configured in the "standard-blueprint" blueprint to the targeted cluster. The blueprint sync process can take a few minutes. Once complete, the cluster will display the current cluster blueprint details and whether the sync was successful or not.
Step 5: Verify Blueprint¶
Users can optionally verify whether the required resources for the custom blueprint were created on the cluster. Click on the Kubectl button on the cluster to open a virtual terminal
First, we will verify if the "amazon-cloudwatch" namespace has been created
kubectl get ns amazon-cloudwatch NAME STATUS AGE amazon-cloudwatch Active 6m17s
Next, we will verify that the required pods were created in the "amazon-cloudwatch" namespace. You should see something like the example below.
kubectl get pod -n amazon-cloudwatch NAME READY STATUS RESTARTS AGE fluent-bit-sp6nt 1/1 Running 0 5m2s fluent-bit-t4hhm 1/1 Running 0 5m2s
Step 6: View Logs in CloudWatch¶
You can now verify in CloudWatch that the logs are being collected from the cluster. The following three Log groups will be created and populated with their appropriate logs.
Congratulations! You have successfully created a custom cluster blueprint with the "cloudwatch-logging" addon and applied it to a cluster. You can now use this blueprint on as many clusters as you require.