Overview
Although the Kubernetes Management Operator on managed EKS clusters provides integrated logging capabilities, organizations may have standardized on Amazon OpenSearch Service for their cluster and application log aggregation.
Fluent Bit is an open source lightweight scalable logging and metrics processor and forwarder that will be installed on the cluster to forward logs to your OpenSearch domain for higher-level reporting. This recipe describes how customers can standardize the configuration, deployment, and lifecycle management of Fluent Bit across their fleet of clusters to forward logs to an OpenSearch cluster.
What Will You Do¶
This exercise assumes you have already provisioned or imported an EKS cluster. In this multi-part recipe, you will perform the following:
Part 1
- Create a custom Helm repository pointing to the public "eks-charts" repository
- Create a new custom catalog containing all helm charts and versions from the public "eks-charts" repository
- Create a namespace to deploy fluentbit
- Create the aws-for-fluent-bit addon utilizing the catalog-app
- Create a blueprint containing the newly created addon
Part 2
- Create an IAM Role for Service Accounts (IRSA) so the pod will have the necessary permissions to interact with AWS services
- Create the necessary role in OpenSearch granting permissions to write cluster logs to the appropriate indices
- Apply the newly created cluster blueprint to your EKS Cluster
Part 3
- Verify cluster logs in your OpenSearch instance
Assumptions¶
- You have an AWS account with sufficient privileges to provision an EKS cluster if one is not available using the Controller and to create IAM policies
- You have provisioned your cluster with an OpenID Connect (OIDC) provider or have the appropriate permissions to create a provider and associate to your cluster
- You have access to an Org with a role to create and deploy addons utilizing blueprints
- You have access to an OpenSearch Domain and the EKS Cluster has network access to Domain endpoint