Overview

Although the Kubernetes Management Operator on managed EKS clusters provides integrated logging capabilities, organizations may have standardized on Amazon OpenSearch Service for their cluster and application log aggregation.

Fluent Bit is an open source lightweight scalable logging and metrics processor and forwarder that will be installed on the cluster to forward logs to your OpenSearch domain for higher-level reporting. This recipe describes how customers can standardize the configuration, deployment, and lifecycle management of Fluent Bit across their fleet of clusters to forward logs to an OpenSearch cluster.

What Will You Do¶

This exercise assumes you have already provisioned or imported an EKS cluster. In this multi-part recipe, you will perform the following:

Part 1

Create a custom Helm repository pointing to the public "eks-charts" repository
Create a new custom catalog containing all helm charts and versions from the public "eks-charts" repository
Create a namespace to deploy fluentbit
Create the aws-for-fluent-bit addon utilizing the catalog-app
Create a blueprint containing the newly created addon

Part 2

Create an IAM Role for Service Accounts (IRSA) so the pod will have the necessary permissions to interact with AWS services
Create the necessary role in OpenSearch granting permissions to write cluster logs to the appropriate indices
Apply the newly created cluster blueprint to your EKS Cluster

Part 3

Verify cluster logs in your OpenSearch instance

Assumptions¶

You have an AWS account with sufficient privileges to provision an EKS cluster if one is not available using the Controller and to create IAM policies
You have provisioned your cluster with an OpenID Connect (OIDC) provider or have the appropriate permissions to create a provider and associate to your cluster
You have access to an Org with a role to create and deploy addons utilizing blueprints
You have access to an OpenSearch Domain and the EKS Cluster has network access to Domain endpoint