Follow the steps documented below to aggregate your application logs to a centralized Elasticsearch cluster.
The Elasticsearch cluster has been installed and is accessible from the managed clusters.
- Login into the Web Console as a Project Admin.
- Click on Integrations > Aggregation Endpoints
- Click on "New Endpoint"
- Provide a Name, select "Elastic Search" from the drop-down for Type
- Select the transport scheme for this endpoint as HTTP or HTTPs
- Enter the Host for Elasticsearch cluster
- Enter the Port to reach Elasticsearch cluster
- Provide the Index Name
- Provide the User and Password to access the Elasticsearch cluster
Here is an example for an "External" log aggregation endpoint for an Elasticsearch cluster.
When Elasticsearch is deployed in the same Kubernetes cluster, the logs aggregation endpoint configuration can use the servicename instead.
Here is an example for an "Internal" log aggretation endpoint where the Host is set to "elasticsearch_service_name"."elasticseach_namespace" and Port to "elasticsearch_serviceport"
Use Endpoint - Workload Wizard¶
The Workload Wizard provides a streamlined experience for users to send container logs to the log endpoints created above.
- Select the endpoint name from the dropdown list of "Logs endpoint" under Workloads > Containers > Logs Aggregation
- Publish the workload and you should start seeing logs appear in your Elasticsearch cluster
Annotations for Yaml or Helm¶
Workloads based on Helm or k8s yaml can also the Log Aggregation endpoints created and managed by the controller.
We provide simple annotations that users can embed into their k8s YAML or Helm charts.
annotations: rafay.dev/logging: "<log_endpoint_name>"
Here is an example yaml for a deployment with the Annotations for log integration to ElasticSearch.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-es-logging annotations: rafay.dev/logging: elasticsearch-cluster-internal spec: selector: matchLabels: app: nginx-es-logging replicas: 2 template: metadata: labels: app: nginx-es-logging spec: containers: - name: nginx-es-logging image: nginx:latest ports: - containerPort: 80
View Logs In Kibana¶
Container logs aggregated in your Elasticsearch clusters can be viewed using Kibana.
- Login to Kibana
- Go to Management > Index Patterns
- Click "Create index pattern"
- Search for the index name
- Click Next step
- Select Time Filter field name as "metadata.timestamp"
- Click "Create index pattern" button to create this index pattern
- Go to Discover menu of Kibana, you should start seeing the container logs where you can start creating Dashboard or Visualize to monitor the logs