Elasticsearch

Follow the steps documented below to aggregate your application logs to a centralized Elasticsearch cluster.

Important

The Elasticsearch cluster has been installed and is accessible from the managed clusters.

Create Endpoint¶

Login into the Web Console as a Project Admin.
Click on Integrations > Aggregation Endpoints
Click on "New Endpoint"
Provide a Name, select "Elastic Search" from the drop-down for Type
Select the transport scheme for this endpoint as HTTP or HTTPs
Enter the Host for Elasticsearch cluster
Enter the Port to reach Elasticsearch cluster
Provide the Index Name
Provide the User and Password to access the Elasticsearch cluster

External Endpoint¶

Here is an example for an "External" log aggregation endpoint for an Elasticsearch cluster.

Create External ES Endpoint

Internal Endpoint¶

When Elasticsearch is deployed in the same Kubernetes cluster, the logs aggregation endpoint configuration can use the servicename instead.

Here is an example for an "Internal" log aggretation endpoint where the Host is set to "elasticsearch_service_name"."elasticseach_namespace" and Port to "elasticsearch_serviceport"

Create Internal ES Endpoints

Use Endpoint - Workload Wizard¶

The Workload Wizard provides a streamlined experience for users to send container logs to the log endpoints created above.

Select the endpoint name from the dropdown list of "Logs endpoint" under Workloads > Containers > Logs Aggregation
Publish the workload and you should start seeing logs appear in your Elasticsearch cluster

Select ES Endpoint

Annotations for Yaml or Helm¶

Workloads based on Helm or k8s yaml can also the Log Aggregation endpoints created and managed by the controller.

We provide simple annotations that users can embed into their k8s YAML or Helm charts.

  annotations:
    rafay.dev/logging: "<log_endpoint_name>"

Here is an example yaml for a deployment with the Annotations for log integration to ElasticSearch.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-es-logging
  annotations:
    rafay.dev/logging: elasticsearch-cluster-internal
spec:
  selector:
    matchLabels:
      app: nginx-es-logging
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-es-logging
    spec:
      containers:
      - name: nginx-es-logging
        image: nginx:latest
        ports:
        - containerPort: 80

View Logs In Kibana¶

Container logs aggregated in your Elasticsearch clusters can be viewed using Kibana.

Login to Kibana
Go to Management > Index Patterns
Click "Create index pattern"
Search for the index name
Click Next step

Kibana Settings

Select Time Filter field name as "metadata.timestamp"
Click "Create index pattern" button to create this index pattern

Kibana Settings

Go to Discover menu of Kibana, you should start seeing the container logs where you can start creating Dashboard or Visualize to monitor the logs

Kibana Settings