Skip to content

Elasticsearch

Follow the steps documented below to aggregate your application logs to a centralized Elasticsearch cluster.

Important

The Elasticsearch cluster has been installed and is accessible from the managed clusters.


Create Endpoint

  • Login into the Web Console as a Project Admin.
  • Click on Integrations > Aggregation Endpoints
  • Click on "New Endpoint"
  • Provide a Name, select "Elastic Search" from the drop-down for Type
  • Select the transport scheme for this endpoint as HTTP or HTTPs
  • Enter the Host for Elasticsearch cluster
  • Enter the Port to reach Elasticsearch cluster
  • Provide the Index Name
  • Provide the User and Password to access the Elasticsearch cluster

External Endpoint

Here is an example for an "External" log aggregation endpoint for an Elasticsearch cluster.

Create External ES Endpoint

Internal Endpoint

When Elasticsearch is deployed in the same Kubernetes cluster, the logs aggregation endpoint configuration can use the servicename instead.

Here is an example for an "Internal" log aggretation endpoint where the Host is set to "elasticsearch_service_name"."elasticseach_namespace" and Port to "elasticsearch_serviceport"

Create Internal ES Endpoints


Use Endpoint - Workload Wizard

The Workload Wizard provides a streamlined experience for users to send container logs to the log endpoints created above.

  • Select the endpoint name from the dropdown list of "Logs endpoint" under Workloads > Containers > Logs Aggregation
  • Publish the workload and you should start seeing logs appear in your Elasticsearch cluster

Select ES Endpoint


Annotations for Yaml or Helm

Workloads based on Helm or k8s yaml can also the Log Aggregation endpoints created and managed by the controller.

We provide simple annotations that users can embed into their k8s YAML or Helm charts.

  annotations:
    rafay.dev/logging: "<log_endpoint_name>"

Here is an example yaml for a deployment with the Annotations for log integration to ElasticSearch.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-es-logging
  annotations:
    rafay.dev/logging: elasticsearch-cluster-internal
spec:
  selector:
    matchLabels:
      app: nginx-es-logging
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-es-logging
    spec:
      containers:
      - name: nginx-es-logging
        image: nginx:latest
        ports:
        - containerPort: 80

View Logs In Kibana

Container logs aggregated in your Elasticsearch clusters can be viewed using Kibana.

  • Login to Kibana
  • Go to Management > Index Patterns
  • Click "Create index pattern"
  • Search for the index name
  • Click Next step

Kibana Settings

  • Select Time Filter field name as "metadata.timestamp"
  • Click "Create index pattern" button to create this index pattern

Kibana Settings

  • Go to Discover menu of Kibana, you should start seeing the container logs where you can start creating Dashboard or Visualize to monitor the logs

Kibana Settings