Skip to content

Datadog Agent

Although the Kubernetes Management Operator on managed clusters provides integrated monitoring and visibility capabilities, organizations may have standardized on Datadog for monitoring their Kubernetes clusters.

This recipe describes how customers can standardize the configuration, deployment and lifecycle management of Datadog across their fleet of clusters.The Datadog Agent needs to be deployed on every Kubernetes cluster so that it can aggregate cluster and applications metrics, traces, and logs. These are then sent to Datadog's service for centralized monitoring and visibility.

What Will You Do

In this exercise,

  • You will create a customized "datadog-agent" addon using Datadog's "official Helm chart"
  • You will use the addon in a custom cluster blueprint
  • You will then apply this cluster blueprint to a managed cluster

Important

This recipe describes the steps to create and use a custom cluster blueprint using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.

Assumptions

  • You have already provisioned or imported one or more Kubernetes clusters using the controller.
  • You have Helm CLI installed locally to download required Helm charts

Step 1: Download Helm Chart

We will be using the Datadog Agent Helm chart from the official repository. In this example, we will be using v2.4.10 of the datadog agent.

  • Add the official Datadog helm repo to your Helm CLI if you haven't already added it.
helm repo add datadog https://helm.datadoghq.com
  • Download the Datadog Helm chart. In this example, we will be using v2.4.10 of the chart (filename: datadog-2.4.10.tgz).
helm fetch datadog/datadog

Step 2: Customize Values

The Datadog Helm chart comes with a very detailed values.yaml file with support for a large number of scenarios. We will be customizing the defaults with our own override "values.yaml"

Copy the details below into a file named "datadog-custom-values.yaml".

  • Replace the "apiKey" value with your Datadog API key.

Note

Reference on how to get Datadog API key can be found here

## Custom values for Datadog Agent
datadog:
  ## Change to your Datadog API key below. Instruction on how to obtain the API key can be found here https://app.datadoghq.com/account/settings#agent/kubernetes
  #
  apiKey: 123456789a123456789b123456789cde

Step 3: Create Addon

  • Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
  • Under Infrastructure, select "Namespaces" and create a new namespace called "datadog-agent"
  • Select "Addons" and "Create" a new Addon called "datadog-agent"
  • Ensure that you select "Helm" for type and select the namespace as "datadog-agent"

Create Addon

  • Upload the Helm chart "datadog-2.4.10.tgz" from the previous step, the "datadog-custom-values.yaml" file and Save

Create Addon

  • Once the addon is created, ensure you publish it and optionally provide a version so that it can be tracked.

Create Addon

Step 4: Create Blueprint

Now, we are ready to assemble a custom cluster blueprint using the newly created Datadog addon. You can add other addons to the same custom blueprint.

  • Under Infrastructure, select "Blueprints"
  • Create a new blueprint and give it a name such as "standard-blueprint"
  • Ensure that you have managed Ingress enabled
  • Select the datadog-agent and other addons as required

Create Blueprint

  • Once the blueprint is created, ensure you publish it and optionally provide a version so that it can be tracked.

Create Blueprint

Step 5: Apply Blueprint

Now, we are ready to apply this custom blueprint to a cluster.

  • Click on Options for the target Cluster in the Web Console
  • Select "Update Blueprint" and select the "standard-blueprint" blueprint we created from the list

Update Blueprint

  • Click on "Save and Publish".

This will start the deployment of the addons configured in the "standard-blueprint" blueprint to the targeted cluster. The blueprint sync process can take a few minutes. Once complete, the cluster will display the current cluster blueprint details and whether the sync was successful or not.

Step 6: Verify Deployment

Users can optionally verify whether the correct resources have been created on the cluster. Click on the Kubectl button on the cluster to open a virtual terminal

First, we will verify if the "datadog-agent" namespace has been created

kubectl get ns datadog-agent

NAME            STATUS   AGE
datadog-agent   Active   27m

Next, we will verify the pods in the "datadog-agent" namespace. You should see something like the example below.

kubectl get pod -n datadog-agent

NAME                                                      READY   STATUS    RESTARTS   AGE
datadog-agent-v2410-6nwf2                                 2/2     Running   0          27m
datadog-agent-v2410-kube-state-metrics-7b659f86dc-hrxwx   1/1     Running   0          27m
datadog-agent-v2410-vm574                                 2/2     Running   0          27m

Next, we will verify the datadog-agent logs and ensure the agent posted payload successfully to Datadog website.

kubectl logs daemonset.apps/datadog-agent-v2410 -c agent -n datadog-agent
2020-08-14 21:39:04 UTC | CORE | INFO | (pkg/forwarder/transaction.go:272 in internalProcess)
Successfully posted payload to "https://7-21-1-app.agent.datadoghq.com/intake/?api_key=*************************60f1e"
the agent will only log transaction success every 500 transactions

Step 7: Verify Metrics

In order to view the collected metrics, login to your Datadog account and check the Kubernetes clusters' metrics and dashboards. Shown below is an illustrative example of what you can see from the Kubernetes Overview dashboard built from the metrics collected by the deployed datadog-agent in your clusters.

Datadog Dashboards

Recap

Congratulations! You have successfully created a custom cluster blueprint with the "datadog-agent" addon and applied to a cluster. You can now use this blueprint on as many clusters as you require.