Although the Kubernetes Management Operator on managed clusters provides integrated monitoring and visibility capabilities, organizations may have standardized on Datadog for monitoring their Kubernetes clusters.
This recipe describes how customers can standardize the configuration, deployment and lifecycle management of Datadog across their fleet of clusters.The Datadog Agent needs to be deployed on every Kubernetes cluster so that it can aggregate cluster and applications metrics, traces, and logs. These are then sent to Datadog's service for centralized monitoring and visibility.
What Will You Do¶
In this exercise,
- You will create a customized "datadog-agent" addon using Datadog's "official Helm chart"
- You will use the addon in a custom cluster blueprint
- You will then apply this cluster blueprint to a managed cluster
This recipe describes the steps to create and use a custom cluster blueprint using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.
- You have already provisioned or imported one or more Kubernetes clusters using the controller.
- You have Helm CLI installed locally to download required Helm charts
Step 1: Download Helm Chart¶
We will be using the Datadog Agent Helm chart from the official repository. In this example, we will be using v2.4.10 of the datadog agent.
- Add the official Datadog helm repo to your Helm CLI if you haven't already added it.
helm repo add datadog https://helm.datadoghq.com
- Download the Datadog Helm chart. In this example, we will be using v2.4.10 of the chart (filename: datadog-2.4.10.tgz).
helm fetch datadog/datadog
Step 2: Customize Values¶
The Datadog Helm chart comes with a very detailed values.yaml file with support for a large number of scenarios. We will be customizing the defaults with our own override "values.yaml"
Copy the details below into a file named "datadog-custom-values.yaml".
- Replace the "apiKey" value with your Datadog API key.
Reference on how to get Datadog API key can be found here
## Custom values for Datadog Agent datadog: ## Change to your Datadog API key below. Instruction on how to obtain the API key can be found here https://app.datadoghq.com/account/settings#agent/kubernetes # apiKey: 123456789a123456789b123456789cde
Step 3: Create Addon¶
- Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
- Under Infrastructure, select "Namespaces" and create a new namespace called "datadog-agent"
- Select "Addons" and "Create" a new Addon called "datadog-agent"
- Ensure that you select "Helm" for type and select the namespace as "datadog-agent"
- Upload the Helm chart "datadog-2.4.10.tgz" from the previous step, the "datadog-custom-values.yaml" file and Save
- Once the addon is created, ensure you publish it and optionally provide a version so that it can be tracked.
Step 4: Create Blueprint¶
Now, we are ready to assemble a custom cluster blueprint using the newly created Datadog addon. You can add other addons to the same custom blueprint.
- Under Infrastructure, select "Blueprints"
- Create a new blueprint and give it a name such as "standard-blueprint"
- Ensure that you have managed Ingress enabled
- Select the datadog-agent and other addons as required
- Once the blueprint is created, ensure you publish it and optionally provide a version so that it can be tracked.
Step 5: Apply Blueprint¶
Now, we are ready to apply this custom blueprint to a cluster.
- Click on Options for the target Cluster in the Web Console
- Select "Update Blueprint" and select the "standard-blueprint" blueprint we created from the list
- Click on "Save and Publish".
This will start the deployment of the addons configured in the "standard-blueprint" blueprint to the targeted cluster. The blueprint sync process can take a few minutes. Once complete, the cluster will display the current cluster blueprint details and whether the sync was successful or not.
Step 6: Verify Deployment¶
Users can optionally verify whether the correct resources have been created on the cluster. Click on the Kubectl button on the cluster to open a virtual terminal
First, we will verify if the "datadog-agent" namespace has been created
kubectl get ns datadog-agent NAME STATUS AGE datadog-agent Active 27m
Next, we will verify the pods in the "datadog-agent" namespace. You should see something like the example below.
kubectl get pod -n datadog-agent NAME READY STATUS RESTARTS AGE datadog-agent-v2410-6nwf2 2/2 Running 0 27m datadog-agent-v2410-kube-state-metrics-7b659f86dc-hrxwx 1/1 Running 0 27m datadog-agent-v2410-vm574 2/2 Running 0 27m
Next, we will verify the datadog-agent logs and ensure the agent posted payload successfully to Datadog website.
kubectl logs daemonset.apps/datadog-agent-v2410 -c agent -n datadog-agent 2020-08-14 21:39:04 UTC | CORE | INFO | (pkg/forwarder/transaction.go:272 in internalProcess) Successfully posted payload to "https://7-21-1-app.agent.datadoghq.com/intake/?api_key=*************************60f1e" the agent will only log transaction success every 500 transactions
Step 7: Verify Metrics¶
In order to view the collected metrics, login to your Datadog account and check the Kubernetes clusters' metrics and dashboards. Shown below is an illustrative example of what you can see from the Kubernetes Overview dashboard built from the metrics collected by the deployed datadog-agent in your clusters.
Congratulations! You have successfully created a custom cluster blueprint with the "datadog-agent" addon and applied to a cluster. You can now use this blueprint on as many clusters as you require.