New Relic
Although the Platform provides basic integrated monitoring and visibility capabilities, organizations may prefer to use a best of breed, internal standard such as New Relic for monitoring their Kubernetes clusters.
This recipe describes how customers can standardize the configuration, deployment and lifecycle management of New Relic's Kubernetes Addons across their fleet of clusters. The New Relic Kubernetes Addon needs to be deployed on every Kubernetes cluster so that it can aggregate cluster and applications metrics, events and logs. These are then sent to New Relic's service for centralized monitoring and visibility.
What Will You Do¶
In this exercise,
- You will create a customized "newrelic" addon using New Relic's "official Helm chart"
- You will use the addon in a custom cluster blueprint
- You will then apply this cluster blueprint to a managed cluster
Important
This recipe describes the steps to create and use a custom cluster blueprint using the Web Console. The entire workflow can also be fully automated and embedded into an automation pipeline.
Assumptions¶
- You have already provisioned or imported one or more Kubernetes clusters using the controller.
- You have Helm CLI installed locally to download required Helm charts
Step 1: Download Helm Chart¶
We will be using the New Relic Kubernetes Integration Helm chart from the official repository. In this example, we will be using bundle version v1.7.1 (nri-bundle-1.7.1.tgz) of the New Relic Helm Chart.
- Add the official New Relic helm repo to your Helm CLI if you haven't already added it.
helm repo add newrelic https://helm-charts.newrelic.com
- Download the New Relic Bundle Helm chart. In this example, we will be using v1.7.1 of the chart (filename: nri-bundle-1.7.1.tgz).
helm fetch newrelic/nri-bundle
Step 2: Customize Values¶
The New Relic Bundle Helm chart comes with a default values.yaml file. We will be customizing the defaults with our own override "values.yaml"
Copy the details below into a file named "nri-bundle-custom-values.yaml".
- Replace the "licenseKey" value with your New Relic License Key.
Note
Reference on how to get New Relic License Key can be found here
## Custom values for New Relic Kubernetes Integration Bundle
## Set Global Config
global:
## Change to your New Relic License Key below.
#
licenseKey: xxxxxxxxxxxxxxxxxxxxxxxx
## Or get New Relic license key from exising secret info below:
# customSecretName:
# customSecretLicenseKey:
#
## Change the cluster name:
#
cluster: demo-eks-cluster
## Set enabled to true to deploy New Relic Infrastructure agent
#
infrastructure:
enabled: true
## Set enabled to true to deploy New Relic Kubernetes Logging
#
logging:
enabled: true
## Set enabled to true to deploy New Relic Kubernetes Infrastructure metadata injection webhook
#
webhook:
enabled: true
## Set enabled to true to deploy New Relic's Prometheus OpenMetrics Integration
#
prometheus:
enabled: true
## Set enabled to true to deploy kube-state-metrics agent
#
ksm:
enabled: true
## Set enabled to true to deploy New Relic Events Routers
#
kubeEvents:
enabled: true
Step 3: Create Addon¶
- Login into the Web Console and navigate to your Project as an Org Admin or Infrastructure Admin
- Under Infrastructure, select "Namespaces" and create a new namespace called "newrelic"
- Select "Addons" and "Create" a new Addon called "newrelic"
- Ensure that you select "Helm" for type and select the namespace as "newrelic"
- Upload the Helm chart "nri-bundle-1.7.1.tgz" from the previous step, the "nri-bundle-custom-values.yaml" file and Save
- Once the addon is created, ensure you publish it and optionally provide a version so that it can be tracked.
Step 4: Create Blueprint¶
Now, we are ready to assemble a custom cluster blueprint using the newly created New Relic addon. You can add other addons to the same custom blueprint.
- Under Infrastructure, select "Blueprints"
- Create a new blueprint and give it a name such as "standard-blueprint"
- Ensure that you have managed Ingress enabled
- Select the newrelic and other addons as required
- Once the blueprint is created, ensure you publish it and optionally provide a version so that it can be version controlled.
Step 5: Apply Blueprint¶
Now, we are ready to apply this custom blueprint to a cluster.
- Click on Options for the target Cluster in the Web Console
- Select "Update Blueprint" and select the "standard-blueprint" blueprint we created from the list
- Click on "Save and Publish".
This will start the deployment of the addons configured in the "standard-blueprint" blueprint to the targeted cluster. The blueprint sync process can take a few minutes. Once complete, the cluster will display the current cluster blueprint details and whether the sync was successful or not.
Step 6: Verify Deployment¶
Users can optionally verify whether the correct resources have been created on the cluster. Click on the Kubectl button on the cluster to open a virtual terminal
- First, we will verify if the "newrelic" namespace has been created
kubectl get ns newrelic
NAME STATUS AGE
newrelic Active 76m
- Second, we will verify the New Relic Kubernetes Integration daemonsets
kubectl get daemonsets -n newrelic
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
newrelic-v171-newrelic-infrastructure 2 2 2 2 2 <none> 77m
newrelic-v171-newrelic-logging 2 2 2 2 2 <none> 77m
- Then, we will verify the New Relic Kubernetes Integration deployments
kubectl get deploy -n newrelic
NAME READY UP-TO-DATE AVAILABLE AGE
newrelic-v171-kube-state-metrics 1/1 1 1 77m
newrelic-v171-nri-kube-events 1/1 1 1 77m
newrelic-v171-nri-metadata-injection 1/1 1 1 77m
newrelic-v171-nri-prometheus 1/1 1 1 77m
- Next, we will verify the pods in the "newrelic" namespace. You should see something like the example below.
kubectl get pod -n newrelic
NAME READY STATUS RESTARTS AGE
newrelic-v171-kube-state-metrics-6df45994d8-fz6h6 1/1 Running 0 76m
newrelic-v171-newrelic-infrastructure-k67v7 1/1 Running 0 76m
newrelic-v171-newrelic-infrastructure-sjmt4 1/1 Running 0 76m
newrelic-v171-newrelic-logging-j6d9c 1/1 Running 0 76m
newrelic-v171-newrelic-logging-s4zkd 1/1 Running 3 76m
newrelic-v171-nri-kube-events-74f446c946-zrtst 2/2 Running 0 76m
newrelic-v171-nri-metadata-injection-679f86cd6c-nzp6k 1/1 Running 0 76m
newrelic-v171-nri-metadata-injection-job-dp79s 0/1 Completed 0 76m
newrelic-v171-nri-prometheus-579ddfd5b4-7hdph 1/1 Running 0 76m
- Then, we will verify the newrelic-infrastructure logs and ensure the agent posted payload successfully to New Relic website.
kubectl logs daemonset/newrelic-v171-newrelic-infrastructure -n newrelic --tail=50
Found 2 pods, using pod/newrelic-v171-newrelic-infrastructure-sjmt4
time="2020-09-18T04:40:52Z" level=info msg="runtime configuration" agentUser=root component="New Relic Infrastructure Agent" executablePath= maxProcs=1 pluginDir="[/etc/newrelic-infra/integrations.d /var/db/newrelic-infra/integrations.d]"
time="2020-09-18T04:40:53Z" level=info msg="Checking network connectivity..." component=AgentService service=newrelic-infra
time="2020-09-18T04:40:53Z" level=info msg=Initializing component=AgentService elapsedTime=341.281776ms service=newrelic-infra version=1.12.3
time="2020-09-18T04:40:53Z" level=info msg="New Relic infrastructure agent is running." component=AgentService elapsedTime=558.561255ms service=newrelic-infra
time="2020-09-18T04:40:53Z" level=info msg="Starting up agent..." component=Agent
time="2020-09-18T04:40:53Z" level=warning msg="failed to connect to DBus. make sure systemd is present." component=NotificationHandler
time="2020-09-18T04:40:53Z" level=warning msg="failed to init shutdown monitor" component=NotificationHandler error="no systemd found"
time="2020-09-18T04:40:53Z" level=info msg="Agent plugin" plugin=metadata/attributes
time="2020-09-18T04:40:53Z" level=info msg="Agent plugin" plugin=metadata/system
time="2020-09-18T04:40:53Z" level=info msg="Agent plugin" plugin=metadata/host_aliases
time="2020-09-18T04:40:53Z" level=info msg="Agent plugin" plugin=metadata/agent_config
time="2020-09-18T04:40:53Z" level=info msg="Agent plugin" plugin=metadata/proxy_config
time="2020-09-18T04:40:53Z" level=info msg="Integration info" arguments="map[]" command=metrics commandLine="[./bin/nri-kubernetes --metrics]" env-vars="map[CLUSTER_NAME:demo-spot-eks ETCD_TLS_SECRET_NAMESPACE:<HIDDEN> KUBERNETES_SERVICE_HOST:10.0.0.1 KUBERNETES_SERVICE_PORT:443 NRK8S_NODE_NAME:aks-agentpool-29632813-vmss000001 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin VERBOSE:0]" instance=nri-kubernetes integration=com.newrelic.kubernetes interval=15 labels="map[]" os=linux prefix=integration/com.newrelic.kubernetes protocolVersion=2 workingDir=/var/db/newrelic-infra/newrelic-integrations
time="2020-09-18T04:40:53Z" level=info msg="Integration health check starting" instance=nri-kubernetes integration=com.newrelic.kubernetes prefix=integration/com.newrelic.kubernetes working-dir=/var/db/newrelic-infra/newrelic-integrations
time="2020-09-18T04:40:53Z" level=info msg="connect got id" agent-guid=Mjg3NzY5M3xJTkZSQXxOQXwzMDQyNDc2Mzk0NTE0Njk5NTM agent-id=304247639451469953 component=IdentityConnectService
time="2020-09-18T04:40:56Z" level=info msg="Integration health check finished with success" instance=nri-kubernetes integration=com.newrelic.kubernetes prefix=integration/com.newrelic.kubernetes working-dir=/var/db/newrelic-infra/newrelic-integrations
- Then, we will verify the newrelic-logging logs and ensure the agent posted payload successfully to New Relic website.
kubectl logs daemonset/newrelic-v171-newrelic-logging -n newrelic --tail=20
Found 2 pods, using pod/newrelic-v171-newrelic-logging-rbpjz
Fluent Bit v1.0.3
Copyright (C) Treasure Data
[2020/09/18 04:00:59] [ info] [storage] initializing...
[2020/09/18 04:00:59] [ info] [storage] in-memory
[2020/09/18 04:00:59] [ info] [storage] normal synchronization mode, checksum disabled
[2020/09/18 04:00:59] [ info] [engine] started (pid=1)
[2020/09/18 04:01:00] [ warn] [filter_kube] merge_json_log is deprecated, enabling 'merge_log' option instead
[2020/09/18 04:01:00] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
[2020/09/18 04:01:00] [ info] [filter_kube] local POD info OK
[2020/09/18 04:01:00] [ info] [filter_kube] testing connectivity with API server...
[2020/09/18 04:01:00] [ warn] [filter_kube] could not get meta for POD node-jke1xgm-172-16-7-21-demo-azure-onprem
[2020/09/18 04:01:00] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
Step 7: Verify Data Collected¶
In order to view the collected metrics, events, logs, ... login to your New Relic account and check the Infrastructure and Logs.
- Shown below is an illustrative example of the Infrastructure Kubernetes Dashboard built from the metrics collected by the deployed New Relic Kubernetes Integration in your clusters.
- And below is another illustrative example of logs from the Kubernetes clusters collected by the deployed New Relic Kubernetes Integration in your clusters.
Recap¶
Congratulations! You have successfully created a custom cluster blueprint with the "newrelic" addon and applied to a managed cluster. You can now use this blueprint on as many clusters as you require.