Using kubernetes datadoghq.com with kiali.io

Question

Pulumi · Accepted Answer

To integrate Datadog for monitoring your Kubernetes infrastructure along with visualizing the service mesh using Kiali, you'd need to deploy Datadog and Kiali into your Kubernetes cluster. Both Datadog and Kiali provide Helm charts that simplify their deployment. Pulumi's Kubernetes provider can deploy Helm charts with ease.

First, you'll need to install Pulumi and set up a Kubernetes cluster. Ensure you have `kubectl` configured to communicate with your Kubernetes cluster. With Pulumi, you can deploy to any cluster you have `kubectl` configured for, including local clusters (like kind or minikube), managed clusters from cloud providers, and more.

Next, let's create a Pulumi program that will deploy the Datadog agents to your cluster for collection of metrics, traces, and logs, and then deploy Kiali to visualize your service mesh. We'll use TypeScript to define our infrastructure.

Below is a detailed program that performs these steps:

1. **Datadog Agent Deployment**: Assumes you have a Datadog account and that you have the Datadog API and APP keys available. These keys are required to send data from your cluster to Datadog. In practice, these should be stored in a secure manner using a secret management tool or Pulumi's secret management.
   
2. **Kiali Deployment**: Kiali requires you already have a service mesh like Istio installed on your cluster. It will connect to Istio's Prometheus instance to collect metrics and display the service mesh topology.

Here is the Pulumi TypeScript program:

```typescript
import * as k8s from '@pulumi/kubernetes';
import * as pulumi from '@pulumi/pulumi';

// Create an instance of the Kubernetes provider configured to a specific cluster context.
const clusterProvider = new k8s.Provider('my-k8s-provider', {
  // Specify your cluster context if not using the default context
});

// Deploy the Datadog agent using the Datadog Helm chart.
const datadogChart = new k8s.helm.v3.Chart('datadog', {
  chart: 'datadog',
  version: '2.21.6', // Use the version of the chart that suits your needs
  fetchOpts: {
    repo: 'https://helm.datadoghq.com/',
  },
  values: {
    apiKey: pulumi.secret('YOUR_DATADOG_API_KEY'), // Replace with your Datadog API key
    appKey: pulumi.secret('YOUR_DATADOG_APP_KEY'), // Replace with your Datadog APP key
    datadog: {
      site: 'datadoghq.com',
    },
  },
}, { provider: clusterProvider });

// Deploy Kiali using its Helm chart.
const kialiChart = new k8s.helm.v3.Chart('kiali', {
  chart: 'kiali-server',
  version: '1.44.0', // Use the version of the chart that suits your needs
  fetchOpts: {
    repo: 'https://kiali.org/helm-charts/',
  },
  // Specify any values that customize the Kiali deployment to your needs
}, { provider: clusterProvider });

// Export the urls to access the deployed services
export const datadogUrl = pulumi.interpolate`https://app.datadoghq.com/`;
export const kialiUrl = 'Please refer to Kiali service details from your cluster with `kubectl get svc -n kiali`

```

In the above program:
- We instantiate the Kubernetes provider, optionally specifying a cluster context.
- The `datadogChart` resource deploys Datadog agents to your cluster, which collect metrics, logs, and traces. Be sure to replace `'YOUR_DATADOG_API_KEY'` and `'YOUR_DATADOG_APP_KEY'` with your actual Datadog API and APP keys.
- The `kialiChart` resource deploys Kiali to your cluster, which enables you to visualize your service mesh configuration.
  
**Note**: For a production setup, you should store your sensitive keys securely, for example using [Pulumi Secrets](https://www.pulumi.com/docs/intro/concepts/secrets/).

To apply this Pulumi program, save the code in a `index.ts` file, and run the following commands:

```bash
pulumi up
```

This will preview and deploy the changes to your cluster. Once the deployment is successful, you can access the Datadog dashboard and Kiali dashboard to monitor and visualize your Kubernetes services.

Remember, before running `pulumi up`, make sure:

- You have the `pulumi` CLI installed and set up.
- You have your Kubernetes context configured correctly.
- You've replaced the placeholders with your actual Datadog API and APP keys.