1. Answers
  2. Deploy the Spark Helm Chart on Azure Kubernetes Service (AKS)

How do I deploy the Spark Helm chart on Azure Kubernetes Service (AKS)?

In this tutorial, we will deploy the Spark Helm chart on Azure Kubernetes Service (AKS) using Pulumi. We will create an AKS cluster, add a Helm chart repository, and deploy the Spark Helm chart to the cluster.

Key Points:

  • Create an Azure Resource Group.
  • Create an Azure Kubernetes Service (AKS) cluster.
  • Add a Helm chart repository.
  • Deploy the Spark Helm chart to the AKS cluster.
import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure-native";
import * as k8s from "@pulumi/kubernetes";

// Create an Azure Resource Group
const resourceGroup = new azure.resources.ResourceGroup("resourceGroup", {
    resourceGroupName: "spark-aks-rg",
    location: "WestUS",
});

// Create an AKS Cluster
const cluster = new azure.containerservice.ManagedCluster("aksCluster", {
    resourceGroupName: resourceGroup.name,
    resourceName: "spark-aks-cluster",
    location: resourceGroup.location,
    agentPoolProfiles: [{
        count: 3,
        maxPods: 110,
        mode: "System",
        name: "agentpool",
        osType: "Linux",
        type: "VirtualMachineScaleSets",
        vmSize: "Standard_DS2_v2",
    }],
    dnsPrefix: resourceGroup.name,
    enableRBAC: true,
    kubernetesVersion: "1.21.2",
    linuxProfile: {
        adminUsername: "aksuser",
        ssh: { publicKeys: [{ keyData: "<YOUR_SSH_PUBLIC_KEY>" }] },
    },
    nodeResourceGroup: "spark-aks-nodes",
    servicePrincipalProfile: {
        clientId: "<YOUR_CLIENT_ID>",
        secret: "<YOUR_CLIENT_SECRET>",
    },
});

// Export the Kubeconfig
const creds = azure.containerservice.listManagedClusterUserCredentialsOutput({
    resourceGroupName: resourceGroup.name,
    resourceName: cluster.name,
});
const kubeconfig = creds.kubeconfigs[0].value.apply(enc => Buffer.from(enc, "base64").toString("ascii"));
export const kubeconfigExport = kubeconfig;

// Create a Kubernetes provider using the exported Kubeconfig
const k8sProvider = new k8s.Provider("k8sProvider", {
    kubeconfig: kubeconfig,
});

// Add the Helm repository and deploy the Spark Helm chart
const sparkHelmRelease = new k8s.helm.v3.Release("sparkHelmRelease", {
    chart: "spark",
    version: "1.0.0",
    repositoryOpts: {
        repo: "https://helm.bitnami.com/bitnami",
    },
    values: {},
}, { provider: k8sProvider });

Summary

In this tutorial, we deployed the Spark Helm chart on Azure Kubernetes Service (AKS) using Pulumi. We started by creating an Azure Resource Group and an AKS cluster. We then added a Helm chart repository and deployed the Spark Helm chart to the AKS cluster. This setup allows us to run Spark jobs on a managed Kubernetes cluster in Azure.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

Sign up

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.

Sign up