How do I deploy the Spark Helm chart on Azure Kubernetes Service (AKS)?
In this tutorial, we will deploy the Spark Helm chart on Azure Kubernetes Service (AKS) using Pulumi. We will create an AKS cluster, add a Helm chart repository, and deploy the Spark Helm chart to the cluster.
Key Points:
- Create an Azure Resource Group.
- Create an Azure Kubernetes Service (AKS) cluster.
- Add a Helm chart repository.
- Deploy the Spark Helm chart to the AKS cluster.
import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure-native";
import * as k8s from "@pulumi/kubernetes";
// Create an Azure Resource Group
const resourceGroup = new azure.resources.ResourceGroup("resourceGroup", {
resourceGroupName: "spark-aks-rg",
location: "WestUS",
});
// Create an AKS Cluster
const cluster = new azure.containerservice.ManagedCluster("aksCluster", {
resourceGroupName: resourceGroup.name,
resourceName: "spark-aks-cluster",
location: resourceGroup.location,
agentPoolProfiles: [{
count: 3,
maxPods: 110,
mode: "System",
name: "agentpool",
osType: "Linux",
type: "VirtualMachineScaleSets",
vmSize: "Standard_DS2_v2",
}],
dnsPrefix: resourceGroup.name,
enableRBAC: true,
kubernetesVersion: "1.21.2",
linuxProfile: {
adminUsername: "aksuser",
ssh: { publicKeys: [{ keyData: "<YOUR_SSH_PUBLIC_KEY>" }] },
},
nodeResourceGroup: "spark-aks-nodes",
servicePrincipalProfile: {
clientId: "<YOUR_CLIENT_ID>",
secret: "<YOUR_CLIENT_SECRET>",
},
});
// Export the Kubeconfig
const creds = azure.containerservice.listManagedClusterUserCredentialsOutput({
resourceGroupName: resourceGroup.name,
resourceName: cluster.name,
});
const kubeconfig = creds.kubeconfigs[0].value.apply(enc => Buffer.from(enc, "base64").toString("ascii"));
export const kubeconfigExport = kubeconfig;
// Create a Kubernetes provider using the exported Kubeconfig
const k8sProvider = new k8s.Provider("k8sProvider", {
kubeconfig: kubeconfig,
});
// Add the Helm repository and deploy the Spark Helm chart
const sparkHelmRelease = new k8s.helm.v3.Release("sparkHelmRelease", {
chart: "spark",
version: "1.0.0",
repositoryOpts: {
repo: "https://helm.bitnami.com/bitnami",
},
values: {},
}, { provider: k8sProvider });
Summary
In this tutorial, we deployed the Spark Helm chart on Azure Kubernetes Service (AKS) using Pulumi. We started by creating an Azure Resource Group and an AKS cluster. We then added a Helm chart repository and deployed the Spark Helm chart to the AKS cluster. This setup allows us to run Spark jobs on a managed Kubernetes cluster in Azure.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.