Deploy the spark-history-server helm chart on Azure Kubernetes Service (AKS)
TypeScriptTo deploy the
spark-history-server
Helm chart on Azure Kubernetes Service (AKS), you'll need to follow several steps:-
Set up an AKS cluster: First, we need to provision an AKS cluster where the Helm chart will be deployed. We will define the resources required for the cluster such as node pools, networking, and identity configurations.
-
Install Helm on your machine: Helm is the package manager for Kubernetes and you need it installed to manage Kubernetes applications. Helm uses charts, which are packages of pre-configured Kubernetes resources.
-
Deploy the Helm chart: After setting up AKS, you will use Helm to deploy the
spark-history-server
onto the cluster.
Here's a TypeScript Pulumi program that:
- Creates an Azure Kubernetes Service (AKS) cluster.
- Deploys the
spark-history-server
chart onto the AKS cluster.
import * as azure from '@pulumi/azure'; import * as pulumi from '@pulumi/pulumi'; import * as k8s from '@pulumi/kubernetes'; import * as azuread from "@pulumi/azuread"; import * as random from "@pulumi/random"; // Create an Azure AD application for AKS const aksApp = new azuread.Application("aks", {}); // Create a service principal for the application so AKS can act on behalf of the application const aksSp = new azuread.ServicePrincipal("aksSp", { applicationId: aksApp.applicationId }); // Create the AD service principal password const aksSpPassword = new azuread.ServicePrincipalPassword("aksSpPassword", { servicePrincipalId: aksSp.id, endDate: "2099-01-01T00:00:00Z", }); // Generate an SSH key for the AKS nodes const sshKey = new random.RandomPassword("sshKey", { length: 20, special: false, }); // Now, let's create an AKS cluster. const resourceGroup = new azure.core.ResourceGroup("aksResourceGroup"); const aksCluster = new azure.containerservice.KubernetesCluster("aksCluster", { resourceGroupName: resourceGroup.name, defaultNodePool: { name: "aksagentpool", nodeCount: 3, vmSize: "Standard_D2_v2", }, dnsPrefix: `${pulumi.getStack()}-kube`, linuxProfile: { adminUsername: "aksuser", sshKey: { keyData: sshKey.result, }, }, servicePrincipal: { clientId: aksApp.applicationId, clientSecret: aksSpPassword.value, }, kubernetesVersion: "1.18.14", }); // Export the kubeconfig export const kubeconfig = aksCluster.kubeConfigRaw; // Create a k8s provider using the generated kubeconfig const k8sProvider = new k8s.Provider("k8sProvider", { kubeconfig: aksCluster.kubeConfigRaw, }); // Deploy the spark-history-server chart using the Helm Chart resource const sparkHistoryServerChart = new k8s.helm.v3.Chart("spark-history-server", { chart: "spark-history-server", version: "<version>", // Replace with specific chart version fetchOpts: { repo: "http://<helm-chart-repository>", // Specify the repository URL here }, }, { provider: k8sProvider }); // Export the necessary values or endpoints if required export const sparkHistoryServerServiceIP = sparkHistoryServerChart.getResourceProperty("v1/Service", "<service-name>", "status").apply(status => status.loadBalancer.ingress[0].ip);
Understanding the Code
- The first section sets up the Azure AD application and service principal, which are required for AKS to interact with other Azure resources securely.
- The SSH key is generated for secure communication with AKS nodes.
- The
azure.containerservice.KubernetesCluster
resource is used to create the AKS cluster with a specified version, node count, and VM sizes for the nodes. - We export the
kubeconfig
of the AKS cluster, which is used bykubectl
and other tools to interact with the Kubernetes cluster. - The
k8s.Provider
resource initializes a Kubernetes provider with thekubeconfig
that we get from the AKS cluster. - The
k8s.helm.v3.Chart
resource represents the Helm chart that we want to deploy. You need to specify the chart name, chart version, and Helm repo URL that hosts thespark-history-server
chart. - Finally, we export the service IP of the Spark History Server which you can use to access the Spark UI. Replace
<service-name>
with the actual name of the service created by the chart.
Requirements
- Ensure you have Pulumi CLI installed.
- Helm must be installed on your local machine.
- Azure CLI should be installed and configured.
- Replace
<version>
with the chart version andhttp://<helm-chart-repository>
with the actual Helm repository URL where thespark-history-server
chart is located. - Replace
<service-name>
with the actual Kubernetes service name created by the helm chart for the spark-history-server.
Run the Pulumi program with the usual
pulumi up
. After successfully running the program, the AKS cluster will be set up and thespark-history-server
will be deployed on it.-