We recommend using Azure Native.
Azure Classic v5.70.0 published on Wednesday, Mar 27, 2024 by Pulumi
Spark on Azure HDInsight
We recommend using Azure Native.
Azure Classic v5.70.0 published on Wednesday, Mar 27, 2024 by Pulumi
An example Pulumi component that deploys a Spark cluster on Azure HDInsight.
Running the App
Create a new stack:
$ pulumi stack init dev
Login to Azure CLI (you will be prompted to do this during deployment if you forget this step):
$ az login
Specify the Azure location to use:
$ pulumi config set azure:location WestUS
Define Spark username and password (make it complex enough to satisfy Azure policy):
$ pulumi config set username <value> $ pulumi config set --secret password <value>
Run
pulumi up
to preview and deploy changes:$ pulumi up Previewing changes: ... Performing changes: ... info: 5 changes performed: + 5 resources created Update duration: 15m6s
Check the deployed Spark endpoint:
$ pulumi stack output endpoint https://myspark1234abcd.azurehdinsight.net/ # For instance, Jupyter notebooks are available at https://myspark1234abcd.azurehdinsight.net/jupyter/ # Follow https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-load-data-run-query to test it out
We recommend using Azure Native.
Azure Classic v5.70.0 published on Wednesday, Mar 27, 2024 by Pulumi