Spark on Azure HDInsight

This example lives in the pulumi/examples repository. Check out just this directory to use it:

$ git clone --filter=blob:none --sparse https://github.com/pulumi/examples pulumi-examples
$ git -C pulumi-examples sparse-checkout set classic-azure-py-hdinsight-spark
$ cd pulumi-examples/classic-azure-py-hdinsight-spark

An example Pulumi component that deploys a Spark cluster on Azure HDInsight.

Prerequisites#

1. Install Pulumi
2. Configure Azure credentials
3. Install Python

Deploying the example#

1. Create a new stack:

   pulumi stack init dev

2. Log in to the Azure CLI (you will be prompted to do this during deployment if you forget this step):

   az login

3. Specify the Azure location and subscription to use:

   pulumi config set azure:location WestUS
   pulumi config set azure:subscriptionId <YOUR_SUBSCRIPTION_ID>

4. Define the Spark username and password (make it complex enough to satisfy Azure policy):

   pulumi config set username <value>
   pulumi config set --secret password <value>

5. Install dependencies:

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt

6. Run pulumi up to preview and deploy changes:

   pulumi up

   Previewing changes:
   ...
   
   Performing changes:
   ...
   info: 5 changes performed:
       + 5 resources created
   Update duration: 15m6s

7. Check the deployed Spark endpoint:

   pulumi stack output endpoint

   https://myspark1234abcd.azurehdinsight.net/

   For instance, Jupyter notebooks are available at https://myspark1234abcd.azurehdinsight.net/jupyter/. Follow https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-load-data-run-query to test it out.

Cleaning up#

Once you are done, destroy the stack and remove it:

pulumi destroy
pulumi stack rm