1. Packages
  2. Azure Classic
  3. How-to Guides
  4. Spark on Azure HDInsight

We recommend using Azure Native.

Azure Classic v5.67.1 published on Tuesday, Feb 27, 2024 by Pulumi

Spark on Azure HDInsight

azure logo

We recommend using Azure Native.

Azure Classic v5.67.1 published on Tuesday, Feb 27, 2024 by Pulumi

    View Code Deploy

    An example Pulumi component that deploys a Spark cluster on Azure HDInsight.

    Running the App

    1. Create a new stack:

      $ pulumi stack init dev
      
    2. Login to Azure CLI (you will be prompted to do this during deployment if you forget this step):

      $ az login
      
    3. Specify the Azure location to use:

      $ pulumi config set azure:location WestUS
      
    4. Define Spark username and password (make it complex enough to satisfy Azure policy):

      $ pulumi config set username <value>
      $ pulumi config set --secret password <value>
      
    5. Run pulumi up to preview and deploy changes:

      $ pulumi up
      Previewing changes:
      ...
      
      Performing changes:
      ...
      info: 5 changes performed:
          + 5 resources created
      Update duration: 15m6s
      
    6. Check the deployed Spark endpoint:

      $ pulumi stack output endpoint
      https://myspark1234abcd.azurehdinsight.net/
      
      # For instance, Jupyter notebooks are available at https://myspark1234abcd.azurehdinsight.net/jupyter/
      # Follow https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-load-data-run-query to test it out
      
    azure logo

    We recommend using Azure Native.

    Azure Classic v5.67.1 published on Tuesday, Feb 27, 2024 by Pulumi