Spark on Azure HDInsight | Python

View Code Deploy

An example Pulumi component that deploys a Spark cluster on Azure HDInsight.

Running the App

  1. Create a new stack:

    $ pulumi stack init dev
  2. Login to Azure CLI (you will be prompted to do this during deployment if you forget this step):

    $ az login
  3. Specify the Azure location to use:

    $ pulumi config set azure:location WestUS
  4. Define Spark username and password (make it complex enough to satisfy Azure policy):

    $ pulumi config set username <value>
    $ pulumi config set --secret password <value>
  5. Run pulumi up to preview and deploy changes:

    $ pulumi up
    Previewing changes:
    Performing changes:
    info: 5 changes performed:
        + 5 resources created
    Update duration: 15m6s
  6. Check the deployed Spark endpoint:

    $ pulumi stack output endpoint
    # For instance, Jupyter notebooks are available at
    # Follow to test it out