Managed Identity Authentication for Automated ML Pipelines

Question

Pulumi · Accepted Answer

To establish Managed Identity Authentication for Automated ML Pipelines, especially if you are deploying them in a cloud environment, you will need several components:

1. A managed identity that provides Azure services with an identity within Azure Active Directory and can be used to authenticate to any service that supports Azure AD authentication.
2. An ML service, such as Azure Machine Learning, to create and manage the ML pipeline.
3. Parameters to control access to your ML pipelines, such as role assignments or policy assignments.

The following Pulumi Python program demonstrates how to set up a managed identity and configure it for an Azure Machine Learning pipeline using Azure Native resources. This will create an Azure Machine Learning Workspace and then an Online Endpoint, which is a scalable web service on Azure used to deploy ML models as services easily and securely.

The `OnlineEndpoint` resource will use a system-assigned identity, which is a type of managed identity automatically created by Azure.

Here's the code example demonstrating how to create these resources:

```python
import pulumi
import pulumi_azure_native as azure_native

# Create an Azure Resource Group
resource_group = azure_native.resources.ResourceGroup('rg')

# Create an Azure Machine Learning Workspace with System-Assigned Identity
# This workspace will provide the platform for building, training, and deploying ML models.
aml_workspace = azure_native.machinelearningservices.Workspace(
    'amlWorkspace',
    location=resource_group.location,
    resource_group_name=resource_group.name,
    identity=azure_native.machinelearningservices.IdentityArgs(
        type='SystemAssigned'  # This enables Managed Identity for the workspace
    ),
    sku=azure_native.machinelearningservices.SkuArgs(
        name="Basic"
    )
)

# Create an Online Endpoint with the Managed Identity,
# which will be used to deploy ML models for real-time inference.
online_endpoint = azure_native.machinelearningservices.OnlineEndpoint(
    'onlineEndpoint',
    location=resource_group.location,
    resource_group_name=resource_group.name,
    workspace_name=aml_workspace.name,
    identity=azure_native.machinelearningservices.IdentityArgs(
        type='SystemAssigned'  # This enables Managed Identity for the endpoint
    ),
    online_endpoint_properties=azure_native.machinelearningservices.OnlineEndpointTypePropertiesArgs(
        auth_mode='AMLToken'  # Select authentication mode, AMLToken for Azure Machine Learning authentication
    ),
)

# Export the important URLs and IDs that can be used to interact with the Machine Learning Workspace
pulumi.export('resource_group_name', resource_group.name)
pulumi.export('aml_workspace_name', aml_workspace.name)
pulumi.export('aml_workspace_id', aml_workspace.id)
pulumi.export('online_endpoint_name', online_endpoint.name)
pulumi.export('online_endpoint_url', online_endpoint.scoring_uri)  # Export the scoring URL to call the online endpoint
pulumi.export('identity_principal_id', aml_workspace.identity.apply(lambda id: id.principal_id))  # Export the Managed Identity Principal ID
```

This program does the following:
1. **Resource Group**: It sets up a new resource group which is a container for related resources for an Azure solution.
2. **Machine Learning Workspace**: Creates a new Azure Machine Learning workspace within the resource group. The workspace is the foundational block in the cloud that you use to experiment, train, and deploy machine learning models.
3. **Managed Identity**: We define an identity with the `SystemAssigned` type, which enables Managed Identity. This identity is automatically created and will be used by the workspace and the online endpoint.
4. **Online Endpoint**: An online endpoint is then created and configured to use the system-assigned managed identity for authentication purposes. This allows secure interactions with other Azure services such as Key Vault or Storage, without having to manage credentials.
5. **Outputs**: We export the workspace name, ID, endpoint name, and URL for external reference and interaction purposes, along with the Managed Identity Principal ID, which can be used for setting appropriate role-based access controls.

Remember, before running this Pulumi program, you must have the Azure Pulumi provider set up and configured with appropriate permissions. You can install the Pulumi CLI and run `pulumi up` to deploy this program. This operation will provision the described resources in Azure.