Managed Identity Authorization for AI Data Pipelines in Azure

Question

Pulumi · Accepted Answer

To set up Managed Identity Authorization for AI Data Pipelines in Azure with Pulumi, we'll walk through the process of creating an Azure Machine Learning workspace, an Online Endpoint for AI inference, and configuring Managed Identity to secure it.

Managed Identity provides an identity for applications to use when connecting to resources that support Azure Active Directory (AAD) authentication. It eliminates the need for credentials to be stored within your code, significantly improving the security of your application.

Here's a step-by-step guide and a Pulumi Python program to accomplish this:

1. **Create an Azure Resource Group**: This will be the container for the Azure resources we are going to deploy.
2. **Deploy an Azure Machine Learning Workspace**: This is necessary for developing and hosting machine learning models.
3. **Set up an Online Endpoint**: This endpoint will be used to serve the machine learning model over HTTP(S) for real-time inference. It is part of the Azure Machine Learning workspace resources.
4. **Enable Managed Identity Authorization**: For the Online Endpoint, you'll activate the identity and establish permissions to access other resources securely.

Now let's write the Pulumi program to implement this:

```python
import pulumi
from pulumi_azure_native import machinelearningservices, resources

# First, we set up a new Azure Resource Group
resource_group = resources.ResourceGroup("ai_resource_group")

# Then, we create the Azure Machine Learning Workspace
ml_workspace = machinelearningservices.Workspace("ml_workspace",
    resource_group_name=resource_group.name,
    sku=machinelearningservices.SkuArgs(name="Basic"),
    location=resource_group.location,
    identity=machinelearningservices.IdentityArgs(type="SystemAssigned"),
)

# Now we create the Online Endpoint for real-time inference
# We secure this endpoint by enabling Managed Identity Authorization
online_endpoint = machinelearningservices.OnlineEndpoint("online_endpoint",
    resource_group_name=resource_group.name,
    workspace_name=ml_workspace.name,
    location=resource_group.location,
    identity=machinelearningservices.IdentityArgs(type="SystemAssigned"),
    online_endpoint_properties=machinelearningservices.OnlineEndpointPropertiesArgs(
        auth_mode="Key",
    ),
)

# Export the important endpoints
pulumi.export("Resource Group Name", resource_group.name)
pulumi.export("Machine Learning Workspace Name", ml_workspace.name)
pulumi.export("Online Endpoint Name", online_endpoint.name)
```

In this program:

- We import the necessary Pulumi packages.

- We create a resource group to hold all the resources needed for the AI data pipeline.

- We then deploy the Azure Machine Learning workspace and enable system-assigned Managed Identity for it, which creates an identity in Azure AD that is automatically managed by Azure. The `sku` is set to "Basic" for the sake of this example, but it should be chosen based on your specific needs.

- Next, we create an Online Endpoint that serves the machine learning model over HTTP(S). We secure the access to this endpoint by activating the Managed Identity authorization method, which in this case is simply using keys (`auth_mode="Key"`).

- Finally, we export the names of the created resources for easy access and reference.

This Pulumi program sets up an AI Data Pipeline with Managed Identity Authorization, enhancing the security posture by removing the need to manage credentials explicitly. The system-assigned identity here will be used to authenticate to any Azure service that supports Azure AD authentication, without having to use service principals or secret management.