Centralized ML Model Catalog Service with AWS ServiceCatalog

Question

Pulumi · Accepted Answer

To create a centralized Machine Learning (ML) Model Catalog Service using AWS Service Catalog, we'll need to define a combination of products, portfolios, and constraints that govern the use of those products.

AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. In the context of a Machine Learning Model Catalog Service, you might have different ML models that have been containerized or architected in various ways. AWS Service Catalog would allow data scientists and developers to deploy these models consistently and within compliance of the organization's rules.

We will use the following AWS Service Catalog resources from Pulumi's AWS provider:

- `Product`: Represents a product in the AWS Service Catalog that we want to offer, which could be a machine learning model packaged in some form, like a Docker container.
- `Portfolio`: A collection of Products. This is the catalog that users will interact with.
- `Constraint`: Governs the use of a Product within a Portfolio.
- `ProvisionedProduct`: Represents a provisioned product of a Product within a Portfolio.
- `PortfolioShare`: Manages access to the Service Catalog Portfolio.

Below is a Pulumi program written in Python. This program creates a simple ML Model Catalog Service where a machine learning model is defined as a product and added to a portfolio. The program assumes you have an existing ML model you want to distribute, which is defined in a CloudFormation template ready to be used by the Service Catalog.

```python
import pulumi
import pulumi_aws as aws

# Define the machine learning model product using a CloudFormation template.
# Replace `template_url` with the URL of your CloudFormation template, which defines the ML model.
ml_model_product = aws.servicecatalog.Product("MLModelProduct",
    name="MLModelProduct",
    owner="data-team",
    description="A machine learning model ready to be deployed",
    distributor="YourOrg",
    support_description="Contact our data team for any issues.",
    support_email="data-team@yourorg.com",
    support_url="https://yourorg.com/data-team",
    provisioning_artifact_parameters=aws.servicecatalog.ProductProvisioningArtifactParametersArgs(
        name="v1.0",
        description="Version 1.0 of the ML model",
        template_url="https://s3.amazonaws.com/path-to-your-cfn-template/template.yaml",
    )
)

# Create a portfolio that will include the ML model.
ml_model_portfolio = aws.servicecatalog.Portfolio("MLModelPortfolio",
    name="MLModelPortfolio",
    description="Portfolio containing machine learning models",
    provider_name="YourOrg"
)

# Associate the product with the portfolio.
ml_model_product_association = aws.servicecatalog.PortfolioProductAssociation("MLModelProductAssociation",
    portfolio_id=ml_model_portfolio.id,
    product_id=ml_model_product.id
)

# Define a constraint limiting the use of the ML model.
# As an example, we are creating a launch constraint that specifies an IAM role.
# This IAM role will dictate permissions when launching the product.
ml_model_constraint = aws.servicecatalog.LaunchRoleConstraint("MLModelConstraint",
    portfolio_id=ml_model_portfolio.id,
    product_id=ml_model_product.id,
    role_arn="arn:aws:iam::123456789012:role/ServiceCatalogLaunchRole" # Replace with an appropriate ARN
)

# Share the portfolio with other AWS accounts (optional).
# This would allow you to distribute the ML model across your organization.
# ml_model_portfolio_share = aws.servicecatalog.PortfolioShare("MLModelPortfolioShare",
#     portfolio_id=ml_model_portfolio.id,
#     principal_id="AWS_ACCOUNT_ID" # Replace with the AWS Account ID you want to share with.
# )

# Export the Portfolio ID and the Product ID.
# This allows you to reference them in other Pulumi stacks, or when provisioning products via the AWS Service Catalog.
pulumi.export("portfolio_id", ml_model_portfolio.id)
pulumi.export("product_id", ml_model_product.id)

# NOTE: Uncomment the PortfolioShare resource if it's needed in your setup.
```

This code snippet performs several actions:

- It creates a `Product` which is a representation of your ML model in the Service Catalog.
- It then creates a `Portfolio` which you can think of as a catalog that groups various service products together.
- The next step is associating the ML model product with the newly created portfolio.
- Then, a `LaunchRoleConstraint` is added to define an IAM role that has the necessary permissions to launch the product.
- Optionally (code commented out), you can share the created portfolio with other AWS accounts within your organization. This enables the ML models to be accessed by other parts of your organization.
- Finally, the portfolio and product IDs are exported which can be useful if you need to reference them