Centralized Artifact Hub for AI Model Versioning on Harbor

Question

Pulumi · Accepted Answer

To create a centralized artifact hub for AI Model Versioning on Harbor, you can use Pulumi to provision the required infrastructure. Harbor is an open-source container image registry that secures images with role-based access control, scans images for vulnerabilities, and signs images as trusted. It's particularly useful in a scenario where you're dealing with models and other artifacts, which need to be versioned and securely managed.

The resources we'll use from the Harbor provider for Pulumi include:

- `harbor.Project`: To create a project in Harbor. It acts as a namespace for all your artifacts.
- `harbor.User`: To create a user that will have access to the Harbor registry.
- `harbor.Registry`: To set up the registry instance we'll be pushing artifacts to or pulling from.
- `harbor.RobotAccount`: To create a robot account for automated tools to perform operations in a project without a manual user account.

Here's a program in Python that utilizes these resources to set up a Harbor artifact hub for versioning AI models:

```python
import pulumi
import pulumi_harbor as harbor

# Create a new project in Harbor for storing AI models
ai_project = harbor.Project("aiModelsProject",
    name="ai-models",
    public=False,
    registry_id=1, # Assumes a registry with ID 1 already exists. Otherwise, create one.
    vulnerability_scanning=True # Enable vulnerability scanning for the artifacts
)

# Create a Harbor user with access to the Harbor registry
ai_user = harbor.User("aiModelUser",
    email="aimodels_user@example.com",
    full_name="AI Models User",
    password=pulumi.Output.secret("initial-password"),
    username="aimodels_user",
)

# Create a robot account for CI/CD pipelines to interact with the project
ci_robot = harbor.RobotAccount("ciRobotAccount",
    name="ci-robot",
    level="project",
    secret=pulumi.Output.secret("robot-account-secret"),
    permissions=[
        {
            "kind": "project",
            "namespace": ai_project.name,
            "accesses": [
                {
                    "resource": "/repository",
                    "action": "push"
                },
                {
                    "resource": "/repository",
                    "action": "pull"
                }
            ]
        }
    ]
)

# Assuming you've got your Harbor registry setup off-screen, you might create it like this:
# registry = harbor.Registry("defaultRegistry",
#     name="default",
#     endpoint_url="https://my-registry.example.com",
#     access_id="admin",
#     access_secret=pulumi.Output.secret("registry-secret"),
#     provider_name="harbor"
# )

# Export out the generated user name and robot account token (for CI/CD pipelines)
pulumi.export("project_name", ai_project.name)
pulumi.export("user_name", ai_user.username)
pulumi.export("robot_account_token", ci_robot.secret)
```

In the code above:

- We create a new `harbor.Project` named `ai-models` for storing AI models. We enable vulnerability scanning on the project — this is handy to ensure all the Docker images stored are free of known vulnerabilities.
- We create a `harbor.User` that we’ll use to push and pull images to and from the Harbor registry. The password is marked as sensitive by Pulumi, so its value is not exposed in plain text in the state file.
- We create a `harbor.RobotAccount` for automated tools (like CI/CD pipelines) to interact with Harbor without the need for a manual user account. This is useful for scenarios like automated model training and deployment pipelines, which can push new versions of AI models to Harbor.
- We export some details about the project name, user name, and robot account secret, which can be used in CI/CD scripts, or for administration purposes.

Make sure that you've set up the registry in Harbor before running this code and replace the `registry_id` with the actual registry ID that you're using.

With these resources, you can start versioning your AI models in a centralized and secure way, leveraging Harbor's capabilities.