1. Kubernetes Secrets for Sensitive Data in Machine Learning Pipelines


    When working with machine learning pipelines on Kubernetes, you often need to handle sensitive data such as API keys, credentials, or other secrets. Kubernetes provides a resource called Secret to securely store and manage sensitive information.

    In Pulumi, to create a Kubernetes Secret, you can use the pulumi_kubernetes.core.v1.Secret resource. The Secret resource allows you to store sensitive data as key-value pairs. You can use the data stored in the Secret in your Kubernetes pods by referencing it in your pod definition.

    Here's a program that demonstrates how to create a Kubernetes Secret using Pulumi. We will use a fictitious example where we have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as our secrets to be used in a machine learning pipeline.

    First, I'll explain how to set up a Secret resource, and then I'll provide the actual Pulumi program in Python:

    1. Import the Pulumi Kubernetes package: This gives you access to the Kubernetes resource APIs that Pulumi provides.

    2. Create a Secret resource: Pass the secrets as key-value pairs under the data or stringData parameter. With stringData, you can directly pass the secrets as strings without encoding them. Pulumi will handle the base64 encoding for you.

    3. Reference the Secret in a Kubernetes Pod: While this example will not create a pod, I'll show where you would reference the Secret.

    Let's see how this works in code:

    import pulumi import pulumi_kubernetes as k8s # This example assumes you have a Kubernetes cluster configured and set as the current context in your kubeconfig. # Creating a Kubernetes Secret with sensitive data for a machine learning pipeline. ml_secret = k8s.core.v1.Secret("ml-secret", metadata=k8s.meta.v1.ObjectMetaArgs( name="ml-secret", # Name of the secret ), # Data within stringData is managed as plaintext and does not require manual base64 encoding. # The Pulumi program will automatically encode it as it deploys to the Kubernetes cluster. string_data={ "AWS_ACCESS_KEY_ID": "youraccesskeyid", # Replace with your AWS Access Key ID "AWS_SECRET_ACCESS_KEY": "yoursecretaccesskey" # Replace with your AWS Secret Access Key } ) # Exports the secret name. This is useful for referencing it in other resources such as pods. pulumi.export('secret_name', ml_secret.metadata['name'])


    • In the string_data, replace "youraccesskeyid" and "yoursecretaccesskey" with your actual AWS credentials or any other secrets you want to manage.

    • Remember to never hard-code credentials in your Pulumi program. Instead, use environment variables or Pulumi configuration to pass sensitive data.

    • When referencing this secret in a Kubernetes pod, you would use the secret's name in the pod's env or envFrom configuration to expose the secret's keys as environment variables inside the containers.

    This program will create a Kubernetes secret named ml-secret with two keys for AWS credentials. You can apply the same pattern to store other kinds of secrets you may need in your machine learning pipelines.