Kubernetes Secrets for Sensitive Data in Machine Learning Pipelines
PythonWhen working with machine learning pipelines on Kubernetes, you often need to handle sensitive data such as API keys, credentials, or other secrets. Kubernetes provides a resource called
Secret
to securely store and manage sensitive information.In Pulumi, to create a Kubernetes
Secret
, you can use thepulumi_kubernetes.core.v1.Secret
resource. TheSecret
resource allows you to store sensitive data as key-value pairs. You can use the data stored in theSecret
in your Kubernetes pods by referencing it in your pod definition.Here's a program that demonstrates how to create a Kubernetes
Secret
using Pulumi. We will use a fictitious example where we haveAWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
as our secrets to be used in a machine learning pipeline.First, I'll explain how to set up a
Secret
resource, and then I'll provide the actual Pulumi program in Python:-
Import the Pulumi Kubernetes package: This gives you access to the Kubernetes resource APIs that Pulumi provides.
-
Create a
Secret
resource: Pass the secrets as key-value pairs under thedata
orstringData
parameter. WithstringData
, you can directly pass the secrets as strings without encoding them. Pulumi will handle the base64 encoding for you. -
Reference the
Secret
in a Kubernetes Pod: While this example will not create a pod, I'll show where you would reference theSecret
.
Let's see how this works in code:
import pulumi import pulumi_kubernetes as k8s # This example assumes you have a Kubernetes cluster configured and set as the current context in your kubeconfig. # Creating a Kubernetes Secret with sensitive data for a machine learning pipeline. ml_secret = k8s.core.v1.Secret("ml-secret", metadata=k8s.meta.v1.ObjectMetaArgs( name="ml-secret", # Name of the secret ), # Data within stringData is managed as plaintext and does not require manual base64 encoding. # The Pulumi program will automatically encode it as it deploys to the Kubernetes cluster. string_data={ "AWS_ACCESS_KEY_ID": "youraccesskeyid", # Replace with your AWS Access Key ID "AWS_SECRET_ACCESS_KEY": "yoursecretaccesskey" # Replace with your AWS Secret Access Key } ) # Exports the secret name. This is useful for referencing it in other resources such as pods. pulumi.export('secret_name', ml_secret.metadata['name'])
Note:
-
In the
string_data
, replace"youraccesskeyid"
and"yoursecretaccesskey"
with your actual AWS credentials or any other secrets you want to manage. -
Remember to never hard-code credentials in your Pulumi program. Instead, use environment variables or Pulumi configuration to pass sensitive data.
-
When referencing this secret in a Kubernetes pod, you would use the secret's name in the pod's
env
orenvFrom
configuration to expose the secret's keys as environment variables inside the containers.
This program will create a Kubernetes secret named
ml-secret
with two keys for AWS credentials. You can apply the same pattern to store other kinds of secrets you may need in your machine learning pipelines.-