Cost Management for Machine Learning Projects on AWS Budgets

Question

Pulumi · Accepted Answer

To implement cost management for Machine Learning projects on AWS, you can make use of AWS Budgets to monitor your costs and adhere to a predefined budget. AWS Budgets allows you to set custom budget limits and receive notifications when costs or usage exceed your budgeted amount. It can be particularly useful for Machine Learning projects where costs can escalate quickly due to resource-intensive workloads.

In this context, I will write a Pulumi Python program to create an AWS Budget that tracks the cost of a specific resource associated with Machine Learning projects, such as Amazon SageMaker. To illustrate, let's assume that you have an Amazon SageMaker instance for your Machine Learning workload and you want to ensure that your spending on SageMaker does not exceed a set threshold.

The following Pulumi program creates an AWS Budget and configures a notification for when the actual or forecasted cost exceeds 90% of the budgeted amount. The notification will be sent to an SNS topic, which you can then subscribe to with your email or SMS for alerts.

```python
import pulumi
import pulumi_aws as aws

# Create an SNS topic that will receive budget notifications
budget_notification_topic = aws.sns.Topic("budgetNotificationTopic")

# Subscribe an email address to the SNS topic (replace with your email address)
email_subscription = aws.sns.TopicSubscription("budgetNotificationEmail",
                                               topic=budget_notification_topic.arn,
                                               protocol="email",
                                               endpoint="your-email@example.com")

# Create a budget for SageMaker services
sagemaker_budget = aws.budgets.Budget("sagemakerBudget",
                                      budget_type="COST",
                                      limit_amount="100",  # Set your budget limit here
                                      limit_unit="USD",
                                      time_unit="MONTHLY",
                                      cost_filters={
                                          "Service": "Amazon SageMaker"
                                      },
                                      notifications=[
                                          {
                                              "comparison_operator": "GREATER_THAN",
                                              "notification_type": "ACTUAL_OR_FORECASTED",
                                              "threshold": 90,  # Notify at 90% of the budgeted amount
                                              "threshold_type": "PERCENTAGE",
                                              "subscriber_sns_topic_arns": [budget_notification_topic.arn]
                                          }
                                      ])

# Exporting the SNS topic ARN to access it outside of Pulumi if necessary
pulumi.export("budget_notification_topic_arn", budget_notification_topic.arn)
```

Here's a brief explanation of the key components:

- `budget_notification_topic`: This creates an Amazon SNS topic to which budget notifications will be published.
- `email_subscription`: This subscribes an email address to the SNS topic, allowing you to receive the budget notifications via email.
- `sagemaker_budget`: This represents the AWS Budget, where you define the parameters of your budget, such as the type (cost in this case), amount, unit, and time period. The `cost_filters` parameter allows you to specify that you want the budget to track costs associated with Amazon SageMaker.
- `notifications`: This contains configuration for notifications, specifying the threshold at which you want to receive alerts (90% in this example), the threshold type (percentage of the budget), and the SNS topic ARN that you are publishing notifications to.

Please replace `"your-email@example.com"` with your actual email address to receive the budget notifications.

To use this program, you'll need to have the Pulumi CLI installed, an AWS account, and the necessary AWS credentials configured on your workstation or in your environment where Pulumi can access them. Once everything is set up, running `pulumi up` with this program will apply the configuration to your AWS account, setting up the budget and notifications for your Machine Learning project's cost management.