AI Asset Management with Periodic AWS S3 Inventory Reports
PythonTo achieve AI Asset Management with periodic AWS S3 inventory reports, we will set up a configuration that involves creating an S3 bucket that will store our assets and a second S3 bucket where the inventory reports will be placed. AWS S3 inventory provides CSV, ORC, or Parquet files that list objects and their corresponding metadata on a daily or weekly basis for a specified S3 bucket.
With Pulumi, we can automate the creation of these resources and configure the inventory to run at the desired frequency. In this Pulumi program, we will use the
aws.s3.Inventory
resource, which allows us to define the parameters for S3 inventory reports such as the frequency, format, and destination where the reports will be stored.The inventory report can be generated daily or weekly and will include all the versions of objects, which is helpful if versioning is enabled on the source bucket. The report will be delivered to the defined destination bucket, and you can specify a prefix under which you want the report to be placed. We will also configure the destination bucket to receive the inventory reports.
Below is a detailed Pulumi program written in Python that sets up this configuration:
import pulumi import pulumi_aws as aws # Create an S3 bucket to store our managed assets asset_bucket = aws.s3.Bucket("assetBucket") # Create a second S3 bucket where inventory reports will be published report_bucket = aws.s3.Bucket("reportBucket") # S3 bucket to hold the inventory report with encryption enabled destination_bucket = aws.s3.Bucket("destinationBucket", server_side_encryption_configuration=aws.s3.BucketServerSideEncryptionConfigurationArgs( rule=aws.s3.BucketServerSideEncryptionConfigurationRuleArgs( apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs( sse_algorithm="AES256", ), ), )) # Define S3 Inventory configuration inventory_configuration = aws.s3.Inventory("inventoryConfiguration", bucket=asset_bucket.id, destination=aws.s3.InventoryDestinationArgs( bucket=aws.s3.InventoryDestinationBucketArgs( format="ORC", bucket_arn=destination_bucket.arn, prefix="inventory", encryption=aws.s3.InventoryDestinationBucketEncryptionArgs( sse_kms=aws.s3.InventoryDestinationBucketEncryptionSseKmsArgs( key_id="alias/aws/s3" # using default AWS managed KMS encryption ) ) ) ), enabled=True, included_object_versions="All", schedule=aws.s3.InventoryScheduleArgs( frequency="Daily" ), ) # Export the URLs of the buckets pulumi.export("asset_bucket_url", asset_bucket.website_endpoint) pulumi.export("report_bucket_url", report_bucket.website_endpoint) pulumi.export("destination_bucket_url", destination_bucket.website_endpoint)
Here's what the code does:
- We first create two S3 buckets:
asset_bucket
for storing the assets anddestination_bucket
where the inventory reports will be stored. - The
destination_bucket
has server-side encryption enabled for added security. - We then set up the inventory configuration by creating an instance of
aws.s3.Inventory
.- We link it to our
asset_bucket
using thebucket
attribute. - We specify the destination where the inventory reports will be delivered, including the bucket ARN, the report format we want, the prefix for the report path, and encryption details.
- The
enabled
flag is set toTrue
to enable the inventory. included_object_versions
is set to"All"
to capture all object versions in the inventory if versioning is enabled.- The
schedule
defines how often the inventory should be generated—in this case, daily.
- We link it to our
Finally, the program exports the URLs for the
asset_bucket
,report_bucket
, anddestination_bucket
so we can easily access these resources.With this Pulumi program, the AWS S3 assets will be automatically managed, and inventory reports will be periodically generated, allowing for efficient tracking and management of assets within S3.
- We first create two S3 buckets: