1. AI Asset Management with Periodic AWS S3 Inventory Reports


    To achieve AI Asset Management with periodic AWS S3 inventory reports, we will set up a configuration that involves creating an S3 bucket that will store our assets and a second S3 bucket where the inventory reports will be placed. AWS S3 inventory provides CSV, ORC, or Parquet files that list objects and their corresponding metadata on a daily or weekly basis for a specified S3 bucket.

    With Pulumi, we can automate the creation of these resources and configure the inventory to run at the desired frequency. In this Pulumi program, we will use the aws.s3.Inventory resource, which allows us to define the parameters for S3 inventory reports such as the frequency, format, and destination where the reports will be stored.

    The inventory report can be generated daily or weekly and will include all the versions of objects, which is helpful if versioning is enabled on the source bucket. The report will be delivered to the defined destination bucket, and you can specify a prefix under which you want the report to be placed. We will also configure the destination bucket to receive the inventory reports.

    Below is a detailed Pulumi program written in Python that sets up this configuration:

    import pulumi import pulumi_aws as aws # Create an S3 bucket to store our managed assets asset_bucket = aws.s3.Bucket("assetBucket") # Create a second S3 bucket where inventory reports will be published report_bucket = aws.s3.Bucket("reportBucket") # S3 bucket to hold the inventory report with encryption enabled destination_bucket = aws.s3.Bucket("destinationBucket", server_side_encryption_configuration=aws.s3.BucketServerSideEncryptionConfigurationArgs( rule=aws.s3.BucketServerSideEncryptionConfigurationRuleArgs( apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationRuleApplyServerSideEncryptionByDefaultArgs( sse_algorithm="AES256", ), ), )) # Define S3 Inventory configuration inventory_configuration = aws.s3.Inventory("inventoryConfiguration", bucket=asset_bucket.id, destination=aws.s3.InventoryDestinationArgs( bucket=aws.s3.InventoryDestinationBucketArgs( format="ORC", bucket_arn=destination_bucket.arn, prefix="inventory", encryption=aws.s3.InventoryDestinationBucketEncryptionArgs( sse_kms=aws.s3.InventoryDestinationBucketEncryptionSseKmsArgs( key_id="alias/aws/s3" # using default AWS managed KMS encryption ) ) ) ), enabled=True, included_object_versions="All", schedule=aws.s3.InventoryScheduleArgs( frequency="Daily" ), ) # Export the URLs of the buckets pulumi.export("asset_bucket_url", asset_bucket.website_endpoint) pulumi.export("report_bucket_url", report_bucket.website_endpoint) pulumi.export("destination_bucket_url", destination_bucket.website_endpoint)

    Here's what the code does:

    • We first create two S3 buckets: asset_bucket for storing the assets and destination_bucket where the inventory reports will be stored.
    • The destination_bucket has server-side encryption enabled for added security.
    • We then set up the inventory configuration by creating an instance of aws.s3.Inventory.
      • We link it to our asset_bucket using the bucket attribute.
      • We specify the destination where the inventory reports will be delivered, including the bucket ARN, the report format we want, the prefix for the report path, and encryption details.
      • The enabled flag is set to True to enable the inventory.
      • included_object_versions is set to "All" to capture all object versions in the inventory if versioning is enabled.
      • The schedule defines how often the inventory should be generated—in this case, daily.

    Finally, the program exports the URLs for the asset_bucket, report_bucket, and destination_bucket so we can easily access these resources.

    With this Pulumi program, the AWS S3 assets will be automatically managed, and inventory reports will be periodically generated, allowing for efficient tracking and management of assets within S3.