Immutable AI Dataset Storage with S3 Object Lock

Question

Pulumi · Accepted Answer

Creating immutable storage for an AI dataset is essential to ensure that the data cannot be altered or deleted after it has been written. This is critical in contexts where data integrity and retention are regulated, or where the immutability of data is required for data analysis consistency over time.

To achieve this using AWS S3, we need to configure a bucket with Object Lock enabled. Object Lock employs a Write Once, Read Many (WORM) model to protect your data from being modified or deleted. It's often used for compliance with regulations that require data to be immutable for a certain period.

In the Pulumi program below, we will create an S3 bucket with Object Lock configuration to enforce immutability. We will set a default retention period which dictates how long each object will be protected from alteration and deletion. Additionally, by declaring this in Pulumi, we can easily replicate, version, and track changes to our infrastructure as code.

Here's how we'll set it up in Python with Pulumi:

1. Import Pulumi's AWS package.
2. Create an S3 bucket with `objectLockEnabled` set to `True`.
3. Apply an Object Lock configuration to the bucket, specifying that objects should be locked in compliance mode and defining the retention period (for example, 90 days).

Let's go through the Pulumi Python program step by step.

```python
import pulumi
import pulumi_aws as aws

# Create a new bucket with Object Lock enabled.
# Documentation: https://www.pulumi.com/registry/packages/aws/api-docs/s3/bucket/#objectlockenabled_python
bucket = aws.s3.Bucket("aiDatasetBucket",
    versioning=aws.s3.BucketVersioningArgs(enabled=True),
    object_lock_configuration=aws.s3.BucketObjectLockConfigurationArgs(
        object_lock_enabled=True
    )
)

# Apply the bucket object lock configuration.
# Note: The bucket must have Object Lock enabled to add this configuration.
# Documentation: https://www.pulumi.com/registry/packages/aws/api-docs/s3/bucketobjectlockconfigurationv2/
object_lock_config = aws.s3.BucketObjectLockConfigurationV2("aiDatasetBucketObjectLockConfig",
    bucket=bucket.id,
    rule=aws.s3.BucketObjectLockConfigurationV2RuleArgs(
        default_retention=aws.s3.BucketObjectLockConfigurationV2RuleDefaultRetentionArgs(
            mode="COMPLIANCE",
            days=90
        )
    )
)

# Export the bucket name and ARN for easy access
pulumi.export('bucket_name', bucket.bucket)
pulumi.export('bucket_arn', bucket.arn)
```

This program will create an S3 bucket with a default retention configuration to ensure that objects within the bucket are stored immutably for 90 days in compliance mode. Compliance mode restricts any deletion or modification of the object before the retention period ends. It's worth mentioning that once the Object Lock is applied, the mode and the retention period for the lock cannot be changed if the lock mode is set to `COMPLIANCE`.

The `versioning` enabled in the bucket is also a good data protection practice. It keeps multiple, versioned copies of an object in one bucket, which can protect against accidental overwrites or deletions.

After deploying this Pulumi program, the S3 bucket will safely store AI datasets, ensuring that they are not altered, providing a stable foundation for any sensitive or regulated workloads.