How do I persist Databricks Delta tables on S3?

In this example, we’ll show how to persist Databricks Delta Tables on Amazon S3. This involves setting up an S3 bucket and configuring necessary permissions so that Databricks can read from and write to objects in the bucket. Additionally, we’ll include stack exports to verify the configuration.

We begin by defining the S3 bucket where the Delta Tables will be stored. We then configure an IAM role and attach policies to allow Databricks to interact with the S3 bucket securely.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an S3 bucket for storing Delta Tables
const databricksDeltaTableStorage = new aws.s3.BucketV2("databricks_delta_table_storage", {
    bucket: "databricks-delta-tables",
    acl: "private",
    tags: {
        Name: "Databricks Delta Table Storage",
        Environment: "Production",
    },
});
// Create an IAM role for Databricks
const databricksRole = new aws.iam.Role("databricks_role", {
    name: "databricks-role",
    assumeRolePolicy: JSON.stringify({
        Statement: [{
            Action: "sts:AssumeRole",
            Effect: "Allow",
            Principal: {
                Service: "databricks.amazonaws.com",
            },
        }],
        Version: "2012-10-17",
    }),
});
// Attach a policy to the role to allow access to the S3 bucket
const databricksPolicy = new aws.iam.RolePolicy("databricks_policy", {
    name: "databricks-policy",
    role: databricksRole.id,
    policy: pulumi.jsonStringify({
        Statement: [{
            Action: [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
            ],
            Effect: "Allow",
            Resource: [pulumi.interpolate`${databricksDeltaTableStorage.arn}/*`],
        }],
        Version: "2012-10-17",
    }),
});
export const bucketName = databricksDeltaTableStorage.bucket;
export const databricksRoleArn = databricksRole.arn;

In summary, we’ve configured an S3 bucket and associated IAM role with a policy allowing necessary permissions for Databricks to manage Delta Tables. We also output the bucket name and IAM role ARN for further use.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.