Centralized Logging with Databricks Workspace Logs.

Question

Pulumi · Accepted Answer

To achieve centralized logging with Databricks Workspace logs, we'll use Pulumi to provision the necessary infrastructure components. This involves setting up a Databricks Workspace and then configuring log delivery to a centralized location for analysis and monitoring.

In this Pulumi program, we are going to use the following resources:

- `databricks.MwsWorkspaces`: This will create a Databricks Workspace where the logs will be generated. Within this workspace, jobs and queries will run, which will produce logs.

- `databricks.MwsLogDelivery`: We will set up log delivery using this resource to send generated logs to a specified location, such as an S3 bucket or an Azure Blob storage, for centralized logging.

- `aws.s3.Bucket`: In this example, we'll store our logs in an AWS S3 bucket. S3 provides a durable, secure, and scalable object storage which makes it suitable for log storage.

- `databricks.MwsCredentials`: These credentials will be used to securely access the AWS resources from Databricks.

Please note that the program assumes that you have appropriate AWS credentials configured in your environment to allow Pulumi to manage AWS resources.

Here is the Pulumi program written in Python to set up centralized logging with Databricks Workspace logs:

```python
import pulumi
import pulumi_aws as aws
import pulumi_databricks as databricks

# AWS S3 bucket where the logs will be stored.
log_bucket = aws.s3.Bucket("databricks-logs-bucket")

# Databricks workspace where the logs will be generated.
databricks_workspace = databricks.MwsWorkspaces("logging-workspace",
    # You need to fill these with appropriate values for your account.
    accountId="your-databricks-account-id",
    workspaceName="central-logging-workspace",
    cloud="aws",  # Specify the cloud provider where the workspace exists.
    pricingTier="premium",  # Choose a pricing tier that fits your usage.
    awsRegion="us-west-2",  # Select the AWS region for the workspace.
)

# Databricks credentials to access AWS resources. This relies on having an IAM role configured for this purpose.
databricks_credentials = databricks.MwsCredentials("credentials",
    roleArn="arn:aws:iam::123456789012:role/DatabricksLogAccessRole",
    accountId=databricks_workspace.accountId,
    credentialsName="databricks-log-credentials",
)

# Log delivery setup to ship databricks workspace logs to S3 bucket.
log_delivery = databricks.MwsLogDelivery("log-delivery",
    accountId=databricks_workspace.accountId,
    workspaceIdsFilters=[databricks_workspace.workspaceId],
    credentialsId=databricks_credentials.credentialsId,
    configId="send-logs-to-s3",
    configName="Ship Databricks Logs to S3",
    logType="WORKSPACE",  # Define the type of logs to export.
    storageConfigurationId=log_bucket.id,  # Reference to the S3 bucket for log storage.
    outputFormat="JSON",  # Choose the format for the exported logs.
    deliveryPathPrefix="databricks/logs",  # The specific path prefix where logs will be stored within the S3 bucket.
    status="ENABLED",  # Enable the log delivery.
)

# Export bucket name and the Databricks Workspace URL for easy access.
pulumi.export("log_bucket_name", log_bucket.id)
pulumi.export("databricks_workspace_url", databricks_workspace.workspaceUrl)
```

Make sure to replace the placeholders like `your-databricks-account-id` and the role ARN `arn:aws:iam::123456789012:role/DatabricksLogAccessRole` with actual values from your setup.

This Pulumi program will orchestrate these resources in the cloud, allowing you to establish a centralized logging mechanism for your Databricks Workspace. The logs are delivered to your S3 bucket in an organized manner, which can then be used for analyzing the logs with various monitoring tools or for compliance and auditing purposes.