Deploy AWS EMR Serverless Applications

The aws:emrserverless/application:Application resource, part of the Pulumi AWS provider, defines an EMR Serverless application: the engine type, release version, capacity limits, and runtime settings that jobs execute against. This guide focuses on four capabilities: application creation with engine selection, capacity pre-allocation and limits, observability configuration, and runtime property tuning.

EMR Serverless applications are standalone resources but may reference CloudWatch log groups or Prometheus endpoints for monitoring. The examples are intentionally small. Combine them with your own job submission logic and monitoring infrastructure.

Create a minimal application for Hive workloads

Most deployments start with a minimal application definition that specifies the engine type and EMR release version, creating the container that jobs run against.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.emrserverless.Application("example", {
    name: "example",
    releaseLabel: "emr-6.6.0",
    type: "hive",
});

import pulumi
import pulumi_aws as aws

example = aws.emrserverless.Application("example",
    name="example",
    release_label="emr-6.6.0",
    type="hive")

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/emrserverless"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := emrserverless.NewApplication(ctx, "example", &emrserverless.ApplicationArgs{
			Name:         pulumi.String("example"),
			ReleaseLabel: pulumi.String("emr-6.6.0"),
			Type:         pulumi.String("hive"),
		})
		if err != nil {
			return err
		}
		return nil
	})
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;

return await Deployment.RunAsync(() => 
{
    var example = new Aws.EmrServerless.Application("example", new()
    {
        Name = "example",
        ReleaseLabel = "emr-6.6.0",
        Type = "hive",
    });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.emrserverless.Application;
import com.pulumi.aws.emrserverless.ApplicationArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var example = new Application("example", ApplicationArgs.builder()
            .name("example")
            .releaseLabel("emr-6.6.0")
            .type("hive")
            .build());

    }
}

resources:
  example:
    type: aws:emrserverless:Application
    properties:
      name: example
      releaseLabel: emr-6.6.0
      type: hive

The type property selects the engine (spark or hive). The releaseLabel specifies the EMR version, which determines available features and engine versions. The name identifies the application for job submission. Without explicit capacity configuration, EMR Serverless uses default allocation behavior.

Pre-warm workers with initial capacity

Applications that need predictable startup times can pre-allocate workers before jobs arrive, reducing cold-start latency.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.emrserverless.Application("example", {
    name: "example",
    releaseLabel: "emr-6.6.0",
    type: "hive",
    initialCapacities: [{
        initialCapacityType: "HiveDriver",
        initialCapacityConfig: {
            workerCount: 1,
            workerConfiguration: {
                cpu: "2 vCPU",
                memory: "10 GB",
            },
        },
    }],
});

import pulumi
import pulumi_aws as aws

example = aws.emrserverless.Application("example",
    name="example",
    release_label="emr-6.6.0",
    type="hive",
    initial_capacities=[{
        "initial_capacity_type": "HiveDriver",
        "initial_capacity_config": {
            "worker_count": 1,
            "worker_configuration": {
                "cpu": "2 vCPU",
                "memory": "10 GB",
            },
        },
    }])

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/emrserverless"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := emrserverless.NewApplication(ctx, "example", &emrserverless.ApplicationArgs{
			Name:         pulumi.String("example"),
			ReleaseLabel: pulumi.String("emr-6.6.0"),
			Type:         pulumi.String("hive"),
			InitialCapacities: emrserverless.ApplicationInitialCapacityArray{
				&emrserverless.ApplicationInitialCapacityArgs{
					InitialCapacityType: pulumi.String("HiveDriver"),
					InitialCapacityConfig: &emrserverless.ApplicationInitialCapacityInitialCapacityConfigArgs{
						WorkerCount: pulumi.Int(1),
						WorkerConfiguration: &emrserverless.ApplicationInitialCapacityInitialCapacityConfigWorkerConfigurationArgs{
							Cpu:    pulumi.String("2 vCPU"),
							Memory: pulumi.String("10 GB"),
						},
					},
				},
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;

return await Deployment.RunAsync(() => 
{
    var example = new Aws.EmrServerless.Application("example", new()
    {
        Name = "example",
        ReleaseLabel = "emr-6.6.0",
        Type = "hive",
        InitialCapacities = new[]
        {
            new Aws.EmrServerless.Inputs.ApplicationInitialCapacityArgs
            {
                InitialCapacityType = "HiveDriver",
                InitialCapacityConfig = new Aws.EmrServerless.Inputs.ApplicationInitialCapacityInitialCapacityConfigArgs
                {
                    WorkerCount = 1,
                    WorkerConfiguration = new Aws.EmrServerless.Inputs.ApplicationInitialCapacityInitialCapacityConfigWorkerConfigurationArgs
                    {
                        Cpu = "2 vCPU",
                        Memory = "10 GB",
                    },
                },
            },
        },
    });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.emrserverless.Application;
import com.pulumi.aws.emrserverless.ApplicationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationInitialCapacityArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationInitialCapacityInitialCapacityConfigArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationInitialCapacityInitialCapacityConfigWorkerConfigurationArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var example = new Application("example", ApplicationArgs.builder()
            .name("example")
            .releaseLabel("emr-6.6.0")
            .type("hive")
            .initialCapacities(ApplicationInitialCapacityArgs.builder()
                .initialCapacityType("HiveDriver")
                .initialCapacityConfig(ApplicationInitialCapacityInitialCapacityConfigArgs.builder()
                    .workerCount(1)
                    .workerConfiguration(ApplicationInitialCapacityInitialCapacityConfigWorkerConfigurationArgs.builder()
                        .cpu("2 vCPU")
                        .memory("10 GB")
                        .build())
                    .build())
                .build())
            .build());

    }
}

resources:
  example:
    type: aws:emrserverless:Application
    properties:
      name: example
      releaseLabel: emr-6.6.0
      type: hive
      initialCapacities:
        - initialCapacityType: HiveDriver
          initialCapacityConfig:
            workerCount: 1
            workerConfiguration:
              cpu: 2 vCPU
              memory: 10 GB

The initialCapacities array defines worker pools by type (HiveDriver, SparkDriver, etc.). Each pool specifies workerCount and workerConfiguration with CPU and memory per worker. EMR Serverless maintains this capacity even when idle, keeping workers warm for immediate job execution.

Set resource limits with maximum capacity

To control costs and prevent runaway consumption, applications can define hard limits on total CPU and memory across all workers.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.emrserverless.Application("example", {
    name: "example",
    releaseLabel: "emr-6.6.0",
    type: "hive",
    maximumCapacity: {
        cpu: "2 vCPU",
        memory: "10 GB",
    },
});

import pulumi
import pulumi_aws as aws

example = aws.emrserverless.Application("example",
    name="example",
    release_label="emr-6.6.0",
    type="hive",
    maximum_capacity={
        "cpu": "2 vCPU",
        "memory": "10 GB",
    })

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/emrserverless"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := emrserverless.NewApplication(ctx, "example", &emrserverless.ApplicationArgs{
			Name:         pulumi.String("example"),
			ReleaseLabel: pulumi.String("emr-6.6.0"),
			Type:         pulumi.String("hive"),
			MaximumCapacity: &emrserverless.ApplicationMaximumCapacityArgs{
				Cpu:    pulumi.String("2 vCPU"),
				Memory: pulumi.String("10 GB"),
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;

return await Deployment.RunAsync(() => 
{
    var example = new Aws.EmrServerless.Application("example", new()
    {
        Name = "example",
        ReleaseLabel = "emr-6.6.0",
        Type = "hive",
        MaximumCapacity = new Aws.EmrServerless.Inputs.ApplicationMaximumCapacityArgs
        {
            Cpu = "2 vCPU",
            Memory = "10 GB",
        },
    });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.emrserverless.Application;
import com.pulumi.aws.emrserverless.ApplicationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationMaximumCapacityArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var example = new Application("example", ApplicationArgs.builder()
            .name("example")
            .releaseLabel("emr-6.6.0")
            .type("hive")
            .maximumCapacity(ApplicationMaximumCapacityArgs.builder()
                .cpu("2 vCPU")
                .memory("10 GB")
                .build())
            .build());

    }
}

resources:
  example:
    type: aws:emrserverless:Application
    properties:
      name: example
      releaseLabel: emr-6.6.0
      type: hive
      maximumCapacity:
        cpu: 2 vCPU
        memory: 10 GB

The maximumCapacity property sets cumulative limits across all workers at any point in time. When either CPU or memory reaches the limit, EMR Serverless stops allocating new resources. This caps costs but may cause jobs to queue if demand exceeds capacity.

Stream logs and metrics to CloudWatch and Prometheus

Production applications need observability into job execution. EMR Serverless streams driver and executor logs to CloudWatch, persists metrics to S3, and exports to Prometheus.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.emrserverless.Application("example", {
    name: "example",
    releaseLabel: "emr-7.1.0",
    type: "spark",
    monitoringConfiguration: {
        cloudwatchLoggingConfiguration: {
            enabled: true,
            logGroupName: "/aws/emr-serverless/example",
            logStreamNamePrefix: "spark-logs",
            logTypes: [
                {
                    name: "SPARK_DRIVER",
                    values: [
                        "STDOUT",
                        "STDERR",
                    ],
                },
                {
                    name: "SPARK_EXECUTOR",
                    values: ["STDOUT"],
                },
            ],
        },
        managedPersistenceMonitoringConfiguration: {
            enabled: true,
        },
        prometheusMonitoringConfiguration: {
            remoteWriteUrl: "https://prometheus-remote-write-endpoint.example.com/api/v1/write",
        },
    },
});

import pulumi
import pulumi_aws as aws

example = aws.emrserverless.Application("example",
    name="example",
    release_label="emr-7.1.0",
    type="spark",
    monitoring_configuration={
        "cloudwatch_logging_configuration": {
            "enabled": True,
            "log_group_name": "/aws/emr-serverless/example",
            "log_stream_name_prefix": "spark-logs",
            "log_types": [
                {
                    "name": "SPARK_DRIVER",
                    "values": [
                        "STDOUT",
                        "STDERR",
                    ],
                },
                {
                    "name": "SPARK_EXECUTOR",
                    "values": ["STDOUT"],
                },
            ],
        },
        "managed_persistence_monitoring_configuration": {
            "enabled": True,
        },
        "prometheus_monitoring_configuration": {
            "remote_write_url": "https://prometheus-remote-write-endpoint.example.com/api/v1/write",
        },
    })

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/emrserverless"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := emrserverless.NewApplication(ctx, "example", &emrserverless.ApplicationArgs{
			Name:         pulumi.String("example"),
			ReleaseLabel: pulumi.String("emr-7.1.0"),
			Type:         pulumi.String("spark"),
			MonitoringConfiguration: &emrserverless.ApplicationMonitoringConfigurationArgs{
				CloudwatchLoggingConfiguration: &emrserverless.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationArgs{
					Enabled:             pulumi.Bool(true),
					LogGroupName:        pulumi.String("/aws/emr-serverless/example"),
					LogStreamNamePrefix: pulumi.String("spark-logs"),
					LogTypes: emrserverless.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArray{
						&emrserverless.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArgs{
							Name: pulumi.String("SPARK_DRIVER"),
							Values: pulumi.StringArray{
								pulumi.String("STDOUT"),
								pulumi.String("STDERR"),
							},
						},
						&emrserverless.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArgs{
							Name: pulumi.String("SPARK_EXECUTOR"),
							Values: pulumi.StringArray{
								pulumi.String("STDOUT"),
							},
						},
					},
				},
				ManagedPersistenceMonitoringConfiguration: &emrserverless.ApplicationMonitoringConfigurationManagedPersistenceMonitoringConfigurationArgs{
					Enabled: pulumi.Bool(true),
				},
				PrometheusMonitoringConfiguration: &emrserverless.ApplicationMonitoringConfigurationPrometheusMonitoringConfigurationArgs{
					RemoteWriteUrl: pulumi.String("https://prometheus-remote-write-endpoint.example.com/api/v1/write"),
				},
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;

return await Deployment.RunAsync(() => 
{
    var example = new Aws.EmrServerless.Application("example", new()
    {
        Name = "example",
        ReleaseLabel = "emr-7.1.0",
        Type = "spark",
        MonitoringConfiguration = new Aws.EmrServerless.Inputs.ApplicationMonitoringConfigurationArgs
        {
            CloudwatchLoggingConfiguration = new Aws.EmrServerless.Inputs.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationArgs
            {
                Enabled = true,
                LogGroupName = "/aws/emr-serverless/example",
                LogStreamNamePrefix = "spark-logs",
                LogTypes = new[]
                {
                    new Aws.EmrServerless.Inputs.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArgs
                    {
                        Name = "SPARK_DRIVER",
                        Values = new[]
                        {
                            "STDOUT",
                            "STDERR",
                        },
                    },
                    new Aws.EmrServerless.Inputs.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArgs
                    {
                        Name = "SPARK_EXECUTOR",
                        Values = new[]
                        {
                            "STDOUT",
                        },
                    },
                },
            },
            ManagedPersistenceMonitoringConfiguration = new Aws.EmrServerless.Inputs.ApplicationMonitoringConfigurationManagedPersistenceMonitoringConfigurationArgs
            {
                Enabled = true,
            },
            PrometheusMonitoringConfiguration = new Aws.EmrServerless.Inputs.ApplicationMonitoringConfigurationPrometheusMonitoringConfigurationArgs
            {
                RemoteWriteUrl = "https://prometheus-remote-write-endpoint.example.com/api/v1/write",
            },
        },
    });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.emrserverless.Application;
import com.pulumi.aws.emrserverless.ApplicationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationMonitoringConfigurationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationMonitoringConfigurationManagedPersistenceMonitoringConfigurationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationMonitoringConfigurationPrometheusMonitoringConfigurationArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var example = new Application("example", ApplicationArgs.builder()
            .name("example")
            .releaseLabel("emr-7.1.0")
            .type("spark")
            .monitoringConfiguration(ApplicationMonitoringConfigurationArgs.builder()
                .cloudwatchLoggingConfiguration(ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationArgs.builder()
                    .enabled(true)
                    .logGroupName("/aws/emr-serverless/example")
                    .logStreamNamePrefix("spark-logs")
                    .logTypes(                    
                        ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArgs.builder()
                            .name("SPARK_DRIVER")
                            .values(                            
                                "STDOUT",
                                "STDERR")
                            .build(),
                        ApplicationMonitoringConfigurationCloudwatchLoggingConfigurationLogTypeArgs.builder()
                            .name("SPARK_EXECUTOR")
                            .values("STDOUT")
                            .build())
                    .build())
                .managedPersistenceMonitoringConfiguration(ApplicationMonitoringConfigurationManagedPersistenceMonitoringConfigurationArgs.builder()
                    .enabled(true)
                    .build())
                .prometheusMonitoringConfiguration(ApplicationMonitoringConfigurationPrometheusMonitoringConfigurationArgs.builder()
                    .remoteWriteUrl("https://prometheus-remote-write-endpoint.example.com/api/v1/write")
                    .build())
                .build())
            .build());

    }
}

resources:
  example:
    type: aws:emrserverless:Application
    properties:
      name: example
      releaseLabel: emr-7.1.0
      type: spark
      monitoringConfiguration:
        cloudwatchLoggingConfiguration:
          enabled: true
          logGroupName: /aws/emr-serverless/example
          logStreamNamePrefix: spark-logs
          logTypes:
            - name: SPARK_DRIVER
              values:
                - STDOUT
                - STDERR
            - name: SPARK_EXECUTOR
              values:
                - STDOUT
        managedPersistenceMonitoringConfiguration:
          enabled: true
        prometheusMonitoringConfiguration:
          remoteWriteUrl: https://prometheus-remote-write-endpoint.example.com/api/v1/write

The monitoringConfiguration property controls three observability channels. The cloudwatchLoggingConfiguration sends logs to a CloudWatch log group, with logTypes filtering by component (SPARK_DRIVER, SPARK_EXECUTOR) and stream (STDOUT, STDERR). The managedPersistenceMonitoringConfiguration persists metrics to S3. The prometheusMonitoringConfiguration exports metrics to a remote write endpoint.

Tune Spark settings with runtime configurations

Applications often override default Spark or Hive settings to optimize performance or adjust logging levels without rebuilding container images.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.emrserverless.Application("example", {
    name: "example",
    releaseLabel: "emr-6.8.0",
    type: "spark",
    runtimeConfigurations: [
        {
            classification: "spark-executor-log4j2",
            properties: {
                "rootLogger.level": "error",
                "logger.IdentifierForClass.name": "classpathForSettingLogger",
                "logger.IdentifierForClass.level": "info",
            },
        },
        {
            classification: "spark-defaults",
            properties: {
                "spark.executor.memory": "1g",
                "spark.executor.cores": "1",
            },
        },
    ],
});

import pulumi
import pulumi_aws as aws

example = aws.emrserverless.Application("example",
    name="example",
    release_label="emr-6.8.0",
    type="spark",
    runtime_configurations=[
        {
            "classification": "spark-executor-log4j2",
            "properties": {
                "rootLogger.level": "error",
                "logger.IdentifierForClass.name": "classpathForSettingLogger",
                "logger.IdentifierForClass.level": "info",
            },
        },
        {
            "classification": "spark-defaults",
            "properties": {
                "spark.executor.memory": "1g",
                "spark.executor.cores": "1",
            },
        },
    ])

package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v7/go/aws/emrserverless"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		_, err := emrserverless.NewApplication(ctx, "example", &emrserverless.ApplicationArgs{
			Name:         pulumi.String("example"),
			ReleaseLabel: pulumi.String("emr-6.8.0"),
			Type:         pulumi.String("spark"),
			RuntimeConfigurations: emrserverless.ApplicationRuntimeConfigurationArray{
				&emrserverless.ApplicationRuntimeConfigurationArgs{
					Classification: pulumi.String("spark-executor-log4j2"),
					Properties: pulumi.StringMap{
						"rootLogger.level":                pulumi.String("error"),
						"logger.IdentifierForClass.name":  pulumi.String("classpathForSettingLogger"),
						"logger.IdentifierForClass.level": pulumi.String("info"),
					},
				},
				&emrserverless.ApplicationRuntimeConfigurationArgs{
					Classification: pulumi.String("spark-defaults"),
					Properties: pulumi.StringMap{
						"spark.executor.memory": pulumi.String("1g"),
						"spark.executor.cores":  pulumi.String("1"),
					},
				},
			},
		})
		if err != nil {
			return err
		}
		return nil
	})
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using Aws = Pulumi.Aws;

return await Deployment.RunAsync(() => 
{
    var example = new Aws.EmrServerless.Application("example", new()
    {
        Name = "example",
        ReleaseLabel = "emr-6.8.0",
        Type = "spark",
        RuntimeConfigurations = new[]
        {
            new Aws.EmrServerless.Inputs.ApplicationRuntimeConfigurationArgs
            {
                Classification = "spark-executor-log4j2",
                Properties = 
                {
                    { "rootLogger.level", "error" },
                    { "logger.IdentifierForClass.name", "classpathForSettingLogger" },
                    { "logger.IdentifierForClass.level", "info" },
                },
            },
            new Aws.EmrServerless.Inputs.ApplicationRuntimeConfigurationArgs
            {
                Classification = "spark-defaults",
                Properties = 
                {
                    { "spark.executor.memory", "1g" },
                    { "spark.executor.cores", "1" },
                },
            },
        },
    });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.aws.emrserverless.Application;
import com.pulumi.aws.emrserverless.ApplicationArgs;
import com.pulumi.aws.emrserverless.inputs.ApplicationRuntimeConfigurationArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var example = new Application("example", ApplicationArgs.builder()
            .name("example")
            .releaseLabel("emr-6.8.0")
            .type("spark")
            .runtimeConfigurations(            
                ApplicationRuntimeConfigurationArgs.builder()
                    .classification("spark-executor-log4j2")
                    .properties(Map.ofEntries(
                        Map.entry("rootLogger.level", "error"),
                        Map.entry("logger.IdentifierForClass.name", "classpathForSettingLogger"),
                        Map.entry("logger.IdentifierForClass.level", "info")
                    ))
                    .build(),
                ApplicationRuntimeConfigurationArgs.builder()
                    .classification("spark-defaults")
                    .properties(Map.ofEntries(
                        Map.entry("spark.executor.memory", "1g"),
                        Map.entry("spark.executor.cores", "1")
                    ))
                    .build())
            .build());

    }
}

resources:
  example:
    type: aws:emrserverless:Application
    properties:
      name: example
      releaseLabel: emr-6.8.0
      type: spark
      runtimeConfigurations:
        - classification: spark-executor-log4j2
          properties:
            rootLogger.level: error
            logger.IdentifierForClass.name: classpathForSettingLogger
            logger.IdentifierForClass.level: info
        - classification: spark-defaults
          properties:
            spark.executor.memory: 1g
            spark.executor.cores: '1'

The runtimeConfigurations array modifies engine properties. Each entry specifies a classification (the configuration file to modify, like spark-defaults or spark-executor-log4j2) and properties (key-value pairs to set). These settings apply to all jobs submitted to the application.

Beyond these examples

These snippets focus on specific application-level features: capacity management, observability, and runtime tuning. They’re intentionally minimal rather than full EMR Serverless deployments.

The examples may reference pre-existing infrastructure such as CloudWatch log groups for logging configuration and Prometheus remote write endpoints for metrics export. They focus on configuring the application rather than provisioning everything around it.

To keep things focused, common application patterns are omitted, including:

Auto-start and auto-stop lifecycle (autoStartConfiguration, autoStopConfiguration)
VPC networking (networkConfiguration)
Custom container images (imageConfiguration)
Interactive session configuration (interactiveConfiguration)
Scheduler configuration for batch and streaming jobs

These omissions are intentional: the goal is to illustrate how each application feature is wired, not provide drop-in EMR modules. See the EMR Serverless Application resource reference for all available configuration options.

Let's deploy AWS EMR Serverless Applications

Get started with Pulumi Cloud, then follow our quick setup guide to deploy this infrastructure.

Try Pulumi Cloud for FREE

Frequently Asked Questions

Application Configuration & Immutability

What properties can't I change after creating an EMR Serverless application?

The name and type properties are immutable and cannot be changed after creation. Modifying these requires replacing the resource.

What application types does EMR Serverless support?

EMR Serverless supports spark and hive application types.

What CPU architectures are available for EMR Serverless applications?

You can choose ARM64 or X86_64 architecture. The default is X86_64.

Capacity Management

What's the difference between initialCapacities and maximumCapacity?

initialCapacities sets the capacity when the application is created, while maximumCapacity defines cumulative limits across all workers at any point in time. Once any maximum limit is hit, no new resources will be created.

How do I configure initial capacity for my application?

Use initialCapacities to specify the worker type (e.g., HiveDriver), worker count, CPU, and memory configuration.

Monitoring & Logging

What monitoring options are available for EMR Serverless applications?

You can configure CloudWatch logging (with log groups and stream prefixes), Prometheus remote write endpoints, and managed persistence monitoring through monitoringConfiguration.

Runtime & Advanced Configuration

How do I customize Spark or Hive runtime settings?

Use runtimeConfigurations with a classification (like spark-defaults or spark-executor-log4j2) and properties to configure settings such as executor memory, cores, or log levels.

What EMR release version do I need for scheduler configuration?

Scheduler configuration for batch and streaming jobs requires release label emr-7.0.0 or above.

Using a different cloud?

Explore analytics guides for other cloud providers:

Azure Guides GCP Guides