The azure-native:kusto:EventGridDataConnection resource, part of the Pulumi Azure Native provider, connects Azure Data Explorer (Kusto) databases to blob storage events via Event Grid, routing new files through Event Hub for ingestion. This guide focuses on one capability: blob storage event ingestion.
Data connections belong to a Kusto database and reference Event Hub, storage accounts, Event Grid subscriptions, and managed identities that must exist separately. The example is intentionally small. Combine it with your own Kusto cluster, Event Hub infrastructure, and storage configuration.
Stream blob storage events into Kusto tables
Analytics pipelines often need to ingest data files as they arrive in blob storage. Event Grid data connections watch for blob creation events and automatically route new files into Kusto tables.
import * as pulumi from "@pulumi/pulumi";
import * as azure_native from "@pulumi/azure-native";
const eventGridDataConnection = new azure_native.kusto.EventGridDataConnection("eventGridDataConnection", {
blobStorageEventType: azure_native.kusto.BlobStorageEventType.Microsoft_Storage_BlobCreated,
clusterName: "kustoCluster",
consumerGroup: "$Default",
dataConnectionName: "dataConnectionTest",
dataFormat: azure_native.kusto.EventGridDataFormat.MULTIJSON,
databaseName: "KustoDatabase8",
databaseRouting: azure_native.kusto.DatabaseRouting.Single,
eventGridResourceId: "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount/providers/Microsoft.EventGrid/eventSubscriptions/eventSubscriptionTest",
eventHubResourceId: "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.EventHub/namespaces/eventhubTestns1/eventhubs/eventhubTest2",
ignoreFirstRecord: false,
kind: "EventGrid",
location: "westus",
managedIdentityResourceId: "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.ManagedIdentity/userAssignedIdentities/managedidentityTest1",
mappingRuleName: "TestMapping",
resourceGroupName: "kustorptest",
storageAccountResourceId: "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount",
tableName: "TestTable",
});
import pulumi
import pulumi_azure_native as azure_native
event_grid_data_connection = azure_native.kusto.EventGridDataConnection("eventGridDataConnection",
blob_storage_event_type=azure_native.kusto.BlobStorageEventType.MICROSOFT_STORAGE_BLOB_CREATED,
cluster_name="kustoCluster",
consumer_group="$Default",
data_connection_name="dataConnectionTest",
data_format=azure_native.kusto.EventGridDataFormat.MULTIJSON,
database_name="KustoDatabase8",
database_routing=azure_native.kusto.DatabaseRouting.SINGLE,
event_grid_resource_id="/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount/providers/Microsoft.EventGrid/eventSubscriptions/eventSubscriptionTest",
event_hub_resource_id="/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.EventHub/namespaces/eventhubTestns1/eventhubs/eventhubTest2",
ignore_first_record=False,
kind="EventGrid",
location="westus",
managed_identity_resource_id="/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.ManagedIdentity/userAssignedIdentities/managedidentityTest1",
mapping_rule_name="TestMapping",
resource_group_name="kustorptest",
storage_account_resource_id="/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount",
table_name="TestTable")
package main
import (
kusto "github.com/pulumi/pulumi-azure-native-sdk/kusto/v3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
_, err := kusto.NewEventGridDataConnection(ctx, "eventGridDataConnection", &kusto.EventGridDataConnectionArgs{
BlobStorageEventType: pulumi.String(kusto.BlobStorageEventType_Microsoft_Storage_BlobCreated),
ClusterName: pulumi.String("kustoCluster"),
ConsumerGroup: pulumi.String("$Default"),
DataConnectionName: pulumi.String("dataConnectionTest"),
DataFormat: pulumi.String(kusto.EventGridDataFormatMULTIJSON),
DatabaseName: pulumi.String("KustoDatabase8"),
DatabaseRouting: pulumi.String(kusto.DatabaseRoutingSingle),
EventGridResourceId: pulumi.String("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount/providers/Microsoft.EventGrid/eventSubscriptions/eventSubscriptionTest"),
EventHubResourceId: pulumi.String("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.EventHub/namespaces/eventhubTestns1/eventhubs/eventhubTest2"),
IgnoreFirstRecord: pulumi.Bool(false),
Kind: pulumi.String("EventGrid"),
Location: pulumi.String("westus"),
ManagedIdentityResourceId: pulumi.String("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.ManagedIdentity/userAssignedIdentities/managedidentityTest1"),
MappingRuleName: pulumi.String("TestMapping"),
ResourceGroupName: pulumi.String("kustorptest"),
StorageAccountResourceId: pulumi.String("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount"),
TableName: pulumi.String("TestTable"),
})
if err != nil {
return err
}
return nil
})
}
using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AzureNative = Pulumi.AzureNative;
return await Deployment.RunAsync(() =>
{
var eventGridDataConnection = new AzureNative.Kusto.EventGridDataConnection("eventGridDataConnection", new()
{
BlobStorageEventType = AzureNative.Kusto.BlobStorageEventType.Microsoft_Storage_BlobCreated,
ClusterName = "kustoCluster",
ConsumerGroup = "$Default",
DataConnectionName = "dataConnectionTest",
DataFormat = AzureNative.Kusto.EventGridDataFormat.MULTIJSON,
DatabaseName = "KustoDatabase8",
DatabaseRouting = AzureNative.Kusto.DatabaseRouting.Single,
EventGridResourceId = "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount/providers/Microsoft.EventGrid/eventSubscriptions/eventSubscriptionTest",
EventHubResourceId = "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.EventHub/namespaces/eventhubTestns1/eventhubs/eventhubTest2",
IgnoreFirstRecord = false,
Kind = "EventGrid",
Location = "westus",
ManagedIdentityResourceId = "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.ManagedIdentity/userAssignedIdentities/managedidentityTest1",
MappingRuleName = "TestMapping",
ResourceGroupName = "kustorptest",
StorageAccountResourceId = "/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount",
TableName = "TestTable",
});
});
package generated_program;
import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.azurenative.kusto.EventGridDataConnection;
import com.pulumi.azurenative.kusto.EventGridDataConnectionArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;
public class App {
public static void main(String[] args) {
Pulumi.run(App::stack);
}
public static void stack(Context ctx) {
var eventGridDataConnection = new EventGridDataConnection("eventGridDataConnection", EventGridDataConnectionArgs.builder()
.blobStorageEventType("Microsoft.Storage.BlobCreated")
.clusterName("kustoCluster")
.consumerGroup("$Default")
.dataConnectionName("dataConnectionTest")
.dataFormat("MULTIJSON")
.databaseName("KustoDatabase8")
.databaseRouting("Single")
.eventGridResourceId("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount/providers/Microsoft.EventGrid/eventSubscriptions/eventSubscriptionTest")
.eventHubResourceId("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.EventHub/namespaces/eventhubTestns1/eventhubs/eventhubTest2")
.ignoreFirstRecord(false)
.kind("EventGrid")
.location("westus")
.managedIdentityResourceId("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.ManagedIdentity/userAssignedIdentities/managedidentityTest1")
.mappingRuleName("TestMapping")
.resourceGroupName("kustorptest")
.storageAccountResourceId("/subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount")
.tableName("TestTable")
.build());
}
}
resources:
eventGridDataConnection:
type: azure-native:kusto:EventGridDataConnection
properties:
blobStorageEventType: Microsoft.Storage.BlobCreated
clusterName: kustoCluster
consumerGroup: $Default
dataConnectionName: dataConnectionTest
dataFormat: MULTIJSON
databaseName: KustoDatabase8
databaseRouting: Single
eventGridResourceId: /subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount/providers/Microsoft.EventGrid/eventSubscriptions/eventSubscriptionTest
eventHubResourceId: /subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.EventHub/namespaces/eventhubTestns1/eventhubs/eventhubTest2
ignoreFirstRecord: false
kind: EventGrid
location: westus
managedIdentityResourceId: /subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.ManagedIdentity/userAssignedIdentities/managedidentityTest1
mappingRuleName: TestMapping
resourceGroupName: kustorptest
storageAccountResourceId: /subscriptions/12345678-1234-1234-1234-123456789098/resourceGroups/kustorptest/providers/Microsoft.Storage/storageAccounts/teststorageaccount
tableName: TestTable
When a blob is created in the storage account, Event Grid publishes an event to the Event Hub specified by eventHubResourceId. The data connection reads from the consumerGroup, applies the dataFormat and mappingRuleName to transform the data, and writes it to tableName in your Kusto database. The blobStorageEventType property filters which storage events trigger ingestion (here, Microsoft.Storage.BlobCreated). The managedIdentityResourceId authenticates access to both Event Hub and storage account.
Beyond these examples
This snippet focuses on blob storage event ingestion: Event Hub routing and consumer groups, and data format and mapping configuration. It’s intentionally minimal rather than a full data ingestion pipeline.
The example references pre-existing infrastructure such as Kusto cluster and database, Event Hub namespace and hub, storage account with Event Grid subscription, and managed identity for authentication. It focuses on configuring the data connection rather than provisioning the surrounding infrastructure.
To keep things focused, common data connection patterns are omitted, including:
- Database routing options (databaseRouting property)
- Header row handling (ignoreFirstRecord)
- Alternative blob storage event types
- Event Grid resource ID configuration
These omissions are intentional: the goal is to illustrate how the data connection is wired, not provide drop-in ingestion modules. See the EventGridDataConnection resource reference for all available configuration options.
Let's configure Azure Kusto Event Grid Data Connections
Get started with Pulumi Cloud, then follow our quick setup guide to deploy this infrastructure.
Try Pulumi Cloud for FREEFrequently Asked Questions
Configuration & Setup
What properties can't I change after creating the data connection?
The
clusterName, dataConnectionName, databaseName, and resourceGroupName properties are immutable and require resource replacement if changed.What's the minimum configuration needed to create an Event Grid data connection?
You need
clusterName, dataConnectionName, databaseName, and resourceGroupName. The examples show this minimal configuration.What are the required properties for a fully configured Event Grid data connection?
Beyond the basic properties, you must specify
consumerGroup, eventHubResourceId, kind (set to “EventGrid”), and storageAccountResourceId.Data Ingestion & Routing
What does the databaseRouting property control?
databaseRouting indicates how database routing information is handled from the data connection. It defaults to Single, meaning only database routing information is allowed.Can I specify data format and mapping per message instead of at the connection level?
Yes,
dataFormat, mappingRuleName, and tableName are optional at the connection level. You can add this information to each message instead.How do I skip header rows when ingesting files?
Set
ignoreFirstRecord to true to skip the first record of every file during ingestion.Authentication & Identity
How do I authenticate the data connection with Event Hub and Storage Account?
Use
managedIdentityResourceId to specify a managed identity (system or user assigned) for authentication with both Event Hub and Storage Account.Event Processing
What blob storage events can I process with this data connection?
Use
blobStorageEventType to specify the blob storage event type to process, such as Microsoft.Storage.BlobCreated.What's the difference between eventGridResourceId and eventHubResourceId?
eventGridResourceId is the resource ID of the Event Grid subscription to storage account events, while eventHubResourceId is where Event Grid sends those events.What consumer group should I use for the Event Hub?
Specify the Event Hub consumer group using
consumerGroup. The full example shows $Default as a common choice.