Distributed Model Inference Eventing with Azure Event Hubs

Question

Pulumi · Accepted Answer

To set up a distributed model inference eventing system with Azure Event Hubs, you would need to create an Azure Event Hubs namespace and an Event Hub within that namespace. Azure Event Hubs is a scalable data streaming platform and event ingestion service, capable of receiving and processing millions of events per second. This service will enable you to capture event data that is produced from distributed sources and make it available for processing in real-time or batched retrieval.

The Event Hubs can be visualized as "channels" that can collect data from various sources. Each Event Hub has partitions that allow event data to be distributed across multiple nodes, providing scalability to the event processing system.

Here's a Pulumi program in Python that sets up a simple Event Hubs namespace and Event Hub:

```python
import pulumi
from pulumi_azure_native import eventhub, resources

# Create an Azure Resource Group
resource_group = resources.ResourceGroup('rg')

# Create an Event Hubs Namespace
event_hub_namespace = eventhub.EventHubNamespace('namespace',
    resource_group_name=resource_group.name,
    location=resource_group.location,
    sku=eventhub.SkuArgs(
        name=eventhub.SkuName.STANDARD,
        tier='Standard'
    )
)

# Create an Event Hub inside the namespace
event_hub = eventhub.EventHub('hub',
    resource_group_name=resource_group.name,
    namespace_name=event_hub_namespace.name,
    partition_count=2,
    message_retention_in_days=1
)

# Export the primary connection string for the Event Hub Namespace, which could be used by the client applications
primary_connection_string = pulumi.Output.all(event_hub_namespace.name, resource_group.name).apply(
    lambda args: eventhub.list_namespace_keys(
        event_hub_namespace_name=args[0],
        resource_group_name=args[1],
        authorization_rule_name='RootManageSharedAccessKey'
    ).primary_connection_string
)

pulumi.export('primaryConnectionString', primary_connection_string)
```

In this program:

- We start by creating a Resource Group to contain our Event Hubs infrastructure using the `ResourceGroup` class from the `pulumi_azure_native.resources` module.
- Next, we define an Event Hubs Namespace using `EventHubNamespace` from `pulumi_azure_native.eventhub`. This acts as a container for all Event Hubs and provides a scope for common settings, like network controls. We choose the standard SKU, which should suffice for typical use cases and supports features like Auto-inflate.
- Inside the namespace, we create an Event Hub through the `EventHub` class. We specify the number of partitions and the message retention period (in days). Partitions are a scaling mechanism in event hubs, and message retention dictates how long messages remain in the event hub.
- Lastly, we use `pulumi.Output.all()` in combination with `.apply()` to register an inline function that retrieves the namespace's primary connection string. This connection string will be used by client applications to send and receive messages from the Event Hub.

The connection string is exported as an output, making it easy to retrieve with the Pulumi CLI after deployment or to be passed to other services perhaps as a secret for secure communications.

Refer to the [Azure Event Hubs Namespace documentation](https://www.pulumi.com/registry/packages/azure-native/api-docs/eventhub/eventhubnamespace/) to learn more about Event Hubs Namespaces, and to the [Azure Event Hub documentation](https://www.pulumi.com/registry/packages/azure-native/api-docs/eventhub/eventhub/) for details on Event Hubs.