Private Connectivity to Azure Event Hub for Real-time AI

Question

Pulumi · Accepted Answer

To create a private connectivity to Azure Event Hub, which can be used for real-time AI or any other purpose, you would typically want to set up the following resources:

1. An **Azure Event Hub namespace**, which acts as a container for multiple Event Hubs.
2. An individual **Event Hub**, which is the data stream where data will be sent.
3. A **Virtual Network (VNet)** to provide private network isolation.
4. A **Private Endpoint**, which allows you to connect securely to services like Event Hub over your VNet.
5. A **Private DNS Zone** for the private link resource, to resolve the domain names of the resources inside your private network.

I'll guide you through the Pulumi Python program necessary to set up this environment. The following code will:

- Create a Resource Group to contain all resources.
- Provision an Event Hub Namespace and an Event Hub within that namespace.
- Set up a VNet and Subnet configured to host the Private Endpoint.
- Create a Network Interface and a Private Endpoint for the Event Hub.
- Establish a Private DNS Zone linked to your VNet with an A record pointing to the Private Endpoint of the Event Hub.

The program also ensures all resources are in the same location and uses the same resource group for clarity and management simplicity.

Let's get started with the Pulumi Python program:

```python
import pulumi
import pulumi_azure_native.network as network
import pulumi_azure_native.resources as resources
import pulumi_azure_native.eventhub as eventhub
import pulumi_azure_native.privatedns as dns

# Create a new resource group to contain all resources
resource_group = resources.ResourceGroup('rg')

# Create an EventHub Namespace
eventhub_namespace = eventhub.Namespace(
    'event-hub-namespace',
    resource_group_name=resource_group.name,
    sku=eventhub.SkuArgs(
        name="Standard",  # Choose between Basic and Standard tiers
    ),
    location=resource_group.location,
)

# Create an EventHub inside the namespace
event_hub = eventhub.EventHub(
    'event-hub',
    resource_group_name=resource_group.name,
    namespace_name=eventhub_namespace.name,
    partition_count=2,  # Define the number of partitions
    message_retention_in_days=1,  # Define the message retention period
)

# Create a Virtual Network
virtual_network = network.VirtualNetwork(
    'virtual-network',
    resource_group_name=resource_group.name,
    location=resource_group.location,
    address_space=network.AddressSpaceArgs(
        address_prefixes=["10.0.0.0/16"],
    ),
)

# Create a Subnet specifically for the private endpoint.
# The 'PrivateLinkService' is a required delegation
subnet_for_pe = network.Subnet(
    'subnet-for-private-endpoint',
    resource_group_name=resource_group.name,
    virtual_network_name=virtual_network.name,
    address_prefix="10.0.0.0/24",
    private_endpoint_network_policies="Disabled", # Disable network policies for the Private Endpoints.
    delegations=[network.DelegationArgs(
        name="delegation",
        service_name="Microsoft.EventHub/namespaces",
    )],
)

# Create the Private Endpoint for the Event Hub
private_endpoint = network.PrivateEndpoint(
    'private-endpoint',
    resource_group_name=resource_group.name,
    location=resource_group.location,
    subnet=network.SubnetArgs(
        id=subnet_for_pe.id,
    ),
    private_link_service_connections=[network.PrivateLinkServiceConnectionArgs(
        name="pls-connection",
        private_link_service_id=eventhub_namespace.id,
        group_ids=["namespace"],
    )],
)

# Create a Private DNS Zone for the Event Hub namespace
private_dns_zone = dns.PrivateZone(
    'private-dns-zone',
    resource_group_name=resource_group.name,
    location='global',  # Private DNS zone should be set to global
    zone_name="privatelink.servicebus.windows.net",  # This is the standard zone name for Event Hubs
)

# Link the VNet to the Private DNS Zone
virtual_network_link = dns.VirtualNetworkLink(
    'virtual-network-link',
    resource_group_name=resource_group.name,
    virtual_network_id=virtual_network.id,
    private_zone_name=private_dns_zone.name,
    registration_enabled=False, # Only validation, not registration
)

# Create a DNS A Record in the Private DNS Zone pointing to the Private Endpoint
dns_a_record = dns.RecordSet(
    'dns-a-record',
    name=private_endpoint.name,
    resource_group_name=resource_group.name,
    zone_name=private_dns_zone.name,
    ttl=300,
    a_records=[dns.ARecordArgs(
        ipv4_address=private_endpoint.network_interfaces.apply(
            lambda nis: nis[0].ip_configurations[0].private_ip_address if nis else None, # Fetch the private IP from the endpoint's NIC
        ),
    )],
    type='A',
)

# Exports
pulumi.export("resource_group", resource_group.name)
pulumi.export("event_hub_namespace", eventhub_namespace.name)
pulumi.export("event_hub", event_hub.name)
pulumi.export("virtual_network", virtual_network.name)
pulumi.export("subnet", subnet_for_pe.name)
pulumi.export("private_endpoint", private_endpoint.id)
pulumi.export("private_dns_zone", private_dns_zone.name)
```

This program sets up secure, private connectivity to Azure Event Hubs, restricting the access to within the virtual network, which is a common practice for sensitive applications or compliance requirements.

Remember to replace any placeholder names like the Event Hub name with your own preferred naming conventions. Now, you're all set! With this code, you have established a secure, private networking foundation to interact with Azure Event Hubs for any real-time processing or AI workloads you might have.