Real-time Image Processing with Azure Event Grid and Functions

Question

Pulumi · Accepted Answer

To set up real-time image processing using Azure Event Grid and Azure Functions, you'll need to follow these steps:

Azure Functions App: This will serve as the compute where your code for processing images is executed. Azure Functions support trigger-based execution, so you can process your images whenever new data is available.
Event Grid Topic: Event Grid is a service that enables you to build scalable event-based applications. You'll create a custom topic to which events (like adding a new image to a blob storage) are sent.
Event Subscription: This lets you route the events from your Event Grid topic to a specific endpoint, in this case, the Azure Function App that will process the images.
Blob Storage: When images are uploaded to blob storage, an event is generated and sent to the Event Grid topic, and then routed to the Azure Function for processing.

Here's a Python program that uses Pulumi to automate the deployment of such a system on Azure:

import pulumi
import pulumi_azure as azure

# Define the resource group where all resources will be deployed
resource_group = azure.core.ResourceGroup("image-processing-rg")

# Set up the Blob storage account where images will be stored
storage_account = azure.storage.Account("imagesaccount",
    resource_group_name=resource_group.name,
    account_tier="Standard",
    account_replication_type="LRS")

# Create a storage container within the storage account
images_container = azure.storage.Container("images",
    storage_account_name=storage_account.name,
    container_access_type="private")

# Create an Azure Functions App
app_service_plan = azure.appservice.Plan("functions-app-service-plan",
    resource_group_name=resource_group.name,
    kind="FunctionApp",
    reserved=True, # Required for Linux/Docker plans
    sku={
        "tier": "Dynamic",
        "size": "Y1"
    })

function_app = azure.appservice.FunctionApp("image-processor-func-app",
    resource_group_name=resource_group.name,
    app_service_plan_id=app_service_plan.id,
    storage_account_name=storage_account.name,
    storage_account_access_key=storage_account.primary_access_key,
    os_type="linux", # Azure Functions can run on Windows or Linux
    site_config={
        "cors": {
            "allowed_origins": ["*"]
        }
    })

# Create an Event Grid Topic
event_grid_topic = azure.eventgrid.Topic("image-events-topic",
    resource_group_name=resource_group.name,
    location=resource_group.location)

# Create an Event Grid Subscription to trigger the function
event_subscription = azure.eventgrid.EventSubscription("image-added-subscription",
    resource_group_name=resource_group.name,
    topic_name=event_grid_topic.name,
    webhook_endpoint=function_app.default_hostname.apply(
        lambda endpoint: f"https://{endpoint}/runtime/webhooks/eventgrid?functionName=<YOUR-FUNCTION-NAME-HERE>&code=<YOUR-FUNCTION-KEY-HERE>"))

# Export the URLs the user needs to interact with the system
pulumi.export('Function App URL', function_app.default_hostname)
pulumi.export('Event Grid Topic Endpoint', event_grid_topic.endpoint)
pulumi.export('Blob Storage Endpoint', pulumi.Output.concat('https://', storage_account.name, '.blob.core.windows.net/', images_container.name))

In the code above:

We create a resource group to contain all the resources.
We then set up a storage account and a container within it. This is where images will be uploaded.
An app service plan is created to host the Azure Function. We also specify that we want to run it in a Linux environment (reserved=True).
The Function App is next. This app uses the storage account set up earlier.
We create an Event Grid Topic, which is where events pertaining to our images will be sent.
The EventSubscription object ties it all together. It specifies that when a new event is detected on our custom topic, it should trigger an Azure Function, whose URL is dynamically provided using Pulumi's apply method.
The apply is used here to concatenate the function app's default hostname with the specific endpoint and function key that Azure requires to trigger the function from Event Grid.
Finally, we export the URLs that are needed to interact with this setup.

Remember to replace the placeholders (<YOUR-FUNCTION-NAME-HERE> and <YOUR-FUNCTION-KEY-HERE>) with your actual Azure Function name and its authorization key.

Please note you will need to have the Azure Functions Core Tools installed and have written a function that can process the image - this function will be deployed into the "image-processor-func-app" Function App created by this script. Before deploying this function to Azure, test it locally to ensure it works as expected.

Deploying this program will create all the necessary Azure resources to have a working real-time image processing system. You can upload images to the specified Blob Storage container, and the Event Grid will trigger the Azure Function to process the images as they come in.

Remember to configure your Pulumi CLI with the appropriate Azure credentials before running this program.