Pub/Sub as Event Source for Serverless AI Functions

Question

Pulumi · Accepted Answer

To accomplish your goal of using Pub/Sub as an event source for serverless AI functions, we'll use Google Cloud Platform (GCP) services. We'll create a Pub/Sub topic to send messages to, and a Cloud Function (Google's serverless function service) that triggers whenever a message is published to the specified topic.

Here's the plan:

1. **Google Cloud Pub/Sub**: We'll create a Pub/Sub topic. This service allows you to send and receive messages between independent applications. You can think of a topic as a channel where publishers send messages and subscribers receive them.

2. **Google Cloud Functions**: We'll create a Cloud Function that gets triggered by our Pub/Sub topic's messages. The function acts as the subscriber in this context. This serverless offering will run your code in response to events such as messages on a Pub/Sub topic.

3. **IAM Policy**: We will set up permissions so the Cloud Function has the necessary access rights to be invoked by Pub/Sub messages.

Below is the Python program using Pulumi to set up this architecture. Before running this program, make sure to have Pulumi installed, a GCP project created and that your GCP account is properly configured with Pulumi using `pulumi config set gcp:project [PROJECT_ID]`.

```python
import pulumi
import pulumi_gcp as gcp

# Create a Pub/Sub topic that will be used to trigger the function.
topic = gcp.pubsub.Topic("my-topic")

# The content of the Cloud Function, a simple Python function in this case
# Replace the contents of the 'main.py' with your actual serverless AI function.
# This can be any Python code that handles incoming Pub/Sub messages.
cloud_function_contents = """
def hello_pubsub(event, context):
    print('Received message: {}'.format(event['data']))
"""

# A Cloud Function is created which listens to the aforementioned topic.
# The content of the function can be updated to perform some AI operations.
# Here, it simply logs the message it receives for demonstration purposes.
function = gcp.cloudfunctions.Function("my-function",
    source_archive_bucket=bucket.name,
    runtime="python37",
    source_archive_object=object.name,
    entry_point="hello_pubsub",
    event_trigger={
        "event_type": "providers/cloud.pubsub/eventTypes/topic.publish",
        "resource": topic.name,
    }
)

# Set the IAM policy to allow invocation of the Cloud Function from Pub/Sub.
iam_policy = gcp.cloudfunctions.FunctionIamPolicy("my-function-iam",
    project=function.project,
    region=function.region,
    cloud_function=function.name,
    bindings=[{
        "role": "roles/cloudfunctions.invoker",
        "members": [f"serviceAccount:{function.service_account_email}"],
    }]
)

# Export the function name and the topic name.
pulumi.export("function_name", function.name)
pulumi.export("topic_name", topic.name)
```

In this code:

- We first create a Pub/Sub topic called `my-topic`.
- We write the contents of our Cloud Function into a string `cloud_function_contents`, which simply logs the incoming message. In a real-world scenario, this would be your AI code.
- We create the Cloud Function, specifying which topic it subscribes to, the runtime environment, and the entry point to the application. `entry_point` denotes the name of the function within your code to execute.
- We specify the IAM policy for our function, allowing it to be invoked by Pub/Sub. We assign the `roles/cloudfunctions.invoker` role to our service account, which is automatically created for our Cloud Function.

You would replace the `cloud_function_contents` with your serverless AI function logic. When messages are sent to the Pub/Sub topic, it would trigger the Cloud Function to execute your AI function.

Do not forget to include dependencies in your `requirements.txt`, for example:

```
pulumi
pulumi_gcp
```

To run this program, save it into a file, e.g., `main.py`, and then run `pulumi up` through the command line. This will start the deployment process in your GCP account configured with Pulumi. After the deployment is successful, you will be able to publish messages to your Pub/Sub topic and have them processed by your serverless AI functions.