Using aws sqs with transcribe

Question

Pulumi · Accepted Answer

AWS SQS (Simple Queue Service) can be used as a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. On the other hand, AWS Transcribe is an automatic speech recognition (ASR) service that makes it easy for developers to add speech-to-text capability to their applications.

Using these two AWS services together might involve a workflow where audio files are processed by AWS Transcribe to convert speech into text, and the results are then published to an SQS queue for further processing or to trigger subsequent workflows.

The workflow could look something like this:

1. Audio files are stored in an S3 bucket.
2. An event triggers AWS Lambda when a new audio file is uploaded.
3. The Lambda function calls AWS Transcribe to start a transcription job.
4. Once the transcription job is completed, the result is posted to an SQS queue.
5. Another Lambda function is triggered by the message in the SQS queue to process the transcription result.

Below is an example Pulumi program written in TypeScript that sets up such a workflow:

```typescript
import * as aws from "@pulumi/aws";

// Create an SQS queue where transcription results will be sent.
const transcribeResultsQueue = new aws.sqs.Queue("transcribeResultsQueue", {
    // Attributes and other configurations can be applied here.
    // For example, setting the visibility timeout, message retention period, etc.
});

// Create an S3 bucket that will store the audio files for transcription.
const audioFilesBucket = new aws.s3.Bucket("audioFilesBucket", {
    // Configurations for the bucket, such as versioning, logging, etc.
});

// IAM Role that AWS Transcribe can assume to access the necessary resources.
const transcribeServiceRole = new aws.iam.Role("transcribeServiceRole", {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.TranscribeServicePrincipal),
});

// Policy to grant AWS Transcribe access to the S3 bucket and the SQS queue.
const transcribePolicy = new aws.iam.RolePolicy("transcribePolicy", {
    role: transcribeServiceRole.id,
    policy: {
        Version: "2012-10-17",
        Statement: [
            {
                Effect: "Allow",
                Action: ["s3:GetObject", "s3:ListBucket"],
                Resource: [audioFilesBucket.arn, `${audioFilesBucket.arn}/*`],
            },
            {
                Effect: "Allow",
                Action: "sqs:SendMessage",
                Resource: transcribeResultsQueue.arn,
            },
        ],
    },
});

// Lambda function that is triggered by new audio files in the S3 bucket and starts the transcription job.
const startTranscriptionFunction = new aws.lambda.CallbackFunction("startTranscriptionFunction", {
    policies: [
        // Add necessary policies to allow Lambda to interact with S3, Transcribe, and SQS.
    ],
    callback: async (event: aws.types.input.lambda.S3Event) => {
        // Code to start a transcription job using AWS Transcribe.
        // Upon completion, the transcription result is sent to the SQS queue set up earlier.
    },
});

// Finally, create an event source mapping to trigger the Lambda function when a new audio file appears in the S3 bucket.
const s3EventSourceMapping = new aws.lambda.EventSourceMapping("s3EventSourceMapping", {
    targetArn: audioFilesBucket.arn,
    functionName: startTranscriptionFunction.arn,
    // Additional configurations like batch size, etc.
});

// Output the SQS Queue URL for reference.
export const queueUrl = transcribeResultsQueue.id;
```

### Explanation:

In the above Pulumi program:

- We create an SQS queue called `transcribeResultsQueue` which will receive messages containing transcription results from AWS Transcribe.
- We then set up an S3 bucket `audioFilesBucket` which will be used to store audio files that need to be transcribed.
- We create an IAM role `transcribeServiceRole` that AWS Transcribe will assume, granting it the necessary permissions to read from the S3 bucket and write to the SQS queue via `transcribePolicy`.
- There's also a placeholder Lambda function `startTranscriptionFunction`. In a real-world scenario, the body of this function would contain code to call AWS Transcribe and start transcription jobs based on events, but this specific logic is not included here for brevity.
- `s3EventSourceMapping` creates a linkage between the S3 bucket and the Lambda function, such that the function is triggered by events in the bucket, like the upload of new audio files.

With these resources in place, you would have a robust, serverless pipeline for processing audio transcriptions, with AWS Transcribe handling the convert-to-text aspect and AWS SQS for managing the messages containing the transcriptions, ready for downstream processing.