Twitter search in Athena

This example lives in the pulumi/examples repository. Check out just this directory to use it:

$ git clone --filter=blob:none --sparse https://github.com/pulumi/examples pulumi-examples
$ git -C pulumi-examples sparse-checkout set aws-ts-twitter-athena
$ cd pulumi-examples/aws-ts-twitter-athena

A sample project that queries Twitter every 2 minutes and stores the results in S3. The project also sets up an Athena table and query. This project demonstrates using aws.cloudwatch.EventRule to run a Lambda on an interval.

Before deploying, register a new Twitter app so you have the consumer and access keys used below.

Prerequisites#

1. Install Pulumi
2. Configure AWS credentials
3. Install Node.js

Deploying the example#

1. Create a new stack:

   pulumi stack init twitter-athena

2. In Twitter, get the keys for your application. Set configuration values for your Twitter consumer key/secret and application key/secret. Use the --secret flag to securely encrypt secret values.

   pulumi config set twitterAccessTokenKey <Value for Consumer Key (API Key)>
   pulumi config set --secret twitterAccessTokenSecret <Value for Consumer Secret (API Secret)>
   pulumi config set twitterConsumerKey <Value for Access Token>
   pulumi config set --secret twitterConsumerSecret <Value for Access Token Secret>

3. Set a search term to query for:

   pulumi config set twitterQuery "Amazon Web Services"

4. Set the AWS region to deploy into:

   pulumi config set aws:region us-west-2

5. Install dependencies:

   npm install

6. Preview and deploy the app. A total of 16 resources are created.

   pulumi up

7. View the stack outputs:

   pulumi stack output

   Please choose a stack: aws-serverless-js-twitter-dev
   Current stack outputs (4):
       OUTPUT                                           VALUE
       athenaDatabase                                   tweets_database
       bucketName                                       tweet-bucket-de18828
       createTableQueryUri                              https://us-west-2.console.aws.amazon.com/athena/home?force#query/saved/e394800e-a35e-44b3-b8ca-8b47b0f74469
       topUsersQueryUri                                 https://us-west-2.console.aws.amazon.com/athena/home?force#query/saved/51fa5744-bab6-4e5f-8cd6-9447b6619f06

8. Navigate to the URL for createTableQueryUri and run the query in the Athena console. This will create a table called tweets.

9. Navigate to the URL for topUsersQueryUri and run the query in Athena. You’ll see tweets for your search term, by users with more than 1000 followers.

   

Cleaning up#

To clean up resources, destroy your stack and remove it:

pulumi destroy
pulumi stack rm