Financial Trading Algorithms with Time-Series Data on AWS Timestream

Question

Pulumi · Accepted Answer

AWS Timestream is a fully managed, serverless time series database designed to handle the scale of IoT sensor, application metrics, DevOps telemetry, and log data. Timestream excels in scenarios where you need to store and analyze large amounts of time-series data with low latency for querying and real-time analytics.

Below I'll walk you through a Pulumi Python program that sets up an AWS Timestream infrastructure suitable for running financial trading algorithms with time-series data. This infrastructure includes:

1. **Timestream Database**: The container to organize and manage Timestream tables within the same AWS account.
2. **Timestream Table**: Where the time-series data is stored within the database. It comes with two storage tiers - memory store for recent data and a magnetic store for historical data.

In our example:

- We'll create a database named `TradingData`.
- We'll create a table named `MarketTicks`, which will store the high-frequency market data used by our financial algorithms.
- We'll configure the memory store and magnetic store retention periods to suitable values based on the access patterns.

### Pulumi Python Program for AWS Timestream

```python
import pulumi
import pulumi_aws as aws

# Create a new Timestream database for our financial trading data
trading_database = aws.timestreamwrite.Database("tradingData",
    database_name="TradingData"
)

# Create a new Timestream table within our database for market ticks.
# We're setting the retention periods for both the memory and magnetic stores.
# - MemoryStoreRetentionPeriodInHours determines how long data is kept in the
#   fast access memory store, you would adjust this based on how often you need to
#   access recent data.
# - MagneticStoreRetentionPeriodInDays determines how long your data is accessible in the
#   more economical magnetic store. For financial data, you may be legally required to keep data
#   for a certain number of years.
market_ticks_table = aws.timestreamwrite.Table("marketTicks",
    database_name=trading_database.database_name,
    table_name="MarketTicks",
    retention_properties={
        "memoryStoreRetentionPeriodInHours": 24,
        "magneticStoreRetentionPeriodInDays": 7 * 365,  # 7 years
    }
)

# Export the database and table names so you can easily query them from the AWS CLI or SDKs.
pulumi.export("trading_database_name", trading_database.database_name)
pulumi.export("market_ticks_table_name", market_ticks_table.table_name)
```

### How to Use This Program

1. Install Pulumi and configure your AWS credentials following the [Pulumi AWS Setup Guide](https://www.pulumi.com/docs/get-started/aws/begin/).
2. Create a new directory, initialize a Pulumi project, and create a Python virtual environment.
3. Save the above code into a file named `__main__.py`.
4. Run `pulumi up` to deploy these resources to AWS.

### What Happens When You Run This Program

- Pulumi will prepare a plan that shows what infrastructure will be deployed. Review this plan to ensure everything looks correct.
- After confirmation, Pulumi will use the AWS SDK to create a new Timestream database and a table within it.
- Once the deployment is complete, Pulumi stores the state of your infrastructure, allowing you to easily manage or change it in the future.
- The output of this program includes the names of the created database and table, which you can use in your applications or command-line tools to interact with Timestream.

This example is a fundamental setup for financial time-series data analysis. In a real-world scenario, you might also want to integrate this with other AWS services like Lambda for data processing, Kinesis for data ingestion, and IAM for access control. If you would like to see how to augment this basic Timestream setup with such features, feel free to ask!