Tracking Confluent Schema Registry Changes with Datadog

Question

Pulumi · Accepted Answer

To track changes in a Confluent Schema Registry and monitor them using Datadog, we can create a Datadog monitor that alerts us when certain conditions are met in the Schema Registry, such as unexpected changes or unreachable endpoints. Pulumi allows us to define infrastructure as code, including the resources for monitoring.

Firstly, let’s explore the resources provided by the `pulumi_datadog` package that we can use for this purpose:

1. `datadog.Monitor`: This resource allows us to define a Datadog monitor. Monitors in Datadog are used to watch over a metric or check that you define and send alerts according to your settings when the monitored item deviates from your desired thresholds.

The following Pulumi program in Python sets up a Datadog monitor that could be used to track changes in a Confluent Schema Registry:

```python
import pulumi
import pulumi_datadog as datadog

# Replace 'your_query_here' with a query appropriate for monitoring Confluent Schema Registry.
# For example, a query might count the number of schema versions, check schema validation results,
# or the result of a custom agent check that you set up to track Schema Registry health.
schema_registry_query = 'avg(last_5m):anomalies(your_metric_here, \'basic\', 2) > 0'

# Create a monitor in Datadog
schema_registry_monitor = datadog.Monitor("schemaRegistryMonitor",
    type="query alert",
    query=schema_registry_query,
    name="Confluent Schema Registry Monitor",
    message="Notification message @pagerduty", # Uses the @-notation to notify a specific user, schedule, or service integration.
    tags=["confluent", "schema-registry", "monitoring"],
    priority=3,
    no_data_timeframe=20,  # Number of the minutes before a no_data event is triggered
    new_host_delay=300, # Delay in seconds for the evaluation of new hosts
    notify_no_data=False,  # Specifies if notifications should be sent when the monitor has no data
    renotify_interval=10, # Time in minutes to re-notify to avoid duplicate alerts
    notify_audit=False,   # Additionally include an audit event log post
    timeout_h=0,          # The monitor will automatically resolve after the timeframe (in hours), without no_data_timeframe also being set
    include_tags=True,    # Include relevant tags on the alert event
    require_full_window=True, # Require full window of data to evaluate the given query
    locked=False,         # Whether changes can be made to the monitor by other users
    thresholds={
        "critical": 1,       # Over this value will trigger the alert
        "warning": 0.5,      # Over this value will result in a warning
    }
)

# Export the URL to the monitor which can be used to view the monitor in the Datadog dashboard
pulumi.export('schema_registry_monitor_url', pulumi.Output.concat('https://app.datadoghq.com/monitors/', schema_registry_monitor.id))
```

## Explanation:

In this Pulumi program:

- We import the necessary Pulumi packages for use in our code, `pulumi` and `pulumi_datadog`.
  
- We define a `schema_registry_query` which is a query that Datadog will evaluate. It's written using Datadog's query language. Replace `'your_metric_here'` with the specific metric you're interested in monitoring. This could be the number of schema validations or any other metric relevant to the Schema Registry.

- We then instantiate a `datadog.Monitor` object, passing the necessary configuration parameters such as the monitor type, query, priority, and notification settings.

- A combination of tags like `"confluent", "schema-registry", "monitoring"` are used to categorize and filter monitors in the Datadog dashboard.

- We define critical and warning thresholds, where critical indicates a level at which an alert will be triggered, and warning serves as a pre-alert level.

- We export the monitor URL so you can directly navigate to your Datadog monitor after deploying this Pulumi code. The URL is generated by concatenating the monitor ID to the base URL of the Datadog app.

This program is just a starting point, and you can customize the configuration of the Datadog monitor to fit the specifics of your Confluent Schema Registry setup and your monitoring requirements.