Automated UI Testing for AI-Integrated Applications on Checkly

Question

Pulumi · Accepted Answer

Automated UI testing is a critical component in ensuring the quality and performance of web applications, especially those with AI integrations where the application behavior might change dynamically. Checkly is a service that provides active monitoring and testing for modern web applications. With Pulumi's Checkly integration, you can automate the deployment and management of Checkly resources like checks, check groups, triggers, and more.

The resources we'll focus on for setting up automated UI testing with Checkly in a Pulumi program are:

- `checkly.CheckGroup`: Organizes a set of checks and maintains common settings like locations from where the tests will be executed, alert settings, and environment variables.

- `checkly.Check`: Defines an individual check, which could be an API check or a browser check. For UI testing, we'll be using a browser check that utilizes a script to perform actions on the application.

- `checkly.Snippet`: Reusable scripts that can be included in checks. This helps to avoid duplicating common setup or teardown scripts across multiple checks.

Below is a Pulumi program written in Python that defines a check group and a UI check for testing a web application:

```python
import pulumi
import pulumi_checkly as checkly

# Create a Checkly check group to organize our checks
check_group = checkly.CheckGroup('example-check-group',
    activated=True,
    concurrency=1,
    locations=[
        "us-east-1",
        "eu-west-1",
    ],
    alert_settings=checkly.CheckGroupAlertSettingsArgs(
        escalation_type="RUN_BASED",
        reminders=checkly.CheckGroupAlertSettingRemindersArgs(
            amount=3,
            interval=5,
        ),
    )
)

# Define a Pulumi script for the browser check UI testing
pulumi_script_ui_check = """
const assert = require("assert");
const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto("https://www.example.com"); // Replace with your app's URL

// Perform UI interactions or check for certain elements
  // Example: assert that a specific element exists
  const element = await page.$('selector-for-element'); // Replace with an actual selector
  assert.ok(element, "Expected element not found");

await browser.close();
})();
"""

# Create a browser check for performing UI tests
ui_check = checkly.Check('example-ui-check',
    type='BROWSER',
    frequency=10,
    activated=True,
    mute_notifications=False,
    double_check=True,
    script=pulumi_script_ui_check,
    group_id=check_group.id,
    locations=[
        "us-east-1",
        "eu-west-1",
    ],
    alert_settings=checkly.CheckAlertSettingsArgs(
        escalation_type="RUN_BASED",
        reminders=checkly.CheckAlertSettingRemindersArgs(
            amount=3,
            interval=5,
        ),
    ),
    environment_variables=[
        checkly.CheckEnvironmentVariableArgs(
            key="MY_VARIABLE",
            value="some-value",
        ),
    ],
)

# Output the Checkly check URL which you can use to view the check results
pulumi.export('ui_check_url', pulumi.Output.concat('https://checklyhq.com/checks/', ui_check.id))
```

Let's review what this Pulumi program does:

- Creates a Checkly check group that acts as a container for your browser checks. The `locations` property specifies the geographical regions where the checks will be executed from, and `alert_settings` configures how and when alerts should be sent.

- Defines a browser `Check` within that group to conduct UI testing. The `script` property contains a JavaScript snippet written for Puppeteer (a Node library which provides a high-level API to control headless Chrome or Chromium) that opens a webpage, performs actions or assertions, and then closes the browser.

- Finally, the program exports the URL to the Checkly UI check, which can be accessed to view the results of the UI tests.

This Pulumi program will deploy the specified infrastructure for automated UI testing every time you run `pulumi up`. Debugging and updating your UI tests become more manageable with Checkly's dashboard, and Pulumi allows you to apply version control to your infrastructure and test scripts.