Google Git Google Git
Sign in
chromium/devtools/devtools-frontend/HEAD/./test/ai_evals
tree: 6383cd40a4f29c0f60025098ac182fb7d3cfbf62 [search]
  1. accessibility/
  2. elements/
  3. helpers/
  4. network/
  5. performance/
  6. report/
  7. ai-eval-state-provider.ts
  8. ai_smoke.test.ts
  9. BUILD.gn
  10. mocha-interface.ts
  11. mocha_hooks.ts
  12. README.md
  13. run-mocha.ts
test/ai_evals/README.md

DevTools AI Evaluations

This directory contains automated test suites for DevTools AI Assistance.

[!WARNING] Google Linux (gLinux) Environment Required This automated test suite runs only on Google Linux (gLinux) workstations and bots. It requires stubby and Linux-specific internal Google tools, which means it does not work for external contributors.

Note for macOS Users: You can still run AI Assistance evaluations on macOS. Use the manual evaluation workflow in ../../scripts/ai_assistance/README.md.

Prerequisites

Before you run this automated test suite, the prerequisites are:

  1. A Google Linux workstation (gLinux).
  2. Authentication with gcloud:
    gcloud auth login
  3. Add "checkout_devtools_ai_evals": True to custom_vars in your .gclient file and run gclient sync.
  4. A Google OTA (test) account. By default, the test runner uses the shared team OTA account stored in test/ai_evals/eval_data/ota_account.txt. If you create your own account instead, make sure to create a production test account, with a password set (see ‘updated password flow’ during account creation).

Build the Tests

Compile the test files before you run the test suite:

autoninja -C out/Default test/ai_evals:ai_evals

Run the Tests

AI Evals tests (uses the default team OTA account from eval_data/ota_account.txt):

./dt test test/ai_evals

To override with a custom OTA account:

./dt test test/ai_evals --ota-username your_ota_account@gmail.com

[!NOTE] Passing --ota-password is generally only needed for OTA accounts where the user or environment cannot change the password dynamically (for example, on CI bots that cannot use stubby and instead rely on a pre-configured password):

./dt test test/ai_evals --ota-username your_ota_account@gmail.com --ota-password your_password

Harness tests:

vpython3 scripts/test_harness.py DevToolsTestHarness.test_ai_evals_auth_helper

Protip: You can add --debug to the harness test command line, to see the output.

Test Execution Steps

When you run the test, the test runner performs these steps:

  1. Uses stubby to get a temporary password for the OTA account (unless --ota-password is provided).
  2. Starts Chrome and signs in to the Google account automatically.
  3. Configures DevTools AI Assistance preferences.
  4. Opens the inspected target page and sends prompts to AI Assistance.
  5. Verifies that DevTools receives structured AI responses.