Writing tests

Test format

The JSON definition: settings, every step type, locators and text matches.

All documentation

A test is a JSON document. It is validated when it is saved and again before every run, unknown fields are rejected so typos are reported, and every change creates a new version. The full JSON Schema is at /api/v1/schema/test-definition.

Flow tests

json

{
  "schemaVersion": 1,
  "kind": "flow",
  "startPath": "/",
  "viewport": "desktop",
  "locale": "en-US",
  "timezone": "UTC",
  "colorScheme": "light",
  "timeouts": {
    "stepMs": 10000,
    "navigationMs": 30000,
    "totalMs": 120000
  },
  "checks": {
    "failOnPageError": true,
    "failOnConsoleError": false,
    "failOnServerError": false,
    "ignore": [
      "analytics.example.com"
    ]
  },
  "acceptDialogs": false,
  "steps": [
    {
      "type": "navigate",
      "path": "/pricing"
    },
    {
      "type": "expectTitle",
      "match": "Pricing"
    }
  ]
}

Only schemaVersion, kind and steps are required; the values above are the defaults.

viewport: desktop (1280 x 800), laptop (1440 x 900), tablet (768 x 1024, touch), mobile (390 x 844, touch), or {"width": 1024, "height": 768}.
timeouts: per step at most 30 s, per navigation at most 45 s, per run at most 300 s.
steps: at most 100. Each step accepts id, name, timeoutMs, continueOnFailure and destructive. Mark a step destructive when it creates records, sends messages or takes payments: running it against a production environment then asks for confirmation (in the API, "confirmSideEffects": true), and it is never retried automatically.

Step types

Type	What it does	Fields
`navigate`	Go to a path on the environment or an allowed URL.	path or url, waitUntil (load, domcontentloaded, networkidle, commit), expectStatus
`click`	Click a button, link or any element.	target, button (left, right, middle), clickCount (1 or 2)
`fill`	Replace the value of a text field.	target, value, sensitive
`type`	Type text key by key, for inputs that react to each key.	target, value, delayMs (0 to 200), sensitive
`clear`	Empty a text field.	target
`select`	Choose options in a select element.	target, values (option values or labels)
`check`	Tick a checkbox or radio button.	target
`uncheck`	Clear a checkbox.	target
`hover`	Move the pointer over an element.	target
`focus`	Move keyboard focus to an element.	target
`press`	Press a key such as Enter, Escape or Control+A.	key (such as Enter, Tab, Control+A), target (optional)
`scrollIntoView`	Scroll until an element is in view.	target
`setViewport`	Change the viewport size mid test.	width, height
`waitFor`	Wait until an element appears or disappears.	target, state (visible, hidden, attached, detached)
`waitForUrl`	Wait until the address matches.	match
`waitForLoadState`	Wait for the page load or network to settle.	state (load, domcontentloaded, networkidle)
`wait`	Pause for a fixed time. Prefer waiting for an element.	ms (at most 10000)
`screenshot`	Save a named screenshot with the run.	label, fullPage, target (optional)
`expectVisible`	The element is on screen.	target
`expectHidden`	The element is hidden or absent.	target
`expectText`	The element text matches.	target, match
`expectValue`	The field value matches.	target, match
`expectAttribute`	An attribute value matches.	target, attribute, match
`expectCount`	How many elements match.	target, count or min and/or max
`expectEnabled`	The control can be used.	target
`expectDisabled`	The control is disabled.	target
`expectChecked`	A checkbox or radio state.	target, checked (default true)
`expectUrl`	The page address matches.	match
`expectTitle`	The document title matches.	match
`expectPageText`	The visible page text matches.	match
`expectNoConsoleErrors`	Nothing was logged as an error so far.	ignore (substrings)
`expectNoFailedRequests`	No request answered with an error status so far.	minStatus (400 or 500), ignore
`expectResponse`	A matching request happened, optionally with a status.	url (a match), method, status, sinceLastAction (default true)
`expectScreenshot`	The page looks like the accepted baseline.	label, fullPage, target, mask, threshold (0.1), maxDiffRatio (0.001)
`expectAccessible`	Common automated accessibility checks pass.	rules (optional subset), scope (optional locator)

Locators

The target of a step finds exactly one element. Actions and assertions fail with "matched N elements" when the locator is ambiguous; add nth to choose one.

json

{"by": "role", "role": "button", "name": "Sign in"}
{"by": "label", "text": "Email"}
{"by": "text", "text": "Welcome back", "exact": true}
{"by": "placeholder", "text": "Search"}
{"by": "testId", "testId": "cart-count"}
{"by": "altText", "text": "Company logo"}
{"by": "title", "text": "Close"}
{"by": "css", "selector": "form .error", "nth": "first"}

Role and label locators test what users and assistive technology see, and survive refactors. CSS selectors are the last resort.

Text matches

Assertions on text, URLs and titles take a match: {"mode": "contains", "value": "Saved"} with mode contains, equals, startsWith, endsWith or regex, and optional caseSensitive. A plain string means contains, and every mode except equals needs a non-empty value. Whitespace is collapsed before comparing, except in regex mode. equals and regex are case sensitive by default; the others are not. Regular expressions use RE2 syntax, which runs in linear time: lookarounds and backreferences are not supported, and patterns are limited to 300 characters.

Crawl tests

{"schemaVersion": 1, "kind": "crawl", "startPath": "/", "maxPages": 20, "maxDepth": 3}. See Crawls.

Accessibility rules

image-alt: Images have alternative text.
button-name: Buttons have an accessible name.
link-name: Links have an accessible name.
form-label: Form fields have a label.
document-title: The page has a title.
html-lang: The page declares its language.
duplicate-id: Referenced ids are unique.
heading-order: Heading levels do not skip.

These are basic automated checks that a script can decide reliably. Passing them does not mean a page conforms to WCAG.