diff --git a/README.md b/README.md index 46d4bd6..23309c0 100644 --- a/README.md +++ b/README.md @@ -137,26 +137,26 @@ Replace directory with the path to the directory containing the CTRF reports you The test object in the report includes the following [CTRF properties](https://ctrf.io/docs/schema/test): -| Name | Type | Required | Details | -| -------------- | ---------------- | -------- | -------------------------------------------------------------------------------------------- | -| `name` | String | Required | The name of the test. | -| `status` | String | Required | The outcome of the test. One of:`passed`, `failed`, `skipped`, `pending`, `other`. | -| `duration` | Number | Required | The time taken for the test execution, in milliseconds. | -| `start` | Number | Optional | The start time of the test as a Unix epoch timestamp. | -| `stop` | Number | Optional | The end time of the test as a Unix epoch timestamp. | -| `suite` | String | Optional | The suite or group to which the test belongs. | -| `message` | String | Optional | The failure message if the test failed. | -| `trace` | String | Optional | The stack trace captured if the test failed. | -| `rawStatus` | String | Optional | The original playwright status of the test before mapping to CTRF status. | -| `tags` | Array of Strings | Optional | The tags retrieved from the test name | -| `type` | String | Optional | The type of test (e.g.,`api`, `e2e`). | -| `filepath` | String | Optional | The file path where the test is located in the project. | -| `retries` | Number | Optional | The number of retries attempted for the test. | -| `flaky` | Boolean | Optional | Indicates whether the test result is flaky. | -| `browser` | String | Optional | The browser used for the test. | -| `screenshot` | String | Optional | A base64 encoded screenshot taken during the test. | -| `steps` | Array of Objects | Optional | Individual steps in the test, especially for BDD-style testing. | -| `extra` | Object | Optional | Custom data relevant to the test. | +| Name | Type | Required | Details | +| ------------ | ---------------- | -------- | ---------------------------------------------------------------------------------- | +| `name` | String | Required | The name of the test. | +| `status` | String | Required | The outcome of the test. One of:`passed`, `failed`, `skipped`, `pending`, `other`. | +| `duration` | Number | Required | The time taken for the test execution, in milliseconds. | +| `start` | Number | Optional | The start time of the test as a Unix epoch timestamp. | +| `stop` | Number | Optional | The end time of the test as a Unix epoch timestamp. | +| `suite` | String | Optional | The suite or group to which the test belongs. | +| `message` | String | Optional | The failure message if the test failed. | +| `trace` | String | Optional | The stack trace captured if the test failed. | +| `rawStatus` | String | Optional | The original playwright status of the test before mapping to CTRF status. | +| `tags` | Array of Strings | Optional | The tags retrieved from the test name | +| `type` | String | Optional | The type of test (e.g.,`api`, `e2e`). | +| `filepath` | String | Optional | The file path where the test is located in the project. | +| `retries` | Number | Optional | The number of retries attempted for the test. | +| `flaky` | Boolean | Optional | Indicates whether the test result is flaky. | +| `browser` | String | Optional | The browser used for the test. | +| `screenshot` | String | Optional | A base64 encoded screenshot taken during the test. | +| `steps` | Array of Objects | Optional | Individual steps in the test, especially for BDD-style testing. | +| `extra` | Object | Optional | Custom data relevant to the test. | ## BDD styled tests @@ -166,30 +166,28 @@ The test object in the report includes the following [CTRF properties](https://c Some features require additional setup or usage considerations. - ### Test steps Test steps are included in the `steps` property of a `Test` object. However, this `steps` property includes only top level steps. This means any children steps i.e. implementation details, are not included in it. Children steps are included in the `extra` property as `childSteps`. - Every step has following properties of its own. -| Name | Type | Required | Details | -| ---------- | ------ | -------- | -------------------------------------------------------------------------------------------- | -| `name` | String | Required | The name/title of the step. | -| `status` | String | Required | The outcome of the step. One of:`passed`, `failed`, `skipped`, `pending`, `other`. | -| `extra` | Object | Optional | Custom data relevant to the step. | +| Name | Type | Required | Details | +| -------- | ------ | -------- | ---------------------------------------------------------------------------------- | +| `name` | String | Required | The name/title of the step. | +| `status` | String | Required | The outcome of the step. One of:`passed`, `failed`, `skipped`, `pending`, `other`. | +| `extra` | Object | Optional | Custom data relevant to the step. | Apart from including child steps, `extra` property is also used to report a steps' category, execution duration and the location in a source file. `extra` object contains following additional properties generated specifically for this JSON report. -| Name | Type | Details | -| -------------- | --------------- | --------------------------------------------------------------------------------------------- | -| `category` | String | The category of the step. One of:`hook`, `expect`, `pw:api`, `test.step` | -| `duration` | String | The execution duration of the step, in milliseconds. | -| `location` | Object | Location the step in test suite, including file name, line number and column number. | -| `childSteps` | Array of object | Child steps of the current step, if any. Child steps have the same properties that of a step. | +| Name | Type | Details | +| ------------ | --------------- | --------------------------------------------------------------------------------------------- | +| `category` | String | The category of the step. One of:`hook`, `expect`, `pw:api`, `test.step` | +| `duration` | String | The execution duration of the step, in milliseconds. | +| `location` | Object | Location the step in test suite, including file name, line number and column number. | +| `childSteps` | Array of object | Child steps of the current step, if any. Child steps have the same properties that of a step. | #### Include test steps only diff --git a/docs/test-steps-only-examples.md b/docs/test-steps-only-examples.md index 3cfbde7..bf9b32d 100644 --- a/docs/test-steps-only-examples.md +++ b/docs/test-steps-only-examples.md @@ -15,11 +15,11 @@ Below are some examples of how the report will look based on the `testStepsOnly` ### Simple test - No `test.step()` used. ```typescript -test("has title @title", async ({ page }) => { - await page.goto("https://playwright.dev/"); +test('has title @title', async ({ page }) => { + await page.goto('https://playwright.dev/') - await expect(page).toHaveTitle(/Playwright/); -}); + await expect(page).toHaveTitle(/Playwright/) +}) ``` A simple test like above, will be reported as: @@ -176,22 +176,27 @@ A simple test like above, will be reported as: ```ts // beforeEach hook, will be included in CTRF report. test.beforeEach(async ({ page }) => { - await page.goto("https://playwright.dev/"); -}); - -test("with nested steps", { tag: [ "@steps", "@nested-steps" ] }, async ({ page }) => { - - await test.step("Verify page title contains Playwright", async () => { - await expect.soft(page).toHaveTitle(/Playwright/); - }); - - // No test.step() usage for this step, but it will be included in CTRF report. - await page.getByRole("link", { name: "Get started" }).click(); - - await test.step("Verify Installation heading is visible", async () => { - await expect.soft(page.getByRole("heading", { name: "Installation" })).toBeVisible(); - }); -}); + await page.goto('https://playwright.dev/') +}) + +test( + 'with nested steps', + { tag: ['@steps', '@nested-steps'] }, + async ({ page }) => { + await test.step('Verify page title contains Playwright', async () => { + await expect.soft(page).toHaveTitle(/Playwright/) + }) + + // No test.step() usage for this step, but it will be included in CTRF report. + await page.getByRole('link', { name: 'Get started' }).click() + + await test.step('Verify Installation heading is visible', async () => { + await expect + .soft(page.getByRole('heading', { name: 'Installation' })) + .toBeVisible() + }) + } +) ``` A test having `test.step()` as steps, will be reported as: @@ -416,6 +421,7 @@ A test having `test.step()` as steps, will be reported as: ``` ### BDD Style test - using [Playwright-bdd](https://vitalets.github.io/playwright-bdd/#/) + ```gherkin @login Feature: User Login @@ -727,11 +733,11 @@ A BDD styled test like the one above, will be reported as: ### Simple test - No `test.step()` used. ```ts -test("has title @title", async ({ page }) => { - await page.goto("https://playwright.dev/"); +test('has title @title', async ({ page }) => { + await page.goto('https://playwright.dev/') - await expect(page).toHaveTitle(/Playwright/); -}); + await expect(page).toHaveTitle(/Playwright/) +}) ``` Simple test above will be reported as: @@ -766,25 +772,31 @@ Simple test above will be reported as: ```ts // beforeEach hook, will NOT be included in CTRF report. test.beforeEach(async ({ page }) => { - await page.goto("https://playwright.dev/"); -}); - -test("with nested steps", { tag: [ "@steps", "@nested-steps" ] }, async ({ page }) => { - await test.step("Navigate to Playwright homepage", async () => { - await page.goto("https://playwright.dev/"); - }); - - await test.step("Verify page title contains Playwright", async () => { - await expect(page).toHaveTitle(/Playwright/); - }); - - // No test.step() used for this step, it will NOT be included in CTRF report. - await page.getByRole("link", { name: "Get started" }).click(); - - await test.step("Verify Installation heading is visible", async () => { - await expect(page.getByRole("heading", { name: "Installation" })).toBeVisible(); - }); -}); + await page.goto('https://playwright.dev/') +}) + +test( + 'with nested steps', + { tag: ['@steps', '@nested-steps'] }, + async ({ page }) => { + await test.step('Navigate to Playwright homepage', async () => { + await page.goto('https://playwright.dev/') + }) + + await test.step('Verify page title contains Playwright', async () => { + await expect(page).toHaveTitle(/Playwright/) + }) + + // No test.step() used for this step, it will NOT be included in CTRF report. + await page.getByRole('link', { name: 'Get started' }).click() + + await test.step('Verify Installation heading is visible', async () => { + await expect( + page.getByRole('heading', { name: 'Installation' }) + ).toBeVisible() + }) + } +) ``` A test with `test.step()` usage, will be reported as: @@ -855,6 +867,7 @@ Feature: User Login When the User tries to login with "locked_out_user" as username and "secret_sauce" as password Then the User should see a locked out error message ``` + A BDD styled test like the one above, is reported as shown below. > ⚠️ Notice that the step in the 'Background' hook is not included. Similarly, any steps or fixtures executed as part of 'After' hooks will be omitted as well. @@ -908,5 +921,7 @@ A BDD styled test like the one above, is reported as shown below. } } ``` + ## BDD styled tests with [Cucumber](https://cucumber.io/docs/guides/overview/) + BDD styled tests that use cucumber as test runner are not supported. Only [playwright-bdd](https://vitalets.github.io/playwright-bdd/#/) is supported because it uses [Playwright Test](https://playwright.dev/docs/test-configuration) as a test runner to execute the tests.