Is it possible to get a parsable file with the list of all passed or failed tests after a Playwright Test run?
When I went to go answer this question at the beginning of my Playwright journey, I wasn't able to find a solution that worked for me. Thankfully the Playwright team has built the reporters functionality where it's easy to create your own custom reports.
This lead me down the path of building my own custom reporter that output the specific details I wanted, in a format that I could parse after a run was complete. My goal was to create a json file that included the duration of the run, the final status (passed/failed), and a list of passed, failed, and skipped tests. Having this information I would be able to programmatically retry any failed tests easily, and list out any failed tests as a part of a slack notification integration.
That leads me to this fun announcement! I created a NPM Package that will do this for you!
To use this package in your playwright repo it is quite simple.
Installation
Run the below command within your playwright directory in order to install the package into your repository. This should add the dependencies to your package.json file.
$ npm install playwright-json-summary-reporter --save-dev
Usage
Now modify your playwright.config.ts file to include the custom reporter:
reporter: [
['playwright-json-summary-reporter'],
['html'], // other reporters
['dot']
],
Now when you run a test there should be a new file summary.json that gets saved to the root of the directory. An example of the file is shown below.
// summary.json
{
"durationInMS": 41565,
"passed": [
"api/create-user-account.spec.ts:34:3",
"api/create-user-account.spec.ts:55:3",
"api/delete-user-account.spec.ts:47:3",
"api/login.spec.ts:12:3",
"api/login.spec.ts:37:3",
"api/login.spec.ts:61:3",
"api/login.spec.ts:84:3",
"ui/codepen/copyToClipboard.spec.ts:5:1",
"ui/codepen/copyToClipboard.spec.ts:17:1",
"ui/internet.app/selectPresentElement.spec.ts:10:1",
"ui/internet.app/selectPresentElement.spec.ts:35:1",
"ui/grep.spec.ts:4:3",
"ui/grep.spec.ts:9:3",
"ui/loginUser.spec.ts:22:3",
"ui/loginUserNoImages.spec.ts:23:3",
"ui/loginUser.spec.ts:37:3",
"ui/loginUserNoImages.spec.ts:40:3",
"ui/registerUser.spec.ts:17:1"
],
"skipped": [],
"failed": [],
"warned": [],
"timedOut": [],
"status": "passed",
"startedAt": 1674431153277
}
With this file in place I can now utilize command line tools such as cat and jq in order to parse and get the data from the file from the command line. The below command will get any values in the failed array and replace the return character with a space, so you now have a list of files that can be used to re-run failures.
failed=`cat summary.json | jq -r '.failed[]' | tr '\n' ' '`
This will also allow us to build out other notifications outside of a Playwright Test Run, such as Slack notifications!
If what I provided above isn't helpful and you are looking for just a plain text file with details of your test run, you should check out Stephen Kilbourn's report summary package.
Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below.