What's the easiest way to only re-run failures after the Playwright Tests are finished?

Out of all the wonderful features that the Playwright Test Runner has around retrying failures, one thing I found myself needing was an easy way to re-run all the tests from a recent run. This is different than test retry and assert retry in that those are completing retries within a test run. I was looking for a way to re-run failures that failed on my local or maybe failed in our CI pipelines. This really comes into play when there is more than 1 failure. I first used this functionty when working with Ruby using rspec test runner,  they had a command line argument--only-failures which could be run with the rspec command to only run failed test from the previous run. Playwright didn't have this functionality so I built it :).

This does require you to have a custom reporter installed playwright-json-summary-reporter. A quick guide on how to install this can be found at the link below.

Is it possible to get a parsable file with the list of all passed or failed tests after a Playwright Test run?

When I went to go answer this question at the beginning of my Playwright journey, I wasn’t able to find a solution that worked for me. Thankfully the Playwright team has built the reporters functionality where it’s easy to create your own custom reports. This lead me down the path

Playwright SolutionsButch Mayhew

Once installed and configured as a custom reporter, you will have a summary.json file that is at the root directory. This file contains a list of all the passed tests along with the tests that timed out or failed. With this we are now able to create a bash script that can parse the json file and generate a list of files that can be passed into a command like npx playwright test <failed tests>.

Below is an example of the bash script. Take that and save it as run-failures.sh.

#!/bin/bash

if [ $# -eq 0 ]
  then
    URL="https://www.automationexercise.com"
  else
    URL=$1
fi

failed=`cat summary.json | jq -r '.failed[]' |  tr '\n' ' '`
timed_out=`cat summary.json | jq -r '.timedOut[]' |  tr '\n' ' '`
echo "URL=$URL npx playwright test $failed $timed_out --workers=1"

# Below is the actual command that gets run
URL=$URL npx playwright test $failed $timed_out --workers=1

When it is run with no parameters  

sh run-failures.sh

it will take the default URL set as "https://www.automationexercise.com" which of course you can modify to the baseURL that your test suite uses.

If you do want to run it against a different url that what you have set as default your command will be

sh run-failures.sh http://localhost:3000

The http://localhost:3000 is an example if I were running the code on my local machine it would use that URL when running the npx playwright test command.

Now I can quickly run only a subset of the tests that failed!

Bonus: for those on Windows Machines, you can try and use this script save it as a run-failures.ps1 and run it as a power shell script. Hopefully it works, it was generated with ChatGPT, please let me know how it turns out via LinkedIn!

# I'm not sure if this works, I generated it with chatgpt from my bash script

if ($args.Length -eq 0) {
  $URL = "https://www.automationexercise.com"
} else {
  $URL = $args[0]
}

$failed = (Get-Content summary.json | ConvertFrom-Json).failed -join ' '
$timed_out = (Get-Content summary.json | ConvertFrom-Json).timedOut -join ' '
Write-Host "URL=$URL npx playwright test $failed $timed_out --workers=1"

# Below is the actual command that gets run
$env:URL=$URL npx playwright test $failed $timed_out --workers=1

Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below.