How Do I Combine Playwright HTML Reports After Running Multiple Playwright Commands?

A Fun Challenge

I recently had someone reach out to me with this question. They wanted to run a set of tests with 3 workers and then a separate test run with  with 1 worker and the command looked something like this.

npx playwright test --grep-invert @api --workers=3 && npx playwright test --grep @api --workers=1

Their question was at the end of the test, there were 2 HTML reports, and just wanted 1 combined html report. Previously I wrote about how I managed merging reports when sharding in CI which can be found below.

How do you handle combining HTML, Json reports, and a Slack Message after a Playwright test is run with the --shard command?

This is a continuation of a series on How to send a slack notification after Playwright test finish. Part 1 discussed how to use an npm package to do this while Part 2 covered how to send slack messages from a github action. This is Part 3 where I cover

Playwright SolutionsButch Mayhew

After writing that article the Playwright team released the merge-report tool in version 1.37. This release provided a new "blob" reporter along with some extra code which we will utilize to solve the problem at hand.

The Blob Reporter

When utilizing the "blob reporter this will create a folder in the root directory named "blob-report" and will generate a report-{randomcharacters}.zip. The idea is you would copy each report.*.zip file to a main directory all-blob-reports after your test runs, and then run the npx playwright merge-reports --reporter html ./all-blob-reports command. This will combine the blob files and generate an HTML test report.

The challenge which is easier to solve in CI with github actions, can also be achieved when running on your local machine to combine reports. This is where the extra code comes in.

One important note in my playwright.config.ts I went ahead and added the option to my HTML reporter to ["html", { open: "never" }], This way if there is a test failure, this action won't interupt us copying and merging reports in the future steps we'll cover.

Creating a Utility to Copy Blob Files

We'll use a file called updatedBlob.ts as shown below.

// updateBlob.ts

import * as fs from "fs";
import * as path from "path";

// Define the source and destination directory paths
const sourceDirectory: string = path.join(__dirname, "blob-report");
const destinationDirectory: string = path.join(__dirname, "all-blob-reports");

// Check for the 'clean' argument
const args = process.argv.slice(2);
const clean = args.includes("clean");

if (clean) {
  // Remove all files from the destination directory
  fs.readdir(destinationDirectory, (err, files) => {
    if (err) {
      console.error("Error reading destination directory:", err);
      process.exit(1);
    }

    files.forEach((file) => {
      const filePath = path.join(destinationDirectory, file);
      fs.unlink(filePath, (err) => {
        if (err) {
          console.error(`Error deleting file ${file}:`, err);
        } else {
          console.log(`Deleted ${file} from ${destinationDirectory}`);
        }
      });
    });
  });
} else {
  // Ensure the destination directory exists
  fs.mkdir(destinationDirectory, { recursive: true }, (err) => {
    if (err) {
      console.error("Error creating destination directory:", err);
      process.exit(1);
    }

    // Read the source directory
    fs.readdir(sourceDirectory, (err, files) => {
      if (err) {
        console.error("Error reading source directory:", err);
        process.exit(1);
      }

      // Filter files that match the pattern report.*.zip
      const reportFiles = files.filter((file) => /^report.*\.zip$/.test(file));

      // Copy each matching file to the destination directory
      reportFiles.forEach((file) => {
        const sourceFilePath = path.join(sourceDirectory, file);
        const destinationFilePath = path.join(destinationDirectory, file);

        fs.copyFile(sourceFilePath, destinationFilePath, (err) => {
          if (err) {
            console.error(`Error copying file ${file}:`, err);
          } else {
            console.log(`Copied ${file} to ${destinationDirectory}`);
          }
        });
      });
    });
  });
}

Running this script will copy a report.*.zip file from the the blob-report folder and copy it to the all-blob-report folder. This script also allows you to pass in an command line argument clean which will remove all files from the all-blob-report folder. With all of these together we can chain some commands from the terminal and achieve our goal. Below is the final goal

The Final Command

npx playwright test --grep-invert @api --workers=3 || true && npx ts-node updateBlob.ts && npx playwright test --grep @api --workers=1 || true && npx ts-node updateBlob.ts && npx playwright merge-reports --reporter html ./all-blob-reports && npx playwright show-report

I've broken down each command below. One thing to note is I added || true to the end of the playwright commands. What this does is makes it so that if the playwright test fails then it will not exit with a non-zero exit code. Without this here, if there was a failure with running any of the tests the reports wouldn't merge with this command.

// runs all tests that aren't tagged with @api with 3 workers
npx playwright test --grep-invert @api --workers=3 || true && 

// copies over the report.*.zip file to all-blob-reports
npx ts-node updateBlob.ts && 

// runs all tests that are tagged with @api with 1 worker
npx playwright test --grep @api --workers=1 || true && 

// copies over the report.*.zip file to all-blob-reports
npx ts-node updateBlob.ts && 

// merges the two report.*.zip files and generates an html report
npx playwright merge-reports --reporter html ./all-blob-reports && 

// This will open the report after the run you can remove this if you'd like
npx playwright show-report
``

The commit for the changes discussed can be found in the playwright-demo repository.

Going through this exercise helped me learn about how the blob reports work along with learning about how to prevent commands from existing with non-zero status codes.

Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below, and be sure to leave a ❤️ to show some love.