Featured

How do you handle combining HTML, Json reports, and a Slack Message after a Playwright test is run with the --shard command?

Butch Mayhew

Feb 20, 2023 • 11 min read

This is a continuation of a series on How to send a slack notification after Playwright test finish. Part 1 discussed how to use an npm package to do this while Part 2 covered how to send slack messages from a github action. This is Part 3 where I cover how to send a single slack notification after a sharded run

💡

As of Aug 11, 2023 the Playwright team has released a new CLI command that will merge a new report type blob. Details about the new command can be found in the docs, and though I don't plan to update all the code examples on this blog post, I did update my repo to account for the new changes, you can see the changes I made in this PR - https://github.com/BMayhew/playwright-demo/pull/30

One the coolest features that helped me seriously consider using Playwright on a new projects was the ability to run tests in parallel out of the box through the --shard command. All the details about this feature set is well explained in the Playwright docs, but what isn't explained is how do you handle reporting after you've run specs in a sharded fashion.

The first time I attempted to run tests with 4 shards this I ended up with 4 html reports and 4 slack messages from my run via GitHub Actions. In the next sections I'll cover the tooling I used in order to combine your html reports and combine the test results in order to send only 1 slack message with the results of your tests. Before I get going too far I have to call out Ben Fellows, specifically this Merging Playwright Reports in GitHub Actions Workflows After Sharding. This article got me in the right direction for a solution on how to combine summary.json results.

💡

All the code samples can be found within this repo https://github.com/BMayhew/playwright-demo

GitHub Actions Configuration

For the GitHub Action we will split up our run into 3 separate jobs that are all reliant on one another using the needs job feature. I'll call these jobs:

install
tests
merge

The install job will consist of ensuring that all the dependencies are installed and cached. The test job will be responsible for running the playwright command with shards (so multiple jobs will run) and generating the playwright report results. The merge job is responsible for getting all the sharded reports and combining them, along with sending the results somewhere, slack for example.

GitHub Actions

The first section of the GitHub Actions you need to add is when/how the job will be run. In my example, it will run whenever there is a pull_request opened, when a different repository kicks off this job, or if I manually kick it off with a workflow_dispatch from the GitHub actions page.

name: Playwright API Checks

on: 
  pull_request:
  repository_dispatch:
  workflow_dispatch:
    inputs:
      app_url:
        description: "APP URL, to run tests against"
        required: true
        default: https://www.automationexercise.com
permissions:
  contents: write
  pages: write

Install job

The install job can be found below. A few things to note, I am using outputs in order to expose certain bits of information to the following job. The next job runs on a completely different machine so doing this is necessary.

jobs:
  install:
    permissions:
      contents: read
      pages: write      
      id-token: write  
    timeout-minutes: 60
    runs-on: ubuntu-latest
    outputs:
      playwright_version: ${{ steps.set-env.outputs.PLAYWRIGHT_VERSION }}
      started_at: ${{ steps.set-env.outputs.STARTED_AT}}
      app_url: ${{ steps.set-env.outputs.APP_URL}}
      pull_request_url: ${{ steps.set-env.outputs.PULL_REQUEST_URL }}

    env:
      APP_URL: ${{ github.event.inputs.app_url }}
      PULL_REQUEST_URL: ${{ github.event.pull_request._links.html.href }}

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16

      - name: Set current date as env variable
        run: echo "STARTED_AT=$(date +%s)" >> $GITHUB_ENV

      - name: Get installed Playwright version
        id: playwright-version
        run: echo "PLAYWRIGHT_VERSION=$(node -e "console.log(require('./package-lock.json').dependencies['@playwright/test'].version)")" >> $GITHUB_ENV
      
      - name: Cache playwright binaries
        uses: actions/cache@v3
        id: playwright-cache
        with:
          path: |
            ~/.cache/ms-playwright
          key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}
      - run: npx playwright install --with-deps
        if: steps.playwright-cache.outputs.cache-hit != 'true'
      - run: npx playwright install-deps
        if: steps.playwright-cache.outputs.cache-hit != 'true'

      - name: Cache node_modules
        uses: actions/cache@v3
        id: node-modules-cache
        with:
          path: |
            node_modules
          key: modules-${{ hashFiles('package-lock.json') }}
      - run: npm ci --ignore-scripts
        if: steps.node-modules-cache.outputs.cache-hit != 'true'
      
      - name: Create Output for ENV Variables
        id: set-env
        run: |
          echo "PLAYWRIGHT_VERSION=${{env.PLAYWRIGHT_VERSION}}" >> $GITHUB_OUTPUT
          echo "STARTED_AT=${{env.STARTED_AT}}" >> $GITHUB_OUTPUT   
          echo "APP_URL=${{env.APP_URL}}" >> $GITHUB_OUTPUT        
          echo "PULL_REQUEST_URL=${{env.PULL_REQUEST_URL}}" >> $GITHUB_OUTPUT

Test job

The test job is run next, as you notice below we are utilizing the GitHub Actions Matrix functionality to run 4 test jobs. The steps include:

setting environment variables
checking out the code
restoring the cached node modules
restoring the cached playwright dependencies
executing the tests utilizing the matrix indexes (1-4 in the example below)
copying the summary.json report to the playwright-report folder (for merging)
uploading the playwright-report directory to report-${matrix-index} in GitHub
setting all outputs so they can be passed to the next job

 tests:
    name: Run Playwright Tests (${{ matrix.shardIndex }}/${{ strategy.job-total }})
    needs: install
    timeout-minutes: 60
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    outputs:
      playwright_version: ${{ steps.set-env.outputs.PLAYWRIGHT_VERSION }}
      started_at: ${{ steps.set-env.outputs.STARTED_AT}}
      app_url: ${{ steps.set-env.outputs.APP_URL}}
      pull_request_url: ${{ steps.set-env.outputs.PULL_REQUEST_URL }}

    env:
      PLAYWRIGHT_VERSION: ${{ needs.install.outputs.playwright_version }}
      STARTED_AT: ${{ needs.install.outputs.started_at }}
      APP_URL: ${{ needs.install.outputs.app_url }}
      PULL_REQUEST_URL: ${{ needs.install.outputs.pull_request_url }}

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3

      - name: Cache node_modules
        uses: actions/cache@v3
        with:
          path: |
            node_modules
          key: modules-${{ hashFiles('package-lock.json') }}

      - name: Cache Playwright
        uses: actions/cache@v3
        with:
          path: |
            ~/.cache/ms-playwright
          key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}

      - name: Install OS Dependencies (required for Webkit)
        run: npx playwright install --with-deps
      
      - name: Set APP_URL if not passed in
        if: env.APP_URL == null
        run: | 
          echo "APP_URL=https://www.automationexercise.com" >> $GITHUB_ENV  
      
      - name: Run Playwright tests
        run: APP_URL=${{ env.APP_URL}} npx playwright test --grep-invert @axe --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --workers=1

      - name: Copy Summary to playwright-report/ folder
        if: always()
        run: cp summary.json playwright-report/summary.json
        
      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: report-${{ matrix.shardIndex }}
          path: playwright-report/
          retention-days: 3

      - name: Create Output for ENV Variables
        if: always()
        id: set-env
        run: |
          echo "PLAYWRIGHT_VERSION=${{env.PLAYWRIGHT_VERSION}}" >> $GITHUB_OUTPUT
          echo "STARTED_AT=${{env.STARTED_AT}}" >> $GITHUB_OUTPUT   
          echo "APP_URL=${{env.APP_URL}}" >> $GITHUB_OUTPUT     
          echo "PULL_REQUEST_URL=${{env.PULL_REQUEST_URL}}" >> $GITHUB_OUTPUT

Now that all tests have been executed and all results have been uploaded to GitHub we are on to our final job.

Merge job

The final job that is run is the merge job. This job steps include:

setting environment variables
checking out the code
restoring the cached node modules
restoring the cached playwright dependencies
downloading all artifacts that have been uploaded report-* folders
removing html-report directory if it exists (in case this is a re-run)
runs custom command npm run merge (more about that in the next section)
reads the combined summary.json report and saves values to variables
uploads all combined artifacts (`html-report` folder).
publishes the combined html-report to GitHub Pages
sends MS Teams notification
sends customized SLACK notification

  merge:
    name: Merge Reports
    if: ${{ always() }}
    needs: [install,tests]
    timeout-minutes: 60
    permissions:
      contents: read
      pages: write
      id-token: write
    concurrency:
      group: "pages"
      cancel-in-progress: true
    env:
      PLAYWRIGHT_VERSION: ${{ needs.tests.outputs.playwright_version }}
      STARTED_AT: ${{ needs.tests.outputs.started_at }}
      APP_URL: ${{ needs.tests.outputs.app_url }}
      PULL_REQUEST_URL: ${{ needs.tests.outputs.pull_request_url }}
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3

      - name: Cache node_modules
        uses: actions/cache@v3
        with:
          path: |
            node_modules
          key: modules-${{ hashFiles('package-lock.json') }}

      - name: Cache Playwright
        uses: actions/cache@v3
        with:
          path: |
            ~/.cache/ms-playwright
          key: playwright-${{ hashFiles('package-lock.json') }}
 
      - uses: actions/download-artifact@v3
        with:
          path: playwright-report/      
      
      - name: Display structure of downloaded files
        run: ls -R

      - name: Remove Previous html-report directory if exists
        run: rm -rf playwright-report/html-report

      - name: Run Report Merge
        run: npm run merge

      - name: Display structure of Merged
        run: ls -R

      - name: Read Summary Report to Get Test Results
        if: always()
        run: |
          STATUS=$(cat ./summary.json | jq -r '.status')
          STATUS="$(echo $STATUS | sed 's/failed/failure/;s/passed/success/')"
          echo "STATUS=$STATUS" >> $GITHUB_ENV 
          PASSED=$(cat ./summary.json | jq -r '.passed[]' | tr '\n' ' ')
          echo "PASSED=$PASSED" >> $GITHUB_ENV 
          TIMEOUT=$(cat ./summary.json | jq -r '.timedOut[]' | tr '\n' ' ' | sed 's/ /--->TIMEOUT /g')
          FAILURES=$(cat ./summary.json | jq -r '.failed[]' | tr '\n' ' ')
          FAILURES+=$TIMEOUT
          echo "FAILURES=$FAILURES" >> $GITHUB_ENV

      - name: Copy Summary to html-report/ folder
        if: always()
        run: cp summary.json html-report/summary.json

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: html-report
          path: html-report/
          retention-days: 3
      
      - name: Setup Pages
        if: always()
        uses: actions/configure-pages@v2

      - name: Upload artifact
        if: always()
        uses: actions/upload-pages-artifact@v1
        with:
          path: html-report/
  
      - name: Deploy to GitHub Pages
        if: always()
        id: deployment
        uses: actions/deploy-pages@v1

      - name: Output time taken
        if: always()
        run: |
         echo "Duration: $(($(($(date +%s) - ${{ env.STARTED_AT }}))/60)) minute(s)"
         echo "DURATION=$(($(($(date +%s) - ${{ env.STARTED_AT }}))/60))" >> $GITHUB_ENV

      - name: Notify MS Teams on Success
        if: success()
        uses: jdcargile/[email protected]
        with:
          github-token: ${{ github.token }} # this will use the runner's token.
          ms-teams-webhook-uri: ${{ secrets.MSTEAMS_WEBHOOK }}
          notification-summary: Results &#x2705; ${{ env.PASSED }} | &#x274C; ${{ env.FAILURES }}
          notification-color: 28a745
          timezone: America/Chicago

      - name: Notify MS Teams on Failure
        if: failure()
        uses: jdcargile/[email protected]
        with:
          github-token: ${{ github.token }} 
          ms-teams-webhook-uri: ${{ secrets.MSTEAMS_WEBHOOK }}
          notification-summary: Results &#x2705; ${{ env.PASSED }} | &#x274C; ${{ env.FAILURES }}
          notification-color: dc3545
          timezone: America/Chicago

      - name: Send Slack Notification
        if: always()
        uses: 8398a7/action-slack@v3
        with:
          status: custom
          fields: repo,eventName,workflow,job,took
          custom_payload: |
            {
              attachments: [{
                color: '${{ env.STATUS }}' === 'success' ? 'good' : 'danger',
                title: `Playwright Demo Automation Results :test_tube:`,
                fields: [{
                  title: 'Site Under Test',
                  value: '${{ env.APP_URL }}',
                  short: true
                },
                {
                  title: 'Triggered By',
                  value: [{'origin': 'pull_request', 'new': 'Pull Request'}, {'origin': 'schedule', 'new': 'Schedule'}, {'origin': 'repository_dispatch', 'new': 'Deploy'}, {'origin': 'workflow_dispatch', 'new': 'GitHub Actions'}].find(item => item.origin === `${process.env.AS_EVENT_NAME}`).new || `${process.env.AS_EVENT_NAME}`,
                  short: true
                },
                {
                  title: 'Repo',
                  value: `${process.env.AS_REPO}`,
                  short: true
                },
                {
                  title: 'Execution Time',
                  value: `Took ${{ env.DURATION }} minute(s)`,
                  short: true
                },
                {
                  title: 'Workflow',
                  value: `${process.env.AS_WORKFLOW}`,
                  short: true
                },
                {
                  title: 'Total Tests',
                  value: (`${{ env.FAILURES }}`.match(/.spec.ts/g) || []).length + (`${{ env.PASSED }}`.match(/.spec.ts/g) || []).length,
                  short: true
                },
                {
                  title: 'Pull Request',
                  value: `${{ env.PULL_REQUEST_URL }}`,
                  short: false
                },
                {
                  title: 'Failures',
                  value: `${{ env.FAILURES }}` === '' ? 'No failures' : `${{ env.FAILURES }}`.match(/.spec.ts/g).length > 10 ? `Too many failures to print. Please go to GitHub to see full list of failures` : '```${{ env.FAILURES }}```'.replace(/ /g, '\n'),
                  short: false
                }]
              }]
            }  
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} # required
          MATRIX_CONTEXT: ${{ toJson(matrix) }}

npm run merge command

The command a new command that I've added uses ts-node to execute a typescript command.

npm install -D ts-node

Once installed you are now about to run typescript files form your package.json script. The below commands for merge runs the lib/metrics/mergeReports.ts file that you'll need to create to merge both your html reports and your summary.json reports.

    "merge": "npx ts-node lib/metrics/mergeReports.ts"

mergeReports.ts

Go ahead and create a file in directory /lib/metrics/ named mergeReports.ts. Within this file add the following code.

// lib/metrics/mergeReports.ts

import fs from "fs";
import path from "path";
import { mergeHTMLReports } from "playwright-merge-html-reports";
import { mergeSummary } from "playwright-merge-json-summary-reports";

const reportPathsToMerge = fs
  .readdirSync(process.cwd() + "/playwright-report", { withFileTypes: true })
  .filter((item) => item.isDirectory())
  .map(({ name }) => path.resolve(process.cwd() + "/playwright-report", name));

async function runReport(paths: string[]) {
  // merges the summary.json in each report-x folder and saves a summary.json to root directory
  await mergeSummary(paths);

  // merges html reports and saves to /html-report
  await mergeHTMLReports(paths, {
    outputFolderName: "html-report",
  });
}

runReport(reportPathsToMerge);

You will notice there are two imports in the above file that will also need to be installed into your playwright project. The links for each of these can be found below. These npm packages will handle building a json summary report (in which we use as a part of the GitHub actions step where you see summary.json), and merging both the summary.json and

npm install playwright-merge-html-reports --save-dev
npm install playwright-merge-json-summary-reports --save-dev

Once everything is installed and the files exist, the GitHub action should be in good working order. More details on dependencies can be found using the links below

playwright-json-summary-reporter

Taking a step back you will still need to install the playwright-json-summary-reporter. The below command will install and add the package to your package.json file.

npm install playwright-json-sumary-reporter --save-dev

This guide will have a more in-depth walk through but from a high level, add these lines to your playwright-config.ts file, specifically within the reporter array. This will ensure that when the playwright tests finish a summary.json file is generated.

 reporter: [
    ['playwright-json-summary-reporter'],
    ['html'],
    ['dot']
  ],

The outcome

We have a combined summary.json report that can be used to generate a single slack message see below for a pass and a failure.

Below is an screenshot of a successful GitHub Action running.

We also have a single html report that is combined from the different sharded runs, it will be named html-report. the report-* are reports that were output from each sharded run.

The way we have built this, we will only have to update the GitHub action file, in oder to scale up our sharding as high as we want, and we will be able to fully enjoy faster feedback loops. As that is the whole point of all this!

Source: https://blog.jerrycodes.com/pytest-split-and-github-actions/

Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below.

GitHub Actions Configuration

GitHub Actions

Install job

Test job

Merge job

npm run merge command

mergeReports.ts

playwright-json-summary-reporter

The outcome

Sign up for more like this.