Sponsored Link
Looking for a unified way to  view,  monitor, and  debug Playwright Test Automation runs?
Try the Playwright Dashboard by today. Use the coupon code PWSOL10 for a 12 month 10% discount.

How do you send test results to a Microsoft Teams channel when Playwright tests finish?

Butch Mayhew

• 8 min read

In the previous article I showed off how to use the playwright-slack-report package to send slack notification from within your Playwright Test run. One of the followup questions people were asking was is it possible to send notifications to Microsoft Teams as well? At the moment this isn't possible with the playwright-slack-report package, but you are in luck, as I'll walk you through how to via GitHub Actions.

It is quite easy to send a notification via GitHub action if a job passed or failed. This is fine if everything passed, but once failures begin to happen, you start asking other questions. How many tests failed, which tests failed, etc. In this walk through we will only be covering how many tests passed and how many failed similar to the playwright-slack-report plugin, but know that it's possible to send any information you have available to slack as needed!

The Method we will utilize includes:

  • Creating an MS Teams channel web hook
  • Saving the web hook within GitHub actions secrets
  • Creating a Playwright Test Summary Class
  • Using Custom Reporter in playwright.config.ts that creates summary.json
  • Within CI, using jq to parse the summary.json
  • Passing the parsed data into jdcargile/[email protected] GitHub action steps to send the notifications

Creating an MS Teams Channel Webhook

The steps are pretty straight forward as long as you have the correct access to create/manage connectors. Within the channel you'd like the notifications to go to, click the three dots and select 'Connectors'.  

Channel Name -> ... -> Connectors

From there search for 'Incoming Webhook' and select Configure

From here you can give it a name, upload an image, and Select 'Create'. This should load a new box which gives you the Webhook url that will allow you to post to the MS Teams channel. This Webhook should be treated like a password.

If you get stuck on any of the above steps, below is a link to the official docs on how to create an Incoming Webhook.

Create an Incoming Webhook - Teams
Create Incoming Webhook to Teams app and post external requests to Teams. Remove Incoming Webhook. Sample code(C#, Node.js) to send card using Incoming Webhook.
Microsoft Learnlaujan

Saving the web hook within GitHub actions secrets

Visit the repo you plan on running your Playwright Tests from and visit the 'Settings' page. If you don't see this as an option, ask the administrator of the repository for access.

Expand Secrets, and Click Actions, to manage the Github Actions Secrets.

From this page you can add a new repository secret name and secret, and add it.

Now everything is configured from the GitHub and MS Teams side, let's jump into the Playwright code!

Creating a Playwright Test Summary Class

I won't do a line by line on the below code, but I am utilizing the @playwright/test/reporter class. I encourage you to review the docs to learn more about the flexibility Playwright offers! The output of this file will be summary.json which is below. I created this file in /lib/metrics folder as I'm primarily using this summary to save metrics to view trends.

// ./lib/metrics/summaryReporter.ts

import * as fs from "fs";
import {
  TestCase,
  TestResult,
  Reporter,
  FullResult,
} from "@playwright/test/reporter";

export interface Summary {
  durationInMS: number;
  passed: string[];
  skipped: string[];
  failed: string[];
  warned: string[];
  timedOut: string[];
  status: FullResult["status"] | "unknown" | "warned" | "skipped";
}

class SummaryReporter implements Reporter, Summary {
  durationInMS = -1;
  passed: string[] = [];
  skipped: string[] = [];
  failed: string[] = [];
  warned: string[] = [];
  timedOut: string[] = [];

  status: Summary["status"] = "unknown";
  startedAt = 0;

  onBegin() {
    this.startedAt = Date.now();
  }

  onTestEnd(test: TestCase, result: TestResult) {
    const title = [];
    const fileName = [];
    let clean = true;
    for (const s of test.titlePath()) {
      if (s === "" && clean) continue;
      clean = false;
      title.push(s);
      if (s.includes("spec.ts")) {
        fileName.push(s);
      }
    }

    // This will publish the file name + line number test begins on
    const z = `${fileName[0]}:${test.location.line}:${test.location.column}`;

    // Using the t variable in the push will push a full test test name + test description
    const t = title.join(" > ");

    const status =
      !["passed", "skipped"].includes(result.status) && t.includes("@warn")
        ? "warned"
        : result.status;
    this[status].push(z);
  }

  onEnd(result: FullResult) {
    this.durationInMS = Date.now() - this.startedAt;
    this.status = result.status;

    // removing duplicate tests from passed array
    this.passed = this.passed.filter((element, index) => {
      return this.passed.indexOf(element) === index;
    });

    // removing duplicate and flakey (passed on a retry) tests from the failed array
    this.failed = this.failed.filter((element, index) => {
      if (!this.passed.includes(element))
        return this.failed.indexOf(element) === index;
    });

    fs.writeFileSync("./summary.json", JSON.stringify(this, null, "  "));
  }
}

export default SummaryReporter;

The summary.json is below, which we will be parsing in the GitHub actions step.

{
  "durationInMS": 35490,
  "passed": [
    "api/create-user-account.spec.ts:34:3",
    "api/create-user-account.spec.ts:55:3",
    "api/delete-user-account.spec.ts:47:3",
    "api/login.spec.ts:12:3",
    "api/login.spec.ts:37:3",
    "api/login.spec.ts:61:3",
    "api/login.spec.ts:84:3",
    "ui/internet.app/selectPresentElement.spec.ts:10:1",
    "ui/internet.app/selectPresentElement.spec.ts:35:1",
    "ui/loginUser.spec.ts:22:3",
    "ui/loginUserNoImages.spec.ts:23:3",
    "ui/loginUser.spec.ts:37:3",
    "ui/loginUserNoImages.spec.ts:40:3",
    "ui/registerUser.spec.ts:17:1"
  ],
  "skipped": [
    "ui/accessibility.spec.ts:4:6"
  ],
  "failed": [],
  "warned": [],
  "timedOut": [],
  "status": "passed",
  "startedAt": 1671335173320
}

Using Custom Reporter in playwright.config.ts that creates summary.json

This is a section of my playwright config where I am configuring the reporter. 

//playwright.config.ts

  globalTeardown: "./global-teardown",
  
  reporter: [["dot"], ["list"], ["html"], ["./lib/metrics/summaryReporter.ts"]],

I also found I had to add a step in the global-teardown.ts to wait for a few seconds in order for the summary.json file to get written to disk when running as a Github Action. If you skip this step you risk your Github Action failing, because the file doesn't exist on the next step.

//global-teardown.ts

async function globalTearDown() {
  //Currently have this wait in check so the /lib/metrics/summaryReporter.ts has time to write the summary.json
  setTimeout(function () {}, 5000);
}

export default globalTearDown;

Within CI, using jq to parse the summary.json

So we are at the step we are going to swap over to viewing the GitHub actions file, a minimal example is posted below in full for reference. Good news jq is installed on the default runners by default. If for some reason you are using a custom docker container you may need to install jq as a step in the GitHub action file.

After the Playwright Tests run, a summary.json file should exist within the GitHub action runner, we set 2 variables PASSED and FAILURES from the output of the jq commands.

The logic below is cat summary.json (reads the file)  the '|' (pipe) says send that data to jq with the -r option saying output to 'string' (not json). The code section '.passed | length' says get the passed object (which includes an array of tests that passed, and get the array length (how many values exist).  If you want to play around with jq check out https://jqplay.org/ which allows you to paste in json and parse in realtime.

     - name: Run Playwright tests
        run: npm run test

      - name: Read Summary Report to Get Test Results
        if: always()
        run: |
          PASSED=$(jq -r '.passed | length' ./summary.json)
          echo "PASSED=$PASSED" >> $GITHUB_ENV 
          FAILURES=$(jq -r '.failed | length' ./summary.json)
          echo "FAILURES=$FAILURES" >> $GITHUB_ENV

Passing the parsed data into GitHub action steps to send the MS Teams notifications

Now that we have the PASSED and FAILURES counts we are able to send this data to MS Teams Channel. There are a ton of different options out there to accomplish this, but I picked one of the more simple solutions - GitHub Action - jdcargile/[email protected].  The documentation can be found at the link if you want to customize the output further. I decided it best to have 2 different steps. The first step will only occur if the overall GitHub actions status is 'Success' while the 2nd step will only run if the overall GitHub actions status is 'Failure'. The only difference between the two are the names, and the notification-colors.

    - name: Notify MS Teams on Success
        if: success()
        uses: jdcargile/[email protected]
        with:
          github-token: ${{ github.token }} # this will use the runner's token.
          ms-teams-webhook-uri: ${{ secrets.MSTEAMS_WEBHOOK }}
          notification-summary: Results ✅ ${{ env.PASSED }} | ❌ ${{ env.FAILURES }}
          notification-color: 28a745
          timezone: America/Chicago

      - name: Notify MS Teams on Failure
        if: failure()
        uses: jdcargile/[email protected]
        with:
          github-token: ${{ github.token }} 
          ms-teams-webhook-uri: ${{ secrets.MSTEAMS_WEBHOOK }}
          notification-summary: Results ✅ ${{ env.PASSED }} | ❌ ${{ env.FAILURES }}
          notification-color: dc3545
          timezone: America/Chicago
Example of Passing Tests and a Failing Tests

If you are looking for a codebase where this is implemented, look no further, this pull request has the changes (plus a few more) that allowed the MS Teams channel notifications.

GitHub Actions Minimal MS Teams Example

name: Playwright API Checks

on: 
  pull_request:

jobs:
  e2e-tests:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    env: 
      SLACK_BOT_USER_OAUTH_TOKEN: ${{ secrets.SLACK_BOT_USER_OAUTH_TOKEN }}

    steps: 
      - uses: actions/checkout@v3

      - name: Get installed Playwright version
        id: playwright-version
        run: echo "PLAYWRIGHT_VERSION=$(node -e "console.log(require('./package-lock.json').dependencies['@playwright/test'].version)")" >> $GITHUB_ENV
    
      - name: Cache playwright binaries
        uses: actions/cache@v3
        id: playwright-cache
        with:
          path: |
            ~/.cache/ms-playwright
          key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}
      - run: npm ci --ignore-scripts
      - run: npx playwright install --with-deps
        if: steps.playwright-cache.outputs.cache-hit != 'true'
      - run: npx playwright install-deps
        if: steps.playwright-cache.outputs.cache-hit != 'true'
   
      - name: Install Dependencies
        run: npm ci
      
      - name: Run Playwright tests
        run: npm run test
      
      - uses: actions/upload-artifact@v2
        if: always()
        with:
          name: playwright-report
          path: playwright-report/

      - name: Read Summary Report to Get Test Results
        if: always()
        run: |
          PASSED=$(jq -r '.passed | length' ./summary.json)
          echo "PASSED=$PASSED" >> $GITHUB_ENV 
          FAILURES=$(jq -r '.failed | length' ./summary.json)
          echo "FAILURES=$FAILURES" >> $GITHUB_ENV 

      - name: Notify MS Teams on Success
        if: success()
        uses: jdcargile/[email protected]
        with:
          github-token: ${{ github.token }} # this will use the runner's token.
          ms-teams-webhook-uri: ${{ secrets.MSTEAMS_WEBHOOK }}
          notification-summary: Results ✅ ${{ env.PASSED }} | ❌ ${{ env.FAILURES }}
          notification-color: 28a745
          timezone: America/Chicago

      - name: Notify MS Teams on Failure
        if: failure()
        uses: jdcargile/[email protected]
        with:
          github-token: ${{ github.token }} 
          ms-teams-webhook-uri: ${{ secrets.MSTEAMS_WEBHOOK }}
          notification-summary: Results ✅ ${{ env.PASSED }} | ❌ ${{ env.FAILURES }}
          notification-color: dc3545
          timezone: America/Chicago

Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below.


If this solution was helpful show some love!

Sign up for more like this.

Enter your email
Subscribe