Is it possible to run only Playwright Tests that changed in GitHub actions on a pull request?

Butch Mayhew

Apr 10, 2023 • 4 min read

Have you ever been reviewing a peer's pull request in GitHub, and asked yourself does this Playwright check even pass? Well, I have, and that is what inspired this post. In my workplace, I want to prevent failing tests from entering the main automation branch, but I also didn't want every single spec file running causing a feedback delay, so this is how I solved my problem.

In my context, I use dedicated repositories for test automation, mainly for regression testing. I have a main code branch, and any new code that will be added to the repository will be merged in through a pull request. This requires the developer to create a new branch from main, write, edit, or delete code and create a pull request to the main branch. During this pull request event when we will run this logic.

The benefits for me include:

When I initially create a pull request I can have confidence that the tests I've written and run locally will also pass in CI.
My peers which I requested a code review on my pull request can quickly see the tests that I modified have run and are passing.
For test suites with 100s or 1000s of tests, only running the changed files on pull request will provide faster feedback rather than waiting on all the tests to run.

The logic consists of getting a list of files that changed using git diff between the base branch main and the head branch my_new_branch, taking that list and getting file names that contain ".spec.ts", adding a space, and saving them to an environment variable named CHANGED. Using the git diff command it is important that the GitHub action command knows about the main branch or other branches, it is important to set the actions/checkout with fetch-depth: 0. This will get all history of the branch. As your repository grows you can modify this value, but it will likely fail using the default of fetch-depth: 1 with the error fatal: Invalid revision range.

Below is an example of a GitHub actions .yml file that shows how this can be accomplished.

# .github-ci.yml

name: Playwright API Checks

on:
  pull_request:
  workflow_dispatch:
    inputs:
      base_url:
        description: "URL, to run tests against"
        required: true
        default: https://automationintesting.online/

jobs:
  playwright-automation-checks:
    timeout-minutes: 10
    runs-on: ubuntu-latest

    env:
      BASE_URL: ${{ github.event.inputs.base_url }}
      CHANGED: ""

    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0      
      
      - uses: actions/setup-node@v3
        with:
          node-version: 16

      - name: Install node_modules and Playwright
      - run: npm ci --ignore-scripts
      - run: npx playwright install --with-deps
      - run: npx playwright install-deps

      - name: Set BASE_URL if not passed in
        if: env.BASE_URL == null
        run: |
          echo "BASE_URL=https://automationintesting.online/" >> $GITHUB_ENV

      - name: Create Test List if pull_request
        if: github.event.pull_request
        run: |
          echo "Creating a list of tests that have changed"
          FILES=$(git diff --name-only ${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }} | grep ".spec.ts" | tr '\n' ' ')
          echo "CHANGED=$FILES" >> $GITHUB_ENV

      - name: Run Playwright tests
        run: |
          echo "CHANGED = ${{ env.CHANGED }}"
          echo "The github event is: ${{ github.event_name }}"
          URL=${{ env.BASE_URL}} npx playwright test --workers=1 ${{ env.CHANGED }}

One thing to note is if there are no files returned that match spec.ts, the CHANGED variable will be empty when running the npx playwright test --workers=1 ${{ env.CHANGED}} command, all the tests will run (which for me is just something I've accepted and am ok with).

One con with this approach is if the feature branch you are committing to isn't up to date with the main branch, that you have a pull request to, any changes that are in the main branch and aren't in your branch will also get picked up in the git diff command. In my first solution (below) using the GitHub API, you won't have this issue. The con wasn't a big enough issue for me working on a small team, but if you work with a larger team implementing the GitHub API approach will more than likely make more sense.

My First Solution

When I first implemented this functionality, my Create Test List if pull_request section looked quite different. The code is below, I utilized the GitHub API, making a curl request to get a full list of files that changed. This required me to create a personal access token in GitHub and add it as a secret in GitHub Actions.

      - name: Create Test List if pull_request
        if: github.event.pull_request
        run: |
          echo "Creating a list of tests that have changed"
          FILES=$(curl --header "Authorization: Bearer ${{ secrets.REPO_ACCESS_TOKEN }}" --request GET 'https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files' | jq -r '.[] | .filename' | grep "spec.ts" |  tr '\n' ' ')
          echo "CHANGED=$FILES" >> $GITHUB_ENV

What's pretty neat is the open API peer review GitHub Action I set up on my previous pull request, actually alerted me to a better way. You can see the conversation I had with the Open AI bot at the link below. One thing I had to add to get things to work was the fetch-depth, by default I was getting the error I shared above.

I will say blindly following the Open AI's comments on everything is a horrible idea, as out of the 40 or so comments, I only took action on 2-3 of them, some of the suggestions would have broken things. The rough cost of the feedback on the p/r above was $0.15.

Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below.

The benefits for me include:

My First Solution

Sign up for more like this.