[Update v1.46] Is it possible to run only Playwright Tests that changed in GitHub actions on a pull request?
Good news everyone! This is now possible to do without any external scripts. It is all baked into the Playwright CLI.
Original solution
If you've been subscribed for some time you may have remembered the original solution. It included getting a list of the file names by running a narly git diff command and piping the results to grep and tr commands. Storing those file names to a CHANGED environment variable, and passing that into the npx playwright test ${{ env.CHANGED }}.
Updated solution
There was a flaw with my approach that I just accepted as I didn't see an easy way to solve the problem. I was only checking and building the CHANGED list of tests run from the *.spec files. This list didn't include any page object files, helpers, fixtures, or data factory files. This was a known risk in my case, but with the built in playwright command they have solved this problem. See version 1.46 release notes below.
New CLI option --only-changed will only run test files that have been changed since the last git commit or from a specific git "ref". This will also run all test files that import any changed files.Running pre-commit and post-commit
One thing to note with this solution, the command --only-changed by itself will only run files that have been changed, but not yet committed. This is important as you commit files to your local branch this command will not pick up those test changes after committed.
In order to pick up changes on committed files you will need to pass in a reference branch to check against. --only-changed=main for example will check your local git changes and do a diff against the main branch and run those changes.
Example on playwright-api-test-demo
Let's update our playwright-api-test-demo repo with the changes both in our package.json file to create a new npm script npm run changed.
// package.json
...
"scripts": {
...
"changed": "npx playwright test --only-changed"
},
We'll also update our on-pr-files-changed.yml file removing all the old CHANGED logic we previously implemented. The main thing we are adding is the --only-changed=origin/$GITHUB_BASE_REF. Using the $GITHUB_BASE_REF here instead of main as the branch will prevent an error I was receiving when running in CI when I was using --only-changed=origin/main as the release notes demonstrated. The specific error I received is below.
Error: Cannot detect changed files for --only-changed mode:
git diff main --name-only
fatal: ambiguous argument 'main': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Error: Error: Cannot detect changed files for --only-changed mode: git diff main --name-only
fatal: ambiguous argument 'main': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
The changes that worked for me were adding the fetch-depth: 0 (the default is 1) and adding the --only-changed flag in my npx playwright test command.
name: Changed Files Playwright API Checks
on:
pull_request:
branches:
- main
...
jobs:
playwright-automation-checks:
...
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
...
- name: Run Playwright tests
run: |
npx playwright test --only-changed=origin/$GITHUB_BASE_REF
....
The full example can be found below.
// on-pr-files-changed.yml
name: Changed Files Playwright API Checks
on:
pull_request:
branches:
- main
permissions:
contents: write
pages: write
id-token: write
jobs:
playwright-automation-checks:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
timeout-minutes: 10
runs-on: ubuntu-latest
env:
BASE_URL: ${{ github.event.inputs.base_url }}
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- uses: actions/setup-node@v3
with:
node-version: 16
- name: Cache node_modules
uses: actions/cache@v3
id: node-modules-cache
with:
path: |
node_modules
key: modules-${{ hashFiles('package-lock.json') }}
- run: npm ci --ignore-scripts
if: steps.node-modules-cache.outputs.cache-hit != 'true'
- name: Get installed Playwright version
id: playwright-version
run: echo "PLAYWRIGHT_VERSION=$(node -e "console.log(require('./package-lock.json').dependencies['@playwright/test'].version)")" >> $GITHUB_ENV
- name: Cache playwright binaries
uses: actions/cache@v3
id: playwright-cache
with:
path: |
~/.cache/ms-playwright
key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}
- run: npx playwright install --with-deps
if: steps.playwright-cache.outputs.cache-hit != 'true'
- run: npx playwright install-deps
if: steps.playwright-cache.outputs.cache-hit != 'true'
- name: Run lint
run: npm run lint
- name: Run prettier
run: npm run prettier
- name: Set BASE_URL if not passed in
if: env.BASE_URL == null
run: |
echo "BASE_URL=https://automationintesting.online/" >> $GITHUB_ENV
- name: Run Playwright tests
run: |
echo "The github event is: ${{ github.event_name }}"
URL=${{ env.BASE_URL }} npx playwright test --grep-invert @unsatisfactory --reporter=github --workers=1 --only-changed=origin/$GITHUB_BASE_REF
Using only-changed on monorepos
One thing to note in my research Pramod Yadav (go give him a follow on LinkedIn) reported a Bug in the GitHub Issues section of the Playwright repository. He attempted to implement this within a monorepo and found there were some extra steps to get things working. Specifically adding git config --global --add safe.directory '*' as a command that is run after checking out the code and before the playwright test execution.
name: Changed Files Playwright API Checks
on:
pull_request:
branches:
- main
...
jobs:
playwright-automation-checks:
...
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Mark repo as safe
run: git config --global --add safe.directory '*'
...
- name: Run Playwright tests
run: |
npx playwright test --only-changed=origin/$GITHUB_BASE_REF
....
Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below, and be sure to leave a ❤️ to show some love.