Is it possible to override retry attempts on a specific spec file in Playwright?

Butch Mayhew

Aug 7, 2023 • 6 min read

This was a question that was recently asked via the Playwright Discord server in the #help-playwright channel. If you aren't there you should join now!

For today's solution I'll be using the https://practicesoftwaretesting.com/ website for the system under test. The pull request with the code added can be found at the link below.

The main topics being covered include test.describe.configure({ retries: x }), expect.poll(), and expect.toPass()

Before we dive into specific solutions I think it's worth asking yourself why you may need retries within your test suite? There are lots of folks who would argue that you should not retry tests within your suite but you should investigate every failure you receive and either fix the bug that exists or improve upon the flakey test to make it more stable.

Practically this is not always possible depending on your context. For my current context, we do rely on retries to help give us feedback. This is a choice and risk we are making, to complete 2 retries for each test within our suite. We do typically review the retries every so often to investigate on why the test originally failed, and work towards fixing logic in the tests that are discovered. Most of the time in my context, the flakiness is more from the application infrastructure we are testing against in our ephemeral and staging environments. Many of the errors I've faced happen to be application load balancer errors 502, 503 where a server took too long to respond. These environments for us are smaller than our production environments, and aren't always up for the task of handling all the traffic our automation tests throw at it gracefully. I've found our tests aren't flaky, our infrastructure is.

That being said, our automation checks have actually helped us identify a load balancer timing issue (down on the TCP/IP level), due to a huge influx of failures, that when re-ran passed. We only came to a solution after digging in to our "retried" tests and investigated with our SRE team to discover the misconfiguration between the application session timeout and the load balancer health check timeout.

Below are two great talks that have really shaped the way I think about Flakiness, and I encourage you to watch and really understand the risks of test retries.

With that disclaimer out of the way let's get into the solution!

test.describe.configure({ retries: x })

For the playwright.config.ts file I have retries set to 2. Within a specific test file I am able to override the retries by using test.describe.configure() at the beginning of the file (and outside of any test.describe() or test() blocks). The below code will set all tests within this file to retry 5 times. This is a very straight forward way to accomplish this task, but there are more ways as well!

import { test, expect } from "@playwright/test";

test.describe.configure({ retries: 5 });

test.describe("Testing retries ", () => {
  test.use({ storageState: ".auth/customer01.json" });

  test("login with valid customer credentials and validate brands page is unreachable", async ({
    page,
  }) => {
    await page.goto("/");

    expect(await page.getByTestId("nav-user-menu").innerText()).toContain(
      "Jane Doe"
    );
    await page.goto("/#/admin/brands");
    await expect(page.getByTestId("email")).toBeVisible();
    await expect(page.url()).toContain("/#/auth/login1");
  });
});

This solution will force you to group your tests together that need to have a more (or less) retries than the default values in the playwright.config.ts file.

It's worth calling out that you can solve a lot of the same problems the test retries solves with both expect.poll() or expect.toPass() functionality within your Playwright tests. Let's look at examples of each below.

expect.poll()

The expect.poll() is used to make any synchronous expect into an asynchronous polling one. In the below example in my repo if I update the assertion method to .toContain("/#/auth/login1") The code within the .poll(async.... will be re-run for 20 seconds (or shorter if it succeeds). This can be really useful to wrap functionality in the poll or in an example in a work project, I have the page.goto() outside of the poll (so it only attempts to go to a url once), and have a return page.url() returning but it happens after 2-3 re-directs so once the re-directs are complete and the correct page.url() is returned the poll is complete because the expect now passes. One thing to note, if an expect within the .poll( async()... fails the test will be failed. See below comment.

import { test, expect } from "@playwright/test";

test.describe("Testing poll ", () => {
  test.use({ storageState: ".auth/customer01.json" });

  test("login with valid customer credentials and validate brands page is unreachable", async ({
    page,
  }) => {
    await page.goto("/");

    expect(await page.getByTestId("nav-user-menu").innerText()).toContain(
      "Jane Doe"
    );

    await expect
      .poll(
        async () => {
          await page.goto("/#/admin/brands");
          
          // if the below assertion fails the test fails
          await expect(page.getByTestId("email")).toBeVisible();
          
          return page.url();
        },
        {
          timeout: 20_000,
        }
      )
      .toContain("/#/auth/login");
  });
});

expect.toPass()

The toPass() method is similar to poll, but the biggest difference is, that the .toPass() functionality will re-try until the timeout until all the assertions pass within the await expect(async().... block of code. If any of the assertions fail it will retry. I added comments in the section below to show that if the assertion fails it will retry the entire block. This is the biggest different between expect.poll() and expect.toPass().

import { test, expect } from "@playwright/test";

test.describe("Testing toPass ", () => {
  test.use({ storageState: ".auth/customer01.json" });

  test("login with valid customer credentials and validate brands page is unreachable", async ({
    page,
  }) => {
    await expect(async () => {
      await page.goto("/");

      expect(await page.getByTestId("nav-user-menu").innerText()).toContain(
        "Jane Doe"
      );
      await page.goto("/#/admin/brands");
      
      // if the below assertion fails, the entire block is retried
      await expect(page.getByTestId("email")).toBeVisible();
      
      await expect(page.url()).toContain("/#/auth/login");
    }).toPass({
      timeout: 10_000,
    });
  });
});

To wrap things up, re-running a test blindly is a huge risk. The whole point of our automated checks (or at least the point of mine) is to give me and the developers confidence that they haven't made the application worse from their changes, giving us confidence to release our code to production.

Thanks for reading! If you found this helpful, reach out and let me know on LinkedIn or consider buying me a cup of coffee. If you want more content delivered to you in your inbox subscribe below, and be sure to leave a ❤️ to show some love.

test.describe.configure({ retries: x })

expect.poll()

expect.toPass()

Sign up for more like this.