Running Visual Regression Tests with Storybook and Playwright for Free

The team behind Storybook offers a promising visual regression testing tool named Chromatic. Unfortunately, with 149 bucks a month, it’s not on the cheap side. Luckily, there are various free options. One of them is Playwright.

In this article, we’ll explore how to set up Playwright with Storybook to catch any regressions in our UI components.

Playwright and Storybook Setup

Although Playwright is primarily known for its end-to-end testing capabilities, it also has our back when it comes to creating screenshots of our UI components and comparing them with previous screenshots to detect any unforeseen changes to the styling of our components, aka visual regression testing.

For this tutorial, I assume you have Storybook already up and running. If not, please follow the official instructions.

To set up Playwright to run our visual regression tests, we first need to install Playwright:

npm init playwright@latest

Next, let’s make some changes to the default configuration:

// playwright.config.ts
import { defineConfig, devices } from "@playwright/test";

// We need to make the base URL configurable so we can
// change it when running tests in a Docker container.
const BASE_URL = process.env.TEST_BASE_URL || "http://localhost:6006";

export default defineConfig({
  // ...
  // Using the `html` reporter for visual diffing.
  reporter: process.env.CI ? "html" : "dot",
  // ...
  use: {
    baseURL: BASE_URL,
    // ...
  },
  // I recommend to run regression tests at
  // least for desktop and mobile devices.
  projects: [
    {
      name: "desktop",
      use: { ...devices["Desktop Chrome"] },
    },
    {
      name: "mobile",
      use: {
        ...devices["iPhone 15"],
      },
    },
  ],
  // Run your local dev server before starting the tests
  webServer: process.env.CI
    ? undefined
    : {
        command: "npm run dev",
        url: BASE_URL,
        reuseExistingServer: true,
      },
});

Now, let’s create a test file that opens all of our Storybook stories and runs visual regression tests on them:

// test/visual.spec.ts
import { expect, test } from "@playwright/test";

// This file is created by Storybook
// when we run `npm run build`
import storybook from "../storybook-static/index.json" with { type: "json" };

// Only run tests on stories, not other documentation pages.
const stories = Object.values(storybook.entries).filter(
  (e) => e.type === "story",
);

for (const story of stories) {
  test(`${story.title} ${story.name} should not have visual regressions`, async ({
    page,
  }, workerInfo) => {
    const params = new URLSearchParams({
      id: story.id,
      viewMode: "story",
    });

    await page.goto(`/iframe.html?${params.toString()}`);
    await page.waitForSelector("#storybook-root");
    await page.waitForLoadState("networkidle");

    await expect(page).toHaveScreenshot(
      `${story.id}-${workerInfo.project.name}-${process.platform}.png`,
      {
        fullPage: true,
        animations: "disabled",
      },
    );
  });
}

In this Playwright test file, we first load the storybook-static/index.json file containing all the information about the pages of our Storybook. Storybook automatically generates this .json file when we run npm run build. With the information provided by storybook-static/index.json, we can open particular stories directly via their <iframe> URLs.

Finally, we use Playwright’s toHaveScreenshot() method on the expect() instance to create a screenshot and compare it with an existing one (if there is any). Note that we generate a file name for the screenshots based on a) the story.id, b) workerInfo.project.name (e.g., desktop or mobile), and c) process.platform (e.g., linux).

# Filename of a screenshot created by the above test
button--primary-desktop-linux.png

With this simple setup, we can create screenshots of our UI components in Storybook and ensure no unintended visual changes happen. But there is a problem. There will definitely be slight visual differences between how specific browsers on certain operating systems render our components.

For example, Windows has a different font rendering than macOS, which has a different font rendering than Linux. That’s to be expected. For our users, this is not a problem; they’re used to the way their operating system renders websites. But it’s a problem when we want to run tests on different developers’ laptops or in our CI pipeline. Some of our colleagues might have Windows or macOS, and our CI pipeline most certainly runs on Linux.

This means the reference screenshots we take on our machine will lead to failing tests on the CI pipeline or our colleagues’ laptops. Luckily, there is a solution for this problem: Docker.

Running Storybook with Docker

Whenever we need to ensure that we can run our application in a controlled environment, Docker is the way to go. In the case of visual regression testing, running our tests not directly on the host system of our dev laptop but within a Docker container helps us keep screenshots consistent between multiple dev machines and CI pipelines.

So, let’s create a straightforward Docker image running Playwright and Chrome:

FROM node:20-bookworm

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci
RUN npx -y playwright@1.47.2 install --with-deps chromium webkit

ENTRYPOINT ["npx", "playwright"]

In this Dockerfile, we specify that we want to build an image based on the latest Node 20 bookworm image and then install npm dependencies and Playwright, including all the OS dependencies for Chromium and WebKit (note that I don’t install the dependencies to run the tests in Firefox or other browsers here because I don’t need cross-browser tests but your requirements might differ!). By specifying npx playwright as ENTRYPOINT, we ensure that we run Playwright immediately when starting the Docker image.

To build this Docker image, we can execute this command:

# Change `my-project-name-test-visual` to your needs
docker build -t my-project-name-test-visual . -f Dockerfile.test

Next, we can run our visual regression tests within Docker by executing the following command:

docker run --rm -it \
  -e TEST_BASE_URL='http://host.docker.internal:6006' \
  -e CI=true \
  -v ${PWD}/storybook-static:/app/storybook-static \
  -v ${PWD}/test:/app/test \
  -v ${PWD}/playwright.config.ts:/app/playwright.config.ts \
  my-project-name-test-visual test

When running this command for the first time, it will fail. This is expected behavior. See below for more information about the general workflow.

Setting the TEST_BASE_URL to http://host.docker.internal:6006 ensures the Docker container can reach Storybook running on our host machine. To run tests on a different machine, you must change the TEST_BASE_URL to fit your needs. Additionally, we mount the storybook-static and test directories and the playwright.config.ts file into the Docker container so it has access to all the files it needs to run our tests.

Visual Regression Testing Workflow

Now that we’ve set everything up so that we can run our tests and ensure consistent results across machines, we’re ready to take a closer look at the general workflow of how to do visual regression testing in practice.

1) Creating Reference Screenshots

First, run npm run build before running the visual regression tests the first time, and don’t forget to rerun it every time you add new stories.

When we first get started, we assume that all of our components look like they should look so we can go ahead and create our initial set of reference screenshots:

docker run --rm -it \
  -e TEST_BASE_URL='http://host.docker.internal:6006' \
  -e CI=true \
  -v ${PWD}/storybook-static:/app/storybook-static \
  -v ${PWD}/test:/app/test \
  -v ${PWD}/playwright.config.ts:/app/playwright.config.ts \
  my-project-name-test-visual test --update-snapshots

Note the --update-snapshots flag at the end of the command. This flag tells Playwright that we don’t want to compare existing screenshots with our current state, but we want to update existing screenshots or create new ones.

If you find it too tedious to copy and paste this long command, you can either add an alias or add it to the scripts section in your package.json file.

2) Make Changes to a Component

Let’s assume we need to update one of our existing components. With visual regression tests in place, what we do first after we’re finished making changes to our component is to rerun our tests with the expectation that they will fail.

docker run --rm -it \
  -e TEST_BASE_URL='http://host.docker.internal:6006' \
  -e CI=true \
  -v ${PWD}/storybook-static:/app/storybook-static \
  -v ${PWD}/test:/app/test \
  -v ${PWD}/playwright.config.ts:/app/playwright.config.ts \
  my-project-name-test-visual test

In this example, we changed the description text of our Card component to bold. And because the newly generated screenshot does not match the old one, the test fails. Now, there are two scenarios:

a) Only this single regression test is failing. In this case, we can update our screenshots so that they reflect the new look of our component:

docker run --rm -it \
  -e TEST_BASE_URL='http://host.docker.internal:6006' \
  -e CI=true \
  -v ${PWD}/storybook-static:/app/storybook-static \
  -v ${PWD}/test:/app/test \
  -v ${PWD}/playwright.config.ts:/app/playwright.config.ts \
  my-project-name-test-visual test --update-snapshots

b) Other tests also fail because our change affects other components, although we didn’t intend to affect them. In this case, we can take a closer look at the failing regression tests by opening the test report:

npx playwright show-report

With this newfound knowledge, we can revisit and refine our changes to ensure they only affect the particular component we intended. After that, we can rerun our regression tests until we’re happy with the result.

Wrapping It Up

Visual regression testing can help ensure that essential styles haven’t unexpectedly changed after updating particular components. With the help of Playwright, setting up visual regression tests is straightforward, and we can quickly create our own free visual regression test suite.

If you liked this article and are also interested in all the other types of tests for web applications, check out my book.