GitHub Actions: Part 3 - Shard your playwright tests for blazing speed

Running your on multiple machines simultaneously

You can read my previous articles, where I explained what GitHub Actions are and how you can set up the Actions to execute your playwright tests.

GitHub Actions: Part 1 - Run your playwright tests on GitHub
GitHub Actions: Part 2 - Schedule and showcase your Playwright tests with GitHub pages

By default, playwright executes your tests parallel with the number of threads you define on your playwright.config. Each thread requires CPU and memory. On one project, I had to increase the timeouts in the pipeline when I set up 3 threads. However, I got tired of increased timeouts, and the tests needed more time to finish. So, I changed the approach and set it up to execute on different machines. This is called sharding.

In this article, I will explain how you can shard your tests to execute your test parallel on different machines. In my next article, I will explain how you can execute your test with Microsoft Playwright Testing, a preview service that can execute your test in different browsers on the cloud.

When you shard your test, the total of your tests is divided into equal parts, and each shard will execute the same number of tests. For example, if you have 12 tests and select 3 shards, each shard will

• Perform 4 tests

• Publish the HTML report as a blob file

After, you need to publish each blob report as an artifact and merge all blob reports into one file.

Sharding 12 tests in 3 shards

To shard your tests, you must execute the following playwright command, splitting your tests into 3 parts.

npx playwright test --shard=1/3
npx playwright test --shard=2/3
npx playwright test --shard=3/3

This command will merge the blob reports generated in one file

npx playwright merge-reports --reporter html ./all-blob-reports

How to create a GitHub Action to shard your playwright tests

GitHub allows you to define variations of your jobs with a matrix that will execute the job with each value in the matrix. For example, I will set a shard index of 1,2,3 and a shard total of 3 to generate the combination of 1/3, 2/3 and 3/3

You must define a matrix with the number of shards you want to execute your tests.

matrix:
    shardIndex: [1, 2, 3]
    shardTotal: [3]

1. On your GitHub Repository, click on Actions.

2. Click on the New workflow button.

3. Click on the link: setup a workflow yourself →

4. As I explained in previous articles, you need to:

   1. Install npm dependencies

   2. Install playwright browsers

   3. Execute your playwright tests with the matrix values defined

       - name: Run Playwright tests
         run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=blob
      

   4. Upload each blob report as an artifact

   5. Execute the merge playwright command

   6. Upload the final HTML report as an artifact

name: Shard tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  playwright-tests:
    name: Run tests
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3]
        shardTotal: [2]
    env:
      BASE_URL: ${{vars.BASE_URL}}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=blob

      - name: Upload blob report to GitHub Actions Artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 5

  merge-reports:
    # Merge reports after playwright-tests, even if some shards have failed
    if: always()
    needs: [playwright-tests]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4

      - name: Install dependencies
        run: |
          npm ci

      - name: Download blob reports from GitHub Actions Artifacts
        uses: actions/download-artifact@v4
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true

      - name: Merge into HTML Report
        run: npx playwright merge-reports --reporter html ./all-blob-reports

      - name: Upload HTML report
        uses: actions/upload-artifact@v4
        with:
          name: html-report-${{ github.run_attempt }}
          path: playwright-report
          retention-days: 3

When you create a PR or merge your PR, the pipeline will be executed, and you will see how the tests are executed in parallel.

GitHub Actions result of executing playwright tests with 3 shards: 1 pass and 2 failed.

You can download the HTML report as an artifact, and the execution time will be reduced, which is important to execute your test faster.

You can check my Playwright Sample with the example of shard tests and other playwright samples.

Thank you for reading. Feel free to suggest any topics or questions. If the article is helpful, you can share it. Enjoy testing!!