Execute your playwright test on the cloud

Microsoft Playwright Testing (Beta) and PerfAgents Uncloud

Feb 17, 2025

You can execute your tests in cloud browsers with Microsoft Playwright Testing up to 100 workers, and with PerfAgents on Aws Marketplace, you can monitor the execution of your tests.

Scale your test with Microsoft Playwright Testing.

It’s a Microsoft service that is still in preview but allows you to execute your tests in multiple browsers and operating systems. You can check the results of your test by workspace. It’s useful when you have a lot of tests and want faster results.

How To Setup

Register at https://playwright.microsoft.com
Create your workspace. Fill in the following info
1. A workspace name
2. Azure subscription
3. Region to store the data
  Workspace creation
You will see a page with the steps to connect to your space. You can connect directly from VS Code, the command line, or a pipeline.
In your playwright test project, execute the following command to install the playwright service testing package.

npm init @azure/microsoft-playwright-testing

This command will create a playwright.service.config.service.ts with the details to connect to the cloud service.

Install the Azure CLI to connect to Azure.

Windows with an installer
Macbook with Homebrew command

brew update && brew install azure-cli

az login

Add environment variable PLAYWRIGHT_SERVICE_URL with your workspace URL. Add it with a .env file or a Bash/PowerShell command.

export PLAYWRIGHT_SERVICE_URL=wss://eastus.api.playwright.microsoft.com/accounts/eastus_7854...

How to execute your tests

VS Code

By selecting the playwright. service.config.ts created, you can test only one test to check that the setup is correct

When finished, it will show the link with your test results.

Microsoft Playwright Testing Test run execution

You can check the details of your test and see if it is failing. It will be uploaded to the dashboard if you set up your playwright.config.ts file to include the video, screenshot, and trace.

  use: {
    trace: 'on-first-retry',
    video:'retain-on-failure',
    screenshot:'on'
  }

Command line

You can execute your test by selecting the playwright.service.config.ts and the number of workers.

npx playwright test --config=playwright.service.config.ts --workers=20

The tests are being executed. When the tests are executed completely, you will see the link with the results of your tests.

GitHub Actions

With GitHub Actions, you have two options:

Azure token, but the suggestion is to use Microsoft Entra ID
Microsoft Entra ID is a new service that allows your employees to connect your services with
- AZURE_CLIENT_ID Your Application (Client) ID
- AZURE_SUBSCRIPTION_ID Your Azure subscription ID
- AZURE_TENANT_ID Your Entra directory (Tenant) ID

Steps:

Create your Entra ID App
Entra ID App for playwright testing
Assign your Entra ID App with a contributor or owner role in your playwright testing workspace.
Playwright Entra Id app added as owner user of the playwright testing workspace
Copy the values of your Entra ID app and add them as secret variables on your GitHub repository inside a GitHub organization.
Add your PLAYWRIGHT_SERVICE_URL as GitHub secret variable
Configure a federated identity credential on a Microsoft Entra application to trust tokens issued to your GitHub repository.
Add a new pipeline to execute your test with the following permissions:
1. token: write
2. contents: read

Add one step to log in to Azure.

name: Login to Azure with AzPowershell (enableAzPSSession true)
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
          enable-AzPSSession: true

Add the PLAYWRIGHT_SERVICE_URL as an environment variable and add the command to execute your tests with the playwright.service.config.ts. With the worker’s argument, set the number of workers. I put 20 because I have a few tests.

- name: Run playwright tests
    env:
        # Regional endpoint for Microsoft Playwright Testing
        PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
    run: npx playwright test --config=playwright.service.config.ts test --workers=20

Final pipeline .yml

name: Playwright testing
on:
  workflow_dispatch:
jobs:
  test:
    timeout-minutes: 60
    name: Run Test
    runs-on: ubuntu-latest
    permissions: # Required when using Microsoft Entra ID to authenticate
      id-token: write
      contents: read
    
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Install Node.js
        uses: actions/setup-node@v4

      - name: Login to Azure with AzPowershell (enableAzPSSession true)
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
          enable-AzPSSession: true

      # for playwright reporting service beta is needed the npm config
      - name: Install dependencies
        run: |
          npm ci

      - name: Install playwright browsers
        run: npx playwright install --with-deps

      - name: Run playwright tests
        env:
          # Regional endpoint for Microsoft Playwright Testing
          PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
        run: npx playwright test --config=playwright.service.config.ts test --workers=20

The tests are executed, and a summary with the link to the playwright testing test results is shown.

You can check the code on my playwright sample Github repo.

Cost

You can try for free for 30 days or with 100 minutes or 1,000 tests. The cost at the moment of I published the article is:

Hosted on Linux OS: $0.01/ 1 test minute
Hosted on Windows OS: $0.02/ 1 test minute

Reporting: $3.50/ 1,000 test results 90-day retention

Monitor your website with PerfAgents at AWS Market Place

PerfAgents is an advanced synthetic monitoring platform to monitor your site, REST API, or user journeys with Selenium, Playwright, Cypress, or Puppeteer with your code or an AI-integrated tool.

With PerfAgents, you can add you tests:

Uploading the file with your playwright test.
Importing from GitHub.
Generating with AI
Writing your test on the platform

After you select your AWS Zone and the frequency to execute your test, you don’t need to set up any pipeline. It will execute automatically and generate some reports.

Start monitoring your website or web app in real-time, and you will be notified if some bug or performance issue occurs.

PerfAgents Forensic Data shows your latest executions

I wrote Tools for Synthetic Monitoring to explain the synthetic testing tools, including PerfAgents.

Steps

Register your free trial at AWS Marketplace
Create your account on PerfAgents
Click on Monitoring > Create Monitoring
On Type select Sequence
Select Playwright
Create or import your test
Set your device (Chromium or WebKit)
Set the AWS Zone (us-east2, us-west1 ) and the frequency to execute your test.
Set your notifications (AWS SNS, Jira, Microsoft Teams, PagerDuty, Email, Slack, Splunk, and Wekhook)
Click on Create
Wait until your tests are executed
Check the report and dashboard.

This video shows a brief demo of the platform.

If you execute your test periodically on production:

You will be notified if some error occurs.
You will be notified if the time execution is longer than the time that you set previously.
With the playwright trace files, you can debug failures easily.

Cost

You can try it free for 7 days, and after that, it will cost $80 per month with 50000 REST API Monitoring Executions, 5000 Ping Monitoring Executions, and 3500 Sequence Monitoring Executions.

Thank you for reading, and feel free to suggest a topic for a new article and share if you think it is useful. Enjoy testing!!

QA tips by Abi

Discussion about this post