A Guide to Azure DevOps Pipelines and Test Plan with Playwright

Execute your tests and publish your results in your Azure DevOps Test Plan

Oct 30, 2024

What is Azure DevOps?

Azure DevOps is a suite of tools and services created by Microsoft that supports the entire software development lifecycle. Includes the following main components:

Azure Boards: To plan, track, and discuss the work. You can add some extensions to include your retrospective and poker planning.
Azure Repos: Provides Git repositories for the source control of your code. You can create custom rules for your repositories, like the number of approvals in the PR, PR templates, and some build checks, and auto-complete your work item after merging the PR.
Azure Pipelines: Provides build and releases for Azure Repos, GitHub, or Bitbucket.
Azure Test Plan: With extra cost per user, it allows you to track your manual tests, link your automated test, and create the bugs from your manual test, as well as some charts and details about the execution of your test.
Artifacts: Allows teams to share standard packages such as npm and Nuget and integrate them into your pipeline.

I will explain the part of the Test Plan and how you can integrate with Playwright with typescript.

Azure Test Plans

It allows you to create your manual test cases and assign testers to execute them.

Test Case Sample, with Id, Name, steps — Test Case Example: Id, Title, and Steps

All tests have an ID; in this example, it is 395. You need this ID to link your automated tests with your tests on the test plan. It has better integration with C# and Visual Studio.

How to link your tests with Playwright and typescript

To link your Playwright with Azure DevOps, you need to follow the next steps:

1. Install the @alex_neo/playwright-azure-reporter package.

NPM

npm install @alex_neo/playwright-azure-reporter

Add the Azure Test Case ID as a tag between [] on your Playwright test.

test(`Login with "${userName}" will load inventory page`, 
    { tag: ['@Basic', '@[395]']}, async ({ page }) => {
   const loginPage = new LoginPage(page);

Get your user token from your Azure DevOps account:
3.1 Go to the user settings icon menu, select Personal access tokens

Azure Devops highlighting where is the Organization Id, the project name, plan Id and access token — Personal Access Token, organization, project, and plan ID.

3.2 Click on + New Token
3.3 Add a name, expiration date, and permission.
3.4 Click on Create

Copy the access token and add it to your .env file. Also, add other environment variables to reuse your pipeline for other projects:
- Organization
- Ado Project
- Test owner
- Plan Id: Id of the test plan. You can find it in the test plan URL. Example: https://dev.azure.com/wbi1521/Playwright/_testPlans/execute?planId=398&suiteId=6- ConfigId: Go to https://dev.azure.com/{organization}/{project}/_apis/test/configurations to get the ids of your project. If multiple configuration IDs are used in one run, a testPointMapper should be used to pick the correct one; otherwise, the results are pushed to all.

ADO_TOKEN='your_token'
ADO_ORGANIZATION='wbi1521'
ADO_PROJECT='Playwright'
ADO_PLAN_ID=398
TEST_OWNER="Abigail Armijo"
ADO_CONFIG_IDS=46

Add the @alex_neo/playwright-azure-reporter to your playwright.config.ts. To display the test results in Azure DevOps, you must set up the JUnit report and the output folder for the HTML playwright report with the option open: ‘never.’

import { AzureReporterOptions } from '@alex_neo/playwright-azure-reporter/dist/playwright-azure-reporter';

const config: PlaywrightTestConfig = {
  testDir: './tests',
  ...
  reporter: [
    ['junit', { outputFile: 'results.xml' }],
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    [
      '@alex_neo/playwright-azure-reporter',
      {
          orgUrl: 'https://dev.azure.com/wbi1521/Playwright',
          token: process.env.ADO_TOKEN,
          planId: process.env.ADO_PLAN_ID ?? 1,
          projectName: process.env.ADO_PROJECT,
          environment: 'QA',
          logging: true,
          testRunTitle: 'Playwright Test Run',
          publishTestResultsMode: 'testRun',
          uploadAttachments: true,
          attachmentsType: ['screenshot', 'video', 'trace'],
          testRunConfig: {
              owner: {
                 displayName: process.env.TEST_OWNER 
               },
               comment: 'Playwright Test Run',
               configurationIds: [process.env.ADO_CONFIG_IDS ?? 1],
                },
            } as AzureReporterOptions
      }]

The result will be uploaded to your test plan every time that you execute your test.

How to execute your test with Azure Pipeline

With pipelines, you can execute the tests with a schedule, for example, all days, every 5 hours, or when a PR is created.

Also, you can add your code from Azure DevOps Repos or with Github.

Click on Pipelines → Pipelines.
Click on Create Pipeline
Azure DevOps Create Pipeline
Select where your code is. In this example, I will select that is inside Azure Repos Git.
Select your repository by name. In my case, it is Playwright.
You can choose any predefined template, which is primarily for development. You can select Node.js.

Will create a default pipeline to execute a node.js application. Change to the main steps to execute your Playwright tests. The diagram of the main steps is:

Trigger event: The pipeline will executed with any commit of the main branch

Step 1: Install the Node.JS

  - task: NodeTool@0
    inputs:
      versionSpec: ">=20.0.0"
    displayName: "Install Node.js>20"

Step 2: Install the dependencies

  - script: |
      npm ci
    displayName: "Install dependencies"

Step 3: Install the playwright browsers

  - script: |
      npx playwright install --with-deps
    displayName: "Install Playwright Browsers"

Step 4: Execute your playwright tests, and because the ADO_TOKEN should be a secret variable, you need to add it as an environment variable.
```
  - script: npx playwright test
    displayName: "Run Playwright tests"
    env:
      ADO_TOKEN: $(ADO_TOKEN)
```

Step 5: You need to publish the JUnit results to show the details of the tests after the pipeline is executed

  - task: PublishTestResults@2
    displayName: 'Publish Junit results'
    condition: always()
    inputs:
      testResultsFiles: "results.xml"
      testRunTitle: "Playwright Junit Test Results"

Step 6: Upload the HTML report as an artifact. Azure DevOps doesn’t allow the HTML reporter to be displayed inside the pipeline.

  - task: PublishBuildArtifacts@1
    displayName: 'Publish HTML Playwright report'
    condition: always()
    inputs:
      PathtoPublish: "playwright-report"
      ArtifactName: "playwright-report"
      publishLocation: "Container"

Your final file is:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: ">=20.0.0"
    displayName: "Install Node.js>20"

  - script: |
      npm ci
    displayName: "Install dependencies"

  - script: |
      npx playwright install --with-deps
    displayName: "Install Playwright Browsers"

  - script: npx playwright test
    displayName: "Run Playwright tests"
    env:
      ADO_TOKEN: $(ADO_TOKEN)

  - task: PublishTestResults@2
    displayName: 'Publish Junit results'
    condition: always()
    inputs:
      testResultsFiles: "results.xml"
      testRunTitle: "Playwright Junit Test Results"

  - task: PublishBuildArtifacts@1
    displayName: 'Publish HTML Playwright report'
    condition: always()
    inputs:
      PathtoPublish: "playwright-report"
      ArtifactName: "playwright-report"
      publishLocation: "Container"

Click on the Variables button to add the environment variables to connect to Azure DevOps
Azure DevOps Click on the Variables Button
Click on the New Variable button
Add your variables:
1. ADO_TOKEN: As a secret variable
2. ADO_ORGANIZATION='wbi1521'
3. ADO_PROJECT='Playwright'
4. ADO_PLAN_ID=398
5. TEST_OWNER="Abigail Armijo"
6. ADO_CONFIG_IDS=46
Finally, click on Save and run
Your tests will be executed
Azure DevOps Executing your tests
Click on your pipeline to see the details and download the artifact.
Azure DevOps Summary shows the artifact to download.
If you click on the Test Tab, you can see the results of your tests.
1. You will see a Summary with the details of the tests
2. You can filter to show all the tests. It’s the same name because I am executing Webkit, Firefox, and Chromium tests.
3. And if some test fails, you can create a bug.

When some tests fail, you can see the video and the trace viewer in the test if you enable those options in your playwright.config.ts

Thank you for reading. Feel free to suggest any topics or questions. If the article is helpful, you can share it. Enjoy testing!!

QA tips by Abi

Discussion about this post