Enhancing Test Reporting: Integrating Allure with Playwright in Azure DevOps
A guide to Allure Report and how to integrate with your Playwright test and Azure Devops
In my previous article, A Guide to Azure DevOps Pipelines and Test Plan with Playwright, I briefly explained what Azure DevOps is and how to link your playwright tests with your Azure DevOps Test Plan tests.
Although the Playwright HTML reporter is very friendly, you can’t see the history of the previous executions, charts about test results, or links to some bugs. You can see the history and create bugs with Azure DevOps, Jenkins, and Bamboo tools. However, Allure Report is another popular reporting tool available for the main testing frameworks.
With Allure, you can generate an HTML report or use the cloud version of Allure TestOps, a test management tool in the cloud with costs starting from $39 per user. I will explain and show some videos about how to integrate with Playwright.
How to integrate Allure report with Playwright
First, you need to install the allure playwright package
npm install --save-dev @playwright/test allure-playwrightIn the playwright.config.ts file, you must add the configuration for the allure report. You can add some imports to get the OS details and node version that will be included in the reporter. Also, you can classify the failed test into categories with regular expressions.
import type { PlaywrightTestConfig } from '@playwright/test';
import os from 'node:os';
import dotenv from 'dotenv';
//more config code
reporter: [
['junit', { outputFile: 'results.xml' }],
['html', { outputFolder: 'playwright-report', open: 'never' }],
['allure-playwright', {
detail: false,
suiteTitle: true,
environmentInfo: {
OS: os.platform(),
Architecture: os.arch(),
NodeVersion: process.version,
},
categories: [ //To classify errors by category
{
name: 'Missing file errors',
messageRegex: '.*ENOENT: no such file or directory.*'
},
{
name: 'Internal Server Error',
messageRegex: '.*Internal Server Error.*',
},
{
name: 'Timeout errors',
messageRegex: '.*timeout.*'
},
{
name: 'Accessibility',
messageRegex: '.*accessibility.*'
}
],
}
]If you wrap your steps with the test.step function, when you execute your tests, the report will show your user-friendly steps. I set up the allure report to not include the code because sometimes, not all end users are good at a tech level, and sometimes, it is easier only to see the steps descriptions.
import { test, expect } from '@playwright/test';
test('test', async ({ page }) => {
const user = await test.step('Log in', async () => {
// ...
return 'john';
});
expect(user).toBe('john');
});You can group your test by feature and suite. And if your test is parameterized, you can add the parameters to the allure reporter.
import * as allure from 'allure-js-commons';
test.describe('Login with Parameters', () => {
[
{ userName: 'standard_user' },
{ userName: 'visual_user' },
].forEach(({ userName }) => {
test(`Login with "${userName}" from parameters will load inventory page`, async ({ page }) => {
await allure.feature('Basic');
await allure.suite('Sauce demo');
await allure.parameter('userName', userName);By default, the report is generated in the allure-report folder.
You can execute your test with the command
npx playwright testYou can see the results in a local web server if you execute this command
allure serve allure-resultsOr you can generate a standalone with this command
allure generate --single-file allure-resultsThis video contains some of the features of the HTML report generated.
How to integrate Allure report into Azure DevOps
To integrate your test in Azure DevOps, you must install the Allure Report extension from Qameta software.
Search for Allure Report
Click the Allure Report by Qameta
Visual Studio Market Place for Azure DevOps Allure Report Click on Get it free
Install the Allure Report Select your Azure DevOps organization to install. If you are not the admin user, you need to send a request, fill in the information to send the request to the admin user, and click on Request.
If you are not the admin user, you need to request access to the administrator to install the Allure Report Extension. You must update your pipeline to include the steps to add the reporter to the Azure DevOps Pipeline. You need to install Allure and add another step to generate the report as an HTML file, publish the results to the Allure extension, and upload the reporter as an artifact.
- script: | npm install -g allure-commandline displayName: "Install Allure" - script: | allure generate --single-file allure-results displayName: 'Generate allure report as single HTML' condition: always() - task: PublishBuildArtifacts@1 displayName: 'Publish Allure report' condition: always() inputs: PathtoPublish: "allure-results" ArtifactName: "allure-results" publishLocation: "Container" - task: PublishAllureReport@1 displayName: 'Publish report to the allure report extension' condition: always() inputs: allureVersion: 2.27.0 testResultsDir: allure-resultsFinally, when you execute your test, you will see an extra tab with the reporter.
The video shows the integration of allure report with Azure DevOps.
How to integrate with TestOps Cloud
Finally, you can enable a bidirectional connection between Azure DevOps and the TestOps Cloud service.
If you want to execute your test from the Azure DevOps Pipeline and connect with TestOps, you only need to add the following steps:
Create an account on Qameta
Get your allure token
In Allure TestOps, click on your user avatar.
On the API Tokens section, click on Edit.
Enter a Token Name, for example Azure DevOps Token.
Copy the token displayed in the dialog.
On your pipeline, you need to install the allurectl command line to upload your test results to TestOps.
- bash: curl -fsSL https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -o allurectl && chmod +x allurectl displayName: Download allurectlOn your pipeline, for the step to execute the playwright test, you need to add the command to allurectl wath to update the steps to TestOps. And you need to add the following environment variables:
ALLURE_ENDPOINT: Your allure endpoint is your allure test site.
ALLURE_PROJECT_ID: The ID of the Allure Project.
ALLURE_RESULTS: Folder where allure results are stored. By default, they are in the allure-results folder.
ALLURE_TOKEN: The token on the step 2
- script: ./allurectl watch -- npx playwright test displayName: "Run Playwright tests" env: ADO_TOKEN: $(ADO_TOKEN) ALLURE_ENDPOINT: https://effiziente.testops.cloud ALLURE_PROJECT_ID: 1 ALLURE_RESULTS: allure-results ALLURE_TOKEN: $(ALLURE_TOKEN)
On the environment variables, add the variables needed for allure TestOps
Every time you execute your tests, the results are updated to TestOps. If one test doesn’t exist, it will create the test. When a test fails, it will update the files if you set it on your playwright.config, screenshots, video, and trace viewer.
If you want to enable the option to execute your test in Azure DevOps from Allure TestOps, you can follow the Integration with Azure DevOps.
The video shows some of the Allure Report on Test Ops features. If you enable the integration with Azure DevOps, you can execute your automated test from Allure TestOps, which will execute your Azure DevOps Pipeline.
Youn can also check my playwright sample code.
Thank you for reading, and feel free to suggest a topic for a new article and share if you think it is useful. Enjoy testing!!