Tools for Synthetic Monitoring
Proactively find bugs before your users with synthetic testing
This is my first sponsored article, and I will share my experience with synthetic testing with Checkly and PerfAgents using Playwright.
What is Synthetic Testing?
It is a type of performance testing in which you periodically test your site to detect failures or poor performance before your customers find the errors and leave your Website or Application. Furthermore, you can execute your test from different regions and countries.
These examples can help you to understand the synthetic testing:
Example 1: E-commerce
If an e-commerce website is slow or the user can’t buy an item because it shows an error, most users don’t report it and simply abandon the shopping cart.
However, if you execute a Ping to your Website or Application every 10 minutes, and your main user journeys (Customer flow testing) to buy an item and main API (REST API testing), for example, login, addItem, payment, every 30 minutes and set up your performance metrics like response time, you can detect an error in your site or receive an alert if the response time exceeds 500 milliseconds. You will be notified of all the error details in Slack, Microsoft Teams, Email, SMS, Jira, etc.
Example 2: LLM (Large Language Model)
With the trend and growth of AI and Generative AI, which sometimes executes many APIs to get results, it’s important to get the results as soon as possible. You also need to check the accuracy of your models.
In one of my previous jobs, I developed something similar to execute tests every 3 hours for different countries. I had a lot of issues with some countries that I couldn’t access from Jenkins. I used a lot of technologies: Jenkins, Slack, TestCafe, Nightwatch, AWS lambda functions, AWS step functions with Python and JS, S3, KMS, Splunk, BigPanda, RunDeck, and Service Now. Some tools had already been configured, so it was a huge effort. It is better not to reinvent the wheel and try some synthetic tools.
What is Synthetic Monitoring?
It is a continuous process that periodically uses automated scripts or tools to simulate:
User flows: Test your main user journeys, such as shopping for an item or testing your site in a new country.
API: Verify the API requests and response time.
Ping: Ping a server to measure that the server is available.
SSL Certificate: Analyzes the validity of your SSL certificate.
Top 3 Benefits
Some of the benefits of synthetic testing are:
Reduce the time to solve the issues: You get alerted soon and get some details of the errors (tracer in Playwright), which helps a lot with the steps for the bug and quick troubleshooting. You can also fix the issue before users report it.
Identify performance problems: You can constantly measure your response time and act when it exceeds your expected response time.
Launch to a new market: Because you can execute your test in different countries, you can monitor your site's behavior in another country and compare your site's response time in different countries.
15 Tools for Synthetic Testing
Some of the tools that help you to monitor are:
I tried Checkly and PerfAgents. I tested it with my website and stopped it in Azure to get the details of the errors.
Checkly
Only Playwright offers different services to monitor your E2E tests, API, and Ping. It is adding more services like visual testing.
I didn’t find a link to explain how to integrate Checkly with your current tests, and when I tried, the support chat was unavailable. After some attempts and bugs, I could set up a playwright test.
I stopped the API App Service in Azure to check the details of the error:
Summary of Features:
Alerts/Integrations: ⭐️⭐️⭐️⭐️⭐️
Slack
Pagerduty
Opsgenie
Microsoft Teams
Discord
Spike.sh
Splunk On-Call
StatusPage
Prometheus
FireHydrant
Coralogix
Incident.io
ilert
Telegram
AWS zones: ⭐️⭐️⭐️⭐️⭐️ You can execute your test for different zones:
6 zones in Americas
8 zones in Europe/Middle East/Africa
8 zones in Asia Pacific
Hobby plan: Free
1 user.
10k API / 1.5k browsers.
Email, slack alerts, webhook alerts.
Team: $80 by Month
Up to 20 users.
100k API / 12k browser.
All alerts.
E2E Testing: ⭐️⭐️⭐️ Focused only on Playwright, this included many samples from the Playwright ambassador, Stefan Judis.
Limitations: ⭐️⭐️⭐️ You need to check the available Playwright version, config options, and available external packages like ExcelJS here: specs and config. It seems that it only runs on a Chrome desktop.
Support: ⭐️⭐️ By chat and maybe for the time zone is not available in the same working hours as Mexico, and the day I had more time, no agents were available. I added all my issues in one chat, and I had to split them into different chats and wait until the next day to check the answer. It includes a Slack channel.
Documentation: ⭐️⭐️⭐️⭐️ Offers a demo session, and I was contacted in Spanish for this demo. However, I like to experiment by myself because, in the demo, everything works with a predefined test. There are also a lot of videos, but most are focused on the new setup.
Easy to Use / UX: ⭐️⭐️⭐️⭐️ In your account setup, you can set ping and API testing, which makes the UI easy to understand and try for the first time. The site is responsive.
PerfAgents
I set up a cypress and playwright, ping, and rest API. You can set it up to execute on:
Desktop for Chromium and WebKit (1980 x 720 or 320 x 420).
Tablet (1024 x 1024, 320 x 420).
Mobile (720 x 420 or 320 x 420).
Incidents dashboard.
Detail of the API error alert when I stopped the Azure App Service for the API.
Summary of Features:
Alerts/Integrations: ⭐️⭐️⭐️⭐️
Slack
Email
Microsoft Teams
Jira
AWS S3
AWS SNS
Pagerduty
Splunk
Webhook
AWS zones: ⭐️⭐️
3 zones in the USA (us-east-1, us-east-2, us-west-1).
2 in the Asia/Pacific (ap-south-2, ap-southeast-2).
1 in the Middle East (me-central-1).
1 in Europe (eu-north-1)
More zones will be included in future releases.
Free trial for 7 days
Uncloud+ Schedule a meeting to know the custom pricing.
Unlimited users.
1M ping/SSL.
1M Rest API.
50K sequence (E2E).
Uncloud++ Schedule a meeting to know the custom pricing.
Unlimited users.
10M ping/SSL.
10M Rest API.
5M sequence (browser).
AI Features.
Premium Support.
E2E Testing: ⭐️⭐️⭐️⭐️⭐️ Includes the main E2E testing frameworks:
Selenium
JS
Python
Java (Soon)
Cypress
Puppeteer
Playwright
Limitations: ⭐️⭐️⭐️⭐️⭐️ You need to check the versions for different frameworks. I can only select 10 files from my GitHub repo.
Support: ⭐️⭐️⭐️⭐️⭐️ Includes a chat and a tool for reporting bugs with screenshots. You need to register for the support ticket system. You will be notified by email about comments about your support tickets.
API Error Detail Documentation: ⭐️⭐️⭐️ Offers a demo session, some written documentation, and some guides onboard, but maybe more videos are needed.
Easy to Use / UX: ⭐️⭐️⭐️⭐️ Setup the test is easy with some initial onboard instructions. The report and alerts are clear.
Include an AI feature to generate your test
Tips to Write the Automated Tests
After checking Checkly and PerfAgents, my suggestions for synthetic testing are:
You must have a clear goal to set up the alerts for the response time and main user journeys to monitor.
Add a test with all steps without dependency, such as page object model, external npm packages, and environment variables.
Check the runtime and npm packages that are available for testing.
Contact support with your doubts.