Meet KaneAI: The New AI Testing Assistant
Generate your automated tests using natural language
I am evaluating different AI tools to understand their capabilities and how to use them. Now is the turn of KaneAI, a new software testing agent being developed by lambdatest. I learned about this tool in the testmuconf 2024 (It is 3 days of sessions related to testing).
I usually prefer to try alone without any demo session to explore if it is easy to use without knowledge. I made an exception to KaneAI because it is in private beta, and with an onboard Friday Zoom meeting, you can get your private access faster.
After I missed one session, I attended the onboard session and liked it a lot. They shared an example of a web test case and another for mobile. They also shared some features in the roadmap, and I got free private access after the onboarding. The lambdatest team was friendly and answered all doubts and the lambda team was open to suggestions. Also, you can send your feedback or report any bug via Slack.
This is the intro video:
AI assistants need clear instructions to generate better results, and KaneAI is no exception. The step description should be very clear and only allow a straightforward action or assert to execute the automated tests correctly and without any error.
I tested with different sites:
My accounts website is slow and has different user roles, dashboards, and grids.
WebdriverIO Swag Store to test the flow to add an item to the cart.
The first time you execute the tests, it is slower because it generates the code.
How to generate your first test
The first time you log in, you can use some predefined examples of the tets or start adding the first step.
Write your step and press enter. In this example, I want to test the login on my website, so the first step is go to https://effizientedemo.azurewebsites.net
You will see a page to add the main details of your test outcomes, constraints, error messages, and user inputs. Fill in the following data:
Outcome: Sales Dashboard
User Input
Company: Demo
User: Demo
Password: Demo
You will see a Dialog box to review your test details and select the Project and Folder to store your test I fill:
Select Project: I created a Demo Project
Select Folder: I created a Login to store login with different user roles, and the wrong user
Type: Functional
Status: Draft
Tags: NormalUser
Click on Schedule Test to see that the test is being generated; the first time, the execution will be slower, and the subsequent executions will be faster.
Click on Live Preview, and the steps will be generated in the left panel. On the right side, you can see how the tests are executed. Also, store a screenshot of each step.
Test Manager
You can see a dashboard with the test results in the Test Manager.
In Test Cases, you will see your test. You can write more steps or record more steps with Manual Interaction.
This is a playground to try different steps for your test; in the beta version, you will see some errors while creating your steps, so I suggest you try and save your tests.
Also, you can import the steps from a CSV file.
With my test related to the webdriverIO e-commerce, the price depends on the size of the clothes, and I got some minor errors. I can test the main flow to add one item to the cart, but it failed to check the final price.
These are the steps that I tried:
Go to https://shop.webdriver.io/
Click on Shop All
Get the item description and item price of any shirt or cap so you can reuse this name and description in a future step.
Click on the item description for the previous step
Assert that the item description is the same as the previous step
Assert that the price is in the range of the last step
I added another test to test the flow of adding a server with the admin user in my sales dashboard website. I got some errors that I reported in Slack, but in general, I think it is a good product that will improve with more beta users and, after some months, will include more features.
These were my steps:
Go to https://effizientedemo.azurewebsites.net/
Type 'Demo' in the 'Company' input field.
Type 'Admin' in the 'User' input field.
Type 'Admin' in the 'Password' input field.
Click on the 'Login' button.
Check the menu includes the option Security
Hover the Security menu
Click in the submenu Servers
Click on the + icon
Write 2 in the key input
Write lambdatest in the name input
Write https://kaneai.lambdatest.com in the Server URL input
Click in save
Check that a row exists with the name lambdatest
Click on the trash icon for the row with the key 2
I didn’t test with the Gherkin syntax (Feature, Scenario, Given, When, Then) because it is mainly used by QA, which is an extra step for me. With playwright, I prefer to auto-generate my test steps in the playwright reporter.
How to generate the code
By now, you can only generate the code in Python Selenium, but there are plans to create the code using main testing frameworks like Cypress and Playwright.
For mobile, it will be available in Android/Appium next week.
Although the code generated does not use the Page Object Model, it looks like it uses Appium capabilities.
Also, when you update and save your test, a new version is created, and you can revert to the previous commit.
Bugs
The first time I created the test, I made a mistake and wrote the wrong password. Kane AI automatically added it as an issue with the steps, the expected outcome, the actual outcome, and the error message. Although the error message is in Spanish, KaneAI can detect and add the error message automatically. I suggested some options to mark the bug as an error typo because it is not a bug.
Hyperexecute
This Lambdatest product can execute your test in different browsers, OS, and geo-locations.
My Feedback
In Slack, I shared some bugs and some feedback about some features:
Option to add the bugs in Jira and Azure DevOps.
Include a vault to store sensitive values like user names and passwords.
Add an example of more complex scenarios like a grid.
Environment variables such as QA and Prod to execute the tests in that environment.
Some bugs were related to the grid, menu, or more complex scenarios related to getting random data, and there were some errors when I tried to run in hyperexecute.
Although it is still a beta version, the lambdatest team is very friendly and supportive, open to suggestions, and with a lot of features that will released in the following weeks and months, so I think in the end, a final version will be a great product and includes all the most important tools that a tester needs like the reporting, the code and the integration with hyper execute to test in different browsers.
AI will be included in more software in the following months, so it’s essential to practice and train the AI tools to get faster and better results, which may be needed in future jobs. Although I enjoyed coding, maybe in the future, I will be more like this in generating the code or training the AI models.
At least in my last 3 projects in different companies, they are using GitHub Copilot. So you can request free private access and start practicing.
Thank you for reading, and enjoy testing.