When looking at continuous testing it is important to find the right tools. With so many tools available it can be hard to narrow the choices and pick the best tools for one’s application. The following is a brief overview of the tools used for Developer Level, alternative tools, and a guide for picking the right tool.
Continuous Automation Tools Used on Developer Level
For the Developer Level application Paladin & Archer uses tools that run tests on functions and web interfaces. These tests run automatically when we send our code through the build pipeline. This continuous automatic testing assures that Developer Level is running properly and that added features will not break the current build. The tools used for Developer Level testing are Mocha, Nightwatch, and Jenkins.
Mocha is a JavaScript test framework running on Node.js. The feature-rich framework allows Developer Level to run tests serially allowing for flexible and accurate reporting. When errors are found Mocha maps them so that they are easy to find within the application. The use of Mocha tests within Developer Level significantly reduces the risk of error-prone code. Below is an example of a Mocha assertion used in Developer Level to check if non-Admin users have been created in the Database.
Having this specific test run every time we have a new build, insures that new users can create an account on the Developer Level App.
Pros of Using Mocha
- JavaScript Based
- Easy asynchronous testing
- Lots of tutorials
- Free
Cons of Using Mocha
- Could be tricky for beginners to set up
- Tests can’t be run in random order
Nightwatch is the testing solution Developer Level uses for browser-based tests. Nightwatch is written on Node.js and uses the W3C WebDriver API to perform commands and assertions on DOM elements. The browser-based tests are written using JavaScript and CSS. The simplicity and flexible structure of Nightwatch allows for Developer Level to have a reliable, adaptable web interface as it continues to improve. Below is an example of a Nightwatch test used to make sure users can log in to Developer Level using the web interface.
Pros of Using Nightwatch
- Easy to install using Node.js and NPM
- Well documented
- Easy to write tests using CSS selector model
- Free
Cons of Using Nightwatch
- No options to use with other frameworks and assertion libraries.
Jenkins is a self-contained, open source automation server which can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software. When a new feature is being pushed to the development or production branch, Jenkins automatically runs the created Mocha and Nightwatch tests. If new code happens to mess with the current build, these automated tests will catch and show the error and because of the automatic rollback feature, Jenkins will revert to the previous working build.
An example of Mocha tests successfully passing a Jenkins build
Pros of Using Jenkins
- Open source
- No fees
- Continuous Automations
- Automatic Rollback
Cons of Using Jenkins
- UI has a learning curve
- Troubleshooting Jenkins can be time consuming
Alternative Continuous Automation Tools
There are plenty of options when it comes to selecting the right tools for continuous automation. Other popular tools are:
Selenium is one of the most popular automation tools. It is used primarily for automating web applications for testing purposes but can do much more. Part of Selenium’s popularity is that it has support from some of the largest browser vendors. It is also the core technology in many browser automation tools, APIs, and Frameworks. This free automation tool has large community support and allows for integration with other tools and frameworks to enhances its capabilities.
Pros of Using Selenium
- Open source
- No fees
- Large support from active community of developers
- Integration with other tools and frameworks
- Scripting languages include Java, C#, Perl, Python, JavaScript, Ruby, and PHP
Cons of Using Selenium
- Programming skills required
- Setup time
- High learning curve
TestSigma is a cloud-based automation testing tool. The AI-driven ecosystem is for web, mobile apps, and API automated testing. With TestSigma you have all the essential elements required for testa automation. TestSigma allows the user to write tests in simple English. No coding skills required.
Pros of Using TestSigma
- Coding skills not required
- Easy to start making tests
- Completely web based
- Great support
Cons of Using TestSigma
- Reports could be better
- Lack of inherent support for DB testing
Jest is a JavaScript Testing Framework that is centered on simplicity. The well-documented tool requires little setup and can be adapted to meet your needs. With the quick results generated from written tests, users can be confident in their JavaScript code-base.
Pros of Using Jest
- Open Source
- Great Documentation
- Can run tests in parallel
- Flexible and easy configuration
Cons of Using Jest
- Learning curve for beginners
Postman is an automation tool used for testing APIs. This tool can be used as a browser extension or can be installed on the user’s Mac, Windows, or Linux machine.
Pros of Using Postman
- Training and tutorial available
- Free version
- Integrates with build systems, like Jenkins
Cons of Using Postman
- Limited features in free version
SoapUI is another useful tool for testing APIs and services. SoapUI supports both REST and SOAP services. Automation testers can choose to use either the open source or pro versions. The drag and drop and point-and-click functions of SoapUI make creating tests easy. It has a comprehensive feature set that allows for advanced asynchronous API testing capabilities. Many of SoapUI’s great features are available in the free open source version.
Pros of Using SoapUI
- Easy to create tests with drag and drop, and point-and-click
- Asynchronous testing
- Feature rich free open source version
- Comprehensive feature set
- Advanced testing capabilities
Cons of Using SoapUI
- Fee for premium features
- SoapUI is a memory hog
How to Pick The Right Tools
There are a few factors to consider when choosing the right tools for continuous automation. What does the tool cost? If the budget is tight an open source tool may be the only viable option. How are the test reports? A simple report might get the job done. What type of documentation and support is available? How much time will it take to integrate the tool? How steep is the learning curve? Will the tools be easy to work with?
In the article ‘How do People Select Test Automation Tools?’ Dr. Vu Nguyen suggested there are six priorities that should be considered:
- How much does it cost?
- Do the test reports meet our needs?
- What kind of training, documentation, tutorials, and/or guidelines are available?
- Is there CI/DevOps support?
- What level of programming skills is required?
- What level of skills and experience is required?
With so many testing automation tools available the main objective should be finding the tools that are the right fit for you.
Why We Chose Our Automation Tools
Developer level is built primarily with JavaScript and Node.js. With both Mocha and Nightwatch being based on Node.js and using JavaScript, the integration into the app was seamless. Both testing tools are easy to use, reliable, and supported with documentation and tutorials. To top it off, a few Paladin & Archer developers had previous experience with Mocha, Nightwatch, and Jenkins. The seamless integration, ease of use, and familiarity made these tools an easy pick for DeveloperLevel’s continuous automation.
Further reference: