Tools battle: SpiraTest, TestRail and TestLodge

The modern IT marketplace of the high technologies offers the multiplicity of great solutions for software testing. In this article we review the pros and cons of the most famous web-based test management tools: SpiraTest, TestRail and TestLodge. Every of the mentioned services is the comprehensive web-based test management software that allows to control, monitor and organize effectively the testing efforts of the QA department.

The main advantages of the aforementioned tools are: SaaS delivery model (allowing to avoid losses on installation and maintenance of the software), simplicity of the user interface (allowing to manage all the test artifacts in one workplace), a wide range of features (requirement management, test management, reporting management, etc.) and ability to integrate with the large number of bug trackers and frameworks.

Outline

• licensing
• access control
• integration with external tools
• working interface
• requirement management
• test design
• test execution
• summary table
• conclusion

Licensing

All the considered applications are commercial and offer 30 days trial version for familiarization with the product. Every of them presents its own licensing system, so SpiraTest and TestRail use the licensing arrangement based on the number of users , while the licensing rules of TestLodge are based on the quantity of activities (the amount of test plans, test cases and test runs) and do not depend on the number of users.

Access control

TestRail provides fine grained access control which allows to edit pre-installed roles, add custom roles and combine users into groups. Each role allows to set permissions for configurations, milestones, test-cases, attachments, test runs, test-results and reports. SpiraTest offers very similar access control system which allows to edit preset roles and create custom roles with a variety of permissions. As for TestLodge, the access control of this app allows to assign admin role to a user and create custom roles which define read/write permissions for test plans, requirements, test suites and test runs.

Integration with existing tools

All the mentioned test management applications support integration with a wide range of bug tracking systems, frameworks and add-ons. But, only SpiraTest offers its own built-in issue tracker, TestRail and TestLodge refused this idea, since they offer the large number of external bug trackers. The most popular of them are listed below:

Working interface

TestLodge and TestRail provide the well-organized and simple UI, allowing to capture quickly an idea of a project. However, this simplicity may be criticized by experienced users who want all the sheaths at once. As for SpiraTest, this tool offers intuitive UI which includes a lot of filters to narrow down the number of shells on the workspace, but at the same time this app suffers from the multiplicity of the "edit" and "update" buttons, that force user to make a lot of redundant "clicks" to finish any edit and update operation.

Requirement management

TestLodge and SpiraTest offer built-in requirement management (RM) modules which allow to perform CRUD operations on requirements, attach files and assign test cases to requirements. TestRail does not provide built-in RM service, but supports integration with external services, like Jira. As for the ability to generate traceability matrix (a document that shows the test coverage of requirements), all the reviewed tools support it quite well.

Test design

All the reviewed apps allow to perform CRUD operations on the test-cases, but TestLodge and TestRail implement simple and fast method of test case management, which allows to add a test-case with the minimum number of details (title, description, steps, expected result).

The next point to review is a search functionality, here SpiraTest and TestRail are taking the lead, since TestLodge does not support this feature at all. The difference between SpiraTest and TestRail is in the fact that SpiraTest offers the global search through all the projects and test artifacts, and TestRail offers search only among the specific test artifacts and does not support searching by keywords.

As regards import/export, TestRail and TestLodge provide easy in use import/export without configuring, while SpiraTest requires to install and configure specific add-on.

And lastly, the document attachments. All the reviewed applications allow to attach any kind of files to test cases, but in contrast to SpiraTest and TestRail, the number of attached documents in TestLodge is limited in accordance with the selected plan (non-trial).

Test execution

SpiraTest and TestRail display the execution results as a tree, while TestLodge presents test results in the row mode (test runs are listed in the rows which are located one by one on the main page of the service). Both types of the interfaces are intuitive and easy in use, but tree-based structure is more comprehensive, since the critical and valuable data are displayed clearer.

As for the test results tracking (document that displays status of test cases for a project and allows to analyze and monitor how the development work is going), here SpiraTest is a bit behind, because TestRail and TestLodge have deduced this feature into a separate unit, allowing to trace the history of launches and tests editing more precisely and clearly.

Summary table

Conclusion

All the considered tools are well-organized test management services that offer a wide range of the capabilities and features, allowing to cover the most part of users' needs. TestLodge and TestRail offer thought-out and convenient UI which in combination with a sufficient number of the features, allows to resolve the majority tasks of small and medium-sized projects. As for SpiraTest, it is powerful but a bit complex test management software, which can be suitable for big and complicated projects.

Hopefully, the provided information will help you to select the most suitable test-management tool for your needs.