The modern IT marketplace of high technologies offers a multiplicity of great solutions for software testing.
In this article, we review the pros and cons of the most famous web-based test management tools:
SpiraTest,
TestRail and
TestLodge.
Every one of the mentioned services is a comprehensive web-based test management software that allows to control, monitor
and organize effectively the testing efforts of the QA department.
The main advantages of the aforementioned tools are:
SaaS delivery model
(allowing for the avoidance of losses on installation and maintenance of the software),
simplicity of the user interface (allowing the management of all the test artifacts in one workplace),
a wide range of features (requirement management, test management, reporting management, etc.) and
ability to integrate with a large number of bug trackers and frameworks.
Outline
- licensing
- access control
- integration with external tools
- working interface
- requirement management
- test design
- test execution
- summary table
- conclusion
Licensing
All the considered applications are commercial and offer a 30-day trial version for familiarization with the product.
Every one of them presents its own licensing system, so SpiraTest and TestRail use the licensing arrangement
based on the number of users, while the licensing rules of TestLodge are based on the quantity of activities
(the amount of test plans, test cases, and test runs) and do not depend on the number of users.
Access Control
TestRail provides fine-grained access control, which allows users to edit pre-installed roles, add custom roles,
and combine users into groups. Each role allows setting permissions for configurations, milestones,
test cases, attachments, test runs, test results, and reports. SpiraTest offers a very similar access control
system, which allows editing preset roles and creating custom roles with a variety of permissions.
As for TestLodge, the access control of this app allows assigning the admin role to a user and creating custom roles
which define read/write permissions for test plans, requirements, test suites, and test runs.
Integration with Existing Tools
All the mentioned test management applications support integration with a wide range of bug tracking systems,
frameworks, and add-ons. But, only SpiraTest offers its own built-in issue tracker; TestRail and TestLodge refused this
idea, since they offer a large number of external bug trackers. The most popular of them are listed below:
| Integration with bug trackers | |||
| SpiraTest | TestRail | TestLodge | |
| Built-in issue tracker | + | not implemented | not implemented |
| Assembla | not supported | + | + |
| Basecamp | not supported | + | + |
| Bitbucket | not supported | + | not supported |
| Bugzilla | + | + | + |
| Fixx | not supported | + | not supported |
| FogBugs | + | + | + |
| Gemini | not supported | + | not supported |
| GitHub | not supported | + | + |
| IBM Rational ClearQuest | + | not supported | not supported |
| Jira | + | + | + |
| LightHouse | not supported | + | + |
| Mantis | + | + | + |
| OnTime | + | + | not supported |
| Pivotal tracker | not supported | + | + |
| Rally | not supported | + | not supported |
| Redbooth | not supported | not supported | + |
| Redmine | + | + | + |
| RTC | + | not supported | not supported |
| TFS | + | + | not supported |
| TRAC | not supported | + | not supported |
| Trello | not supported | + | + |
| Unfuddle | not supported | + | + |
| Vault | not supported | + | not supported |
| Youtrack | not supported | + | not supported |
Working Interface
TestLodge and TestRail provide a well-organized and simple UI, allowing users to capture quickly
an idea of a project. However, this simplicity may be criticized by experienced users who want
all the sheaths at once. As for SpiraTest, this tool offers an intuitive UI that includes a lot of filters
to narrow down the number of shells on the workspace, but at the same time, this app suffers
from the multiplicity of the “edit” and “update” buttons, which force users to make a lot of redundant “clicks”
to finish any edit and update operation.
Requirement Management
TestLodge and SpiraTest offer built-in requirement management (RM) modules that allow performing CRUD operations on
requirements, attaching files, and assigning test cases to requirements.
TestRail does not provide built-in RM service, but supports integration with external services, like Jira.
As for the ability to generate a traceability matrix (a document that shows the test
coverage of requirements), all the reviewed tools support it quite well.
Test Design
All the reviewed apps allow performing CRUD operations on the test cases, but TestLodge and TestRail
Implement a simple and fast method of test case management, which allows adding a test case with the minimum number of details (title, description, steps, expected result).
The next point to review is a search functionality, here SpiraTest and TestRail are taking the lead, since TestLodge
does not support this feature at all. The difference between SpiraTest and TestRail is in the fact that SpiraTest offers
a global search through all the projects and test artifacts, and TestRail offers search only among the specific test artifacts and does not support searching by keywords.
As regards import/export, TestRail and TestLodge provide easy-to-use import/export without configuring,
while SpiraTest requires installing and configuring a specific add-on.
| Import/export formats of the test-artifacts | |||
| SpiraTest | TestRail | TestLodge | |
| Import | MS Excel; MS Word; MS Project | XML | Google spreadsheet |
| Export | MS Excel; MS Word; MS Project | XML; CSV; Excel | XML; HTML |
And lastly, the document attachments. All the reviewed applications allow attaching any kind of files to test cases,
In contrast to SpiraTest and TestRail, the number of attached documents in TestLodge is limited by the selected plan (non-trial).
Test Execution
SpiraTest and TestRail display the execution results as a tree, while TestLodge presents test results in the row mode
(test runs are listed in rows, which are located one by one on the main page of the service).
Both types of interfaces are intuitive and easy to use, but the tree-based structure is more comprehensive,
since the critical and valuable data are displayed more clearly.
As for the test results tracking (document that displays the status of test cases for a project
and allows to analyze and monitor how the development work is going), here SpiraTest is a bit behind, because TestRail
and TestLodge have deduced this feature into a separate unit, allowing for the tracing of the history of launches and tests
editing more precisely and clearly.
Summary Table
| SpiraTest | TestRail | TestLodge | |
| General points | |||
| Trial mode | 30 days | 30 days | 30 days |
| Licensing | based on the number of users | based on the number of users | based on the number of activities |
| Requirement management | |||
| CRUD operations | + | only through external RM tool | + |
| Complexity of adding a requirement | too many fields | only through external RM tool | simple |
| Structure of requirements | tree-based structure | depends on external RM tool | row-based structure |
| Traceability matrix | + | + | + |
| Integration with external RM tools | + | + | + |
| Test design | |||
| CRUD operations | + | + | + |
| Complexity of creating a single test case | too many steps | simple method | simple method |
| Search functionality | global search | limited search | not implemented |
| Custom fields | + | + | + |
| Import/export | + | + | + |
| Documents attachment to the test case | + | + | limits are set in compliance to the selected plan |
| Test execution | |||
| The execution interface | tree-based structure | tree-based structure | row-based structure |
| Manual execution | in a browser window | in a pop-up window | in a browser window |
| Test results tracking | + | + | + |
| Defect management | |||
| Inbuilt (own) defect management system | + | not implemented | not implemented |
| Integration with issue trackers | + | + | + |
Conclusion
All the considered tools are well-organized test management services that offer a wide range of the capabilities and features, allowing to cover the most part of users’ needs. TestLodge and TestRail offer a thought-out and convenient UI, which, in combination with a sufficient number of features, allows for resolving the majority of tasks of small and medium-sized projects.
As for SpiraTest, it is a powerful but complex test management software, which can be suitable for big and complicated projects.
Hopefully, the provided information will help you to select the most suitable test-management tool for your needs.
