The modern IT marketplace of the high technologies offers the multiplicity of great solutions for software testing. In this article we review the pros and cons of the most famous web-based test management tools: SpiraTest, TestRail and TestLodge. Every of the mentioned services is the comprehensive web-based test management software that allows to control, monitor and organize effectively the testing efforts of the QA department.

The main advantages of the aforementioned tools are: SaaS delivery model (allowing to avoid losses on installation and maintenance of the software), simplicity of the user interface (allowing to manage all the test artifacts in one workplace), a wide range of features (requirement management, test management, reporting management, etc.) and ability to integrate with the large number of bug trackers and frameworks.

Outline

Licensing

All the considered applications are commercial and offer 30 days trial version for familiarization with the product. Every of them presents its own licensing system, so SpiraTest and TestRail use the licensing arrangement based on the number of users , while the licensing rules of TestLodge are based on the quantity of activities (the amount of test plans, test cases and test runs) and do not depend on the number of users.

Access control

TestRail provides fine grained access control which allows to edit pre-installed roles, add custom roles and combine users into groups. Each role allows to set permissions for configurations, milestones, test-cases, attachments, test runs, test-results and reports. SpiraTest offers very similar access control system which allows to edit preset roles and create custom roles with a variety of permissions. As for TestLodge, the access control of this app allows to assign admin role to a user and create custom roles which define read/write permissions for test plans, requirements, test suites and test runs.

Integration with existing tools

All the mentioned test management applications support integration with a wide range of bug tracking systems, frameworks and add-ons. But, only SpiraTest offers its own built-in issue tracker, TestRail and TestLodge refused this idea, since they offer the large number of external bug trackers. The most popular of them are listed below:

Integration with bug trackers

SpiraTest TestRail TestLodge
Built-in issue tracker + not implemented not implemented
Assembla not supported + +
Basecamp not supported + +
Bitbucket not supported + not supported
Bugzilla + + +
Fixx not supported + not supported
FogBugs + + +
Gemini not supported + not supported
GitHub not supported + +
IBM Rational
ClearQuest
+ not supported not supported
Jira + + +
LightHouse not supported + +
Mantis + + +
OnTime + + not supported
Pivotal tracker not supported + +
Rally not supported + not supported
Redbooth not supported not supported +
Redmine + + +
RTC + not supported not supported
TFS + + not supported
TRAC not supported + not supported
Trello not supported + +
Unfuddle not supported + +
Vault not supported + not supported
Youtrack not supported + not supported


Working interface

TestLodge and TestRail provide the well-organized and simple UI, allowing to capture quickly an idea of a project. However, this simplicity may be criticized by experienced users who want all the sheaths at once. As for SpiraTest, this tool offers intuitive UI which includes a lot of filters to narrow down the number of shells on the workspace, but at the same time this app suffers from the multiplicity of the "edit" and "update" buttons, that force user to make a lot of redundant "clicks" to finish any edit and update operation.

Requirement management

TestLodge and SpiraTest offer built-in requirement management (RM) modules which allow to perform CRUD operations on requirements, attach files and assign test cases to requirements. TestRail does not provide built-in RM service, but supports integration with external services, like Jira. As for the ability to generate traceability matrix (a document that shows the test coverage of requirements), all the reviewed tools support it quite well.

Test design

All the reviewed apps allow to perform CRUD operations on the test-cases, but TestLodge and TestRail implement simple and fast method of test case management, which allows to add a test-case with the minimum number of details (title, description, steps, expected result).

The next point to review is a search functionality, here SpiraTest and TestRail are taking the lead, since TestLodge does not support this feature at all. The difference between SpiraTest and TestRail is in the fact that SpiraTest offers the global search through all the projects and test artifacts, and TestRail offers search only among the specific test artifacts and does not support searching by keywords.

As regards import/export, TestRail and TestLodge provide easy in use import/export without configuring, while SpiraTest requires to install and configure specific add-on.

Import/export formats of the test-artifacts

SpiraTest TestRail TestLodge
Import MS Excel; MS Word;
MS Project
XML Google spreadsheet
Export MS Excel; MS Word;
MS Project
XML; CSV;
Excel
XML; HTML


And lastly, the document attachments. All the reviewed applications allow to attach any kind of files to test cases, but in contrast to SpiraTest and TestRail, the number of attached documents in TestLodge is limited in accordance with the selected plan (non-trial).

Test execution

SpiraTest and TestRail display the execution results as a tree, while TestLodge presents test results in the row mode (test runs are listed in the rows which are located one by one on the main page of the service). Both types of the interfaces are intuitive and easy in use, but tree-based structure is more comprehensive, since the critical and valuable data are displayed clearer.

As for the test results tracking (document that displays status of test cases for a project and allows to analyze and monitor how the development work is going), here SpiraTest is a bit behind, because TestRail and TestLodge have deduced this feature into a separate unit, allowing to trace the history of launches and tests editing more precisely and clearly.

Summary table


SpiraTest TestRail TestLodge
General points
Trial mode 30 days 30 days 30 days
Licensing based on the
number of users
based on the
number of users
based on the
number of activities
Requirement management
CRUD operations + only through
external RM tool
+
Complexity of adding
a requirement
too many fields only through
external RM tool
simple
Structure of
requirements
tree-based structure depends on
external RM tool
row-based structure
Traceability matrix + + +
Integration with
external RM tools
+ + +
Test design
CRUD operations + + +
Complexity of creating
a single test case
too many steps simple method simple method
Search functionality global search limited search not implemented
Custom fields + + +
Import/export + + +
Documents attachment to the test case + + limits are set in compliance to the selected plan
Test execution
The execution interface tree-based structure tree-based structure row-based structure
Manual execution in a browser window in a pop-up window in a browser window
Test results tracking + + +
Defect management
Inbuilt (own) defect management system + not implemented not implemented
Integration with
issue trackers
+ + +


Conclusion

All the considered tools are well-organized test management services that offer a wide range of the capabilities and features, allowing to cover the most part of users' needs. TestLodge and TestRail offer thought-out and convenient UI which in combination with a sufficient number of the features, allows to resolve the majority tasks of small and medium-sized projects. As for SpiraTest, it is powerful but a bit complex test management software, which can be suitable for big and complicated projects.

Hopefully, the provided information will help you to select the most suitable test-management tool for your needs.

Looking to hire a quality assurance engineer?
Don't hesitate to contact us.