Investing in test automation is mandatory for digital transformation initiatives. There are two main reasons. First, testing is a significant cost in development projects, so there is a strong motivation for cost reductions. Second, there is always a demand for higher quality. Test automation promises more exhaustive regression testing cycles, which drives higher quality.
However, test automation can lead to two issues within the larger context of a digital transformation initiative:
-
Creating more silos and losing visibility: The ideal test automation tool often depends on the specific system or product being tested, as well as its technical architecture. Additionally, we may need to use several tools to cover different types and levels of tests, like API or UX tests. While it is crucial to select the right tool for the job, we cannot neglect integration. Otherwise, we risk losing visibility into test coverage status. Without centralized visibility of coverage, we may find ourselves back in a typical testing conundrum: the challenge of striking a balance between over-investing and under-investing in testing efforts.
-
Automatic test results are not leveraged for release documentation, forcing the team to duplicate tests across the manual and automated test types. I have seen automatic test results are not utilized for release documentation, forcing teams to duplicate tests between manual and automated tests. These teams continue to run some of their manual tests, despite the existence of parallel automatic tests, to ensure they have it documented. This is inefficient — human testers should focus on finding bugs instead of performing formalities, and it also delays releases.
Anchoring our SDLC in Jira helps avoid these pitfalls. To optimize our test automation efforts, we need to integrate our command and control center in Atlassian and our test automation pipelines. This integration addresses two key issues: First, program managers gain visibility into the progress of test automation efforts. The current status and results of automated test executions are available in Xray and feed into the Xray reports. Second, all the data in Jira and Xray is readily available for release documentation, just as with manual tests.
We added a couple of custom fields to the Test issue in Jira so that the QA can specify whether we need to automate this Test Case. If the Test is a candidate for automation, there is an extra field that keeps track of the status of the development of the automated Test. This helps manage the test automation effort.” 
Tony Sexton - QA Manager at Caris Life Sciences
Can we integrate any test automation platform with Jira and Xray?
Yes, you can integrate tests and results from any test automation platform with Xray. The specifics of the integration will vary, impacting the complexity of the setup process. Some common platforms can generate XML reports that Xray's API can easily parse. In some instances, you can also use commercial third-party connectors, such as the Automator connector that works between Tosca and Xray.
If a ready-made implementation isn't available for your test platform, Xray provides a comprehensive API, including one that utilizes a standardized JSON format for importing execution results. With this API, there is always an option to develop a custom bridge between Xray and any specialized test automation tool. However, in most cases, a more straightforward solution is available.
What does the integration with test automation platforms look like?
The integration between automatic tests and Jira and Xray has two elements:
-
Every test in the test automation suite links to a single test in Jira/Xray. Throughout its lifetime, this single Xray test item is executed numerous times. Each execution is represented by a test run in Xray.
-
Importing results from test runs into Jira: Each execution of the test pipelines imports the results into Xray. This process contributes to the overall reports generated in Xray. Additionally, test runs feed into the release documentation (like the traceability reports). The test pipeline also imports the evidence generated during each test run. This feature is beneficial for collecting comprehensive test evidence.
Identification and linking of test cases.
Each automatic test is linked to an Xray test item. There are two methods to establish the link between Xray tests and their automatic test implementation:
-
Start with Xray: Create the Xray test first, then connect it to the implementation in the automated test suite.
-
Begin with implementing the automated test. When the pipeline imports the initial execution results of this test into Xray, this will automatically create a new test case during the import process. Subsequent test executions will be linked to the original Xray test created from that first import. A valuable aspect of this process is that the pipeline can also import links to requirements items in Jira. If a test developer is writing a test to cover specific requirements, they can annotate the test code with references to the corresponding requirement work item in Jira. This annotation will establish the link within Jira when the test case is created.
The ideal flow for you will depend on the context. For example:
-
The need to create integration or feature tests is often scoped as a work item in the scope of a Sprint. It is common practice to create a Test in Xray to provide visibility into the progress of this testing task through various stages, such as implementation and code review. In these cases, it is preferable to follow the first flow: create the Xray test first and then link it to its implementation once it is ready.
-
Programmers typically develop unit tests as part of their work on each story. In those cases, it is more efficient to use the second approach: the test item in Xray will be automatically generated upon the first import of test execution results.
Transitioning from automated testing to automated release documentation.
Setting up an integrated Software Development Life Cycle (SDLC) platform that incorporates automated tests requires a significant initial effort. Still, it yields substantial benefits over the lifespan of your system or product. One of these benefits is the ability to integrate your automated tests with the release documentation.
By storing test results and evidence from your automated tests in Xray, you can effectively demonstrate test coverage and enhance traceability. For instance, release documentation that utilizes snapshot tables will effortlessly include both manual and automated test results.
Integrated test automation maximizes the ROI from your test automation efforts.
To summarize, test automation without integration with Jira and Xray can be problematic and may hinder your digital transformation goals. It creates silos and leads to unnecessary duplication of both manual and automated testing efforts.
Xray offers a robust API that allows for the integration of various automated testing platforms. You can connect multiple automated testing tools to a single Jira site, enabling you to support all the spectrum of automation tools your organization needs.
When you integrate your test automation tools with Xray, program managers gain valuable visibility into the status of test coverage. Moreover, your streamlined system for release documentation leverages the results from automated tests. As a result, your digital transformation project is now operating at peak efficiency.
Got a question? Let's dive in