Track and Govern Continuous Testing with ALM Octane
Most organizations fail to establish continuous testing as part of their software delivery process due to missing governance on automation coverage as well as the measurement of key indicators. This is caused by not existing or wrong feedback on the business risks associated with a software release.
Before we start our continuous testing initiatives, we need to answer the following 2 questions:
- Why do we need automated testing as part of the continuous delivery pipeline?
- What is our expectation towards automation?
Use Automation as a Servant to the Teams
The need of automated testing as part of continuous delivery is crucial to the speed of delivery and to obtain early and fast feedback during the release execution. This will also reduce the manual process in executing test cases and help organization to focus on exploratory testing. Through this we can achieve the following key performance indicators:
- Reduce development cycle time — Good automation should help increase the development speed.
- Reduce regression cost — By investing in automation, you can reduce repetitive execution of manual regression tests and improve your testing strategy by allowing QA to focus on other essential tasks.
- Reduce defect cost — By comparing the number of defects detected early by automation (low cost defects) vs. defects submitted after manual runs (high cost defects), you can determine the cost of defects.
- Reduce number of escaped defects — The escaped defect reflects how many defects were missed by your testing activities. It also indicates whether your testing strategy is improving when you reduce the number of escaped defects that reach production.
The expectation from automated testing as part of the delivery pipeline is a common team goal. Automation always serves a purpose and especially in quality assurance of a software delivery it becomes a servant for the agile teams to drive the key performance indicator towards speed and faster feedbacks.
In addition to the key performance indicator mentioned above, one of the main automation ROI benefits is the early detection of defects. Tests that are part of an automated pipeline can detect defects at an earlier stage than manual or production tests. The earlier a defect is detected, the lower the development cost and the better return on investment (ROI) for your automation.
Track the transformation of tests (executed manually) into automation as part of your delivery pipeline.
To start implementing continuous testing with high automation ratio, you need to invest in the right selection of tests which should be automated and are executed manually today. Whatever your selection criteria (Risk-based, Requirement-based, etc.) are, make sure to track the transformation of tests (executed manually) into automation. This helps you to streamline the conversion of test scripts to automated tests.
Once your selection is done, make sure to apply a workflow to gain insight on the transformation. For instance, all tests which are selected can be marked with “Ready for automation”.
ALM Octane allows you to implement a workflow for transitioning tests into automation by a pre-defined status model. This includes the Automation Status highlighted above, which can be implemented on the Test Type:
- Manual Tests
- Gherkin Tests
- BDD Scenarios
To understand the overall process, watch the following tutorial on our Octane Academy on Youtube:
Govern Continuous Testing through the Automation ROI
Finally, once the transition of your test automation journey has started, you can start governing your key performance indicator to identify as early as possible, if the automation is implemented for the right areas.
ALM Octane provides an Automation ROI Widget in the Dashboard module to measure the impact of automation on the overall release execution across multiple releases. As automated testing is part of a continuous process in continuous testing, the Automation ROI Widget provides all relevant insights to justify a return on investment or change the direction to be more focused.
ALM Octane uses the automation coverage calculation to indicate,
- that automation is in place and has a satisfying ratio to tests
- if automation is applied correctly to see a decrease in the following indicators:
- A reduction in the development cycle
- A reduction in the regression cost by shortening the regression cycle time
- Improve QA efficiency by eliminating the need to execute repetitive manual testing efforts
- Lower the risk level and improve the system’s stability
The Automation Coverage in ALM Octane is calculated as follow:
This is then applied to the release execution to measure if
- Development cycle time has been affected (preferably decreased)
- Regression cost, the manual execution of tests has been reduced
- Defect cost, the detection time is fast to obtain quick feedback
- and the overall escaped defects number is decreased.
If none of the criteria is met across releases, this is an indication, that your automation transformation is not focusing on the right areas and you must rethink the selection process.
Otherwise, you will gain insight across releases by introducing Automation:
To see it in action, visit our Octane Academy on Youtube:
