Client Situation:
A legacy health exchange middleware project, Electronic Health Exchange (eHX), had generated a complex set of functions that had been built with very little automated testing in place. The federal project owners knew that they needed greater stability and faster turn-around but were unclear on how to accomplish this. The project had been run by a large government technical contractor which had focused on producing functionality without much emphasis on automated testing.
Business Impact:
The eHX project was at risk of missing government mandated deadlines that would significantly impact the health care veterans would receive. As with most projects that fall behind on their release schedules, the eHX project was experiencing both development delays and high defect rates as a direct result of incomplete regression testing.
No automated tests had been written in the early stages, and as the project progressed and philosophies such as Test Driven Development became more common, the expertise and advocacy for such approaches were not built into the team. Therefore, the code base continued to grow to a substantial size until, as with many projects, the volume and significance of defects measurably impacted the success of the project. They began to institute some manual and automated regression tests, however, again due to the significant number of interdependencies, the total code coverage was fairly insignificant.
When the government contract came up for renewal, the incumbent prime contractor lost the bid to VetsEZ, another prime government contractor with substantial software development experience. VetsEZ quickly recognized the serious lack of continuous testing and subcontracted with Paladin & Archer because of their expertise in that area.
Resolution: Implement the Continuous Testing Framework
Paladin & Archer (P&A) implemented the Continuous Testing Framework using a partnership-based approach with VetsEZ.
P&A began by conducting an assessment to determine the current state of quality in the project including capacity, capability, groundwork, and the best place to start. Analysis of the capacity of the existing project team showed that they did not have internal resources available for the implementation and would, therefore, need additional people added to the effort. Analysis of the capability of the team showed that they did not have the specialized training in continuous testing or continuous quality but did have a number of individuals who could be trained. Analysis of previous groundwork laid for the implementation of continuous testing showed that it was not significant. Few tests had been written and those which had been were flawed in their architecture and would produce false positives. Further analysis of the groundwork included assessment of existing tools of which there were none; however, VetsEZ was open to using any tools available that were approved by the federal client. Finally, as per the Framework, the assessment discovered the 20% of the functionality that would deliver 80% of the benefit of continuous automated testing.
After P&A completed the assessment, they produced a roadmap and plan for implementing the continuous testing strategy. P&A identified the appropriate tools for version control and continuous integration. These were:
- SoapUI and JUnit for test automation
- A custom GUI (built by P&A) to allow non-engineers to run the tests
- Jenkins for continuous integration and continuous delivery
- Bitbucket (Git) for version control
Upon careful review in validating the existing tests, P&A recommended that they be rewritten to produce more than false positives. Using the 80/20 method P&A then defined new tests including test data and mock services that would return the highest value in the shortest amount of time. Finally, the continuous testing standard operating procedures were defined, the entire system was documented, and the individuals responsible for the system were determined.
Once the roadmap was complete, P&A developed and installed the continuous testing framework. First P&A acquired and set up the approved tools and continuous integration mechanisms. Then P&A began writing tests based on the backlog prioritization. It is important to note that this backlog did not introduce additional complexity to the existing development process; rather, the existing development backlog was used. Once a few tests were ready P&A set up mechanisms to monitor the entire continuous testing process. As new defects were discovered they were added to the backlog. Finally, P&A created documentation for the entire process and trained the testing staff on how to write and execute the tests.
The final step in the process was to establish a monitoring and maintenance schedule. At the writing of this document this maintenance schedule is ongoing. The process includes reviews to reassess the current situation, update the existing plan, and trigger development of new tests.
Outcomes:
As a result of the solution Paladin & Archer implemented with VetsEZ for the eHX project, the following outcomes were achieved:
- Defects found after development systematically decreased each Build (B):
- Defects were found that were initially passed by the Quality Assurance Team and User Acceptance Testing.
- VetsEZ increased its Quality Assurance Surveillance Plan (QASP) rating to 5.0/5.0 on multiple reviews and as of this writing continues to do so.
- Defects were found not only in code but also in the requirements definitions by using Test Driven Development. This shifted the focus on quality left on the software development process.
- Changes that had previously been determined to be too risky were now able to be released.