Quality Improvements to an Existing Application

Introduction

I was approached by a company that needed production support for a bespoke finance application they were using for monthly invoicing.

The application was written in C# .Net WPF WebAPI and MySQL and several hundred invoices were being processed a month through it in a particular way.

The contract developer they had engaged with on a freelance basis had left suddenly before the proper embedding of the application had been completed. It was reported to be “very happy path”.

I spent several hours looking through the codebase and found that what should have been a relatively simple finance application was in fact coded in such a complicated manner that I initially declined the work because I couldn’t guarantee being able to reliably fix bugs whilst not introducing new ones.

It transpired that other developers had felt the same and they were struggling to find someone to maintain the business-critical application. Some months the application had refused to load due to data quality issues.

Problem Definition

I felt the only way I could possibly support the existing application was to firstly develop a second version of it by aggressively refactoring and simplifying the existing codebase and database.

Preventing this from happening was the lack of detailed documentation combined with the poorly architected codebase such that I couldn’t fully understand the business and finance rules as implemented.

Furthermore, there were no automated or even manual test scripts to validate the correct operation of the application upon production release.

Bottom line: The current application was too complicated for me to maintain and yet the refactoring of it to a simpler codebase was being blocked by not having any way to ensure the application was still working as expected after any change.

Way Forward

I proposed the existing application be treated as a “black box” so that sufficient end-to-end integration tests could be developed to validate correct operation.

Until then, no production support of the existing application could effectively be done, and any that was performed would be minor to just ensure the application could be loaded and critical monthly tasks performed.

All bug fixes and new features beyond this would need to be performed on a newly built (version 2) of the application that would have a drastically simplified codebase to allow better on-going support and manageability.

The contract would stipulate that acceptance of version 2 would be the successful execution of the same end to end tests developed for the existing application. Should the development of tests not be achievable (ie. too much complexity) then an option to walk away from the engagement was acceptable.

Work Performed

An isolated test environment was relatively easy to create given a couple of factors that went in our favour.

Firstly, there were well-defined input and output files (which were manually orchestrated each monthly run) that could be used to drive the black box testing and check output against.

Secondly, the application UI was built in Microsoft .Net WPF with extensive use of view models which could be programmatically driven by the tests to replicate the required user interactions.

Figure 1. Isolated test environment which was created for the application

The following activities were required to fully complete the test environment setup and ensure it could also support the development of version 2 of the application:

• Initial check-in and versioning of uncontrolled source code to a private repository
• Creation of a unit test project to hold end to end integration tests (and later unit tests)
• Setup of a dedicated build machine to perform regular builds and execution of tests
• Working with finance users to define relevant input files / interactions during a typical monthly process
• Creation of an isolated test database, with the same schema as the production database, to be exclusively used for test running by the build process (Jenkins in this case)
• Development of test setup which recreates a known, pre-seeded test database with the required reference data prior to each test being executed
• Setting up of code coverage so that the test coverage of each build can be measured and compared
• Migrating the existing MySQL database to Azure so that transparent, whole database encryption could be used (replacing in-table encrypted columns which was limiting supportability)

Outcomes

Version 2 of the application was produced with a > 50% reduced code base and no loss in functionality.

Unplanned production outages due to application bugs has been drastically reduced.

Broad end-to-end integration test coverage for typical monthly processes has been achieved.

The development of many unit tests throughout the version 2 build to reduce overall reliance upon end to end integration tests alone. 

The following charts and images are taken from Jenkins, my build machine and show the actual quality measures in place and improving over time.

Figure 2. Test count trend over time

Figure 3. Code coverage report for the most recent build

Figure 4. Line-by-line code coverage of a given class