On September 26, 1983, three weeks after the Soviet Union shot down a Korean Airlines passenger jet carrying a US congressman and raising Cold War tensions to new heights, Lt. Col. Stanislav Petrov was quietly sitting at his desk in a bunker near Moscow when he received some bad news. Satellites had detected five ICBMs coming from the United States and they were headed his way.
If these missiles were real, then protocol was clear: the Soviet Union was to launch a full scale nuclear counterattack.
Thankfully, Lt. Col. Petrov suspected that the US would make more of a show of starting the apocalypse with only five nukes and correctly determined that the system was malfunctioning. What was the issue? A computer program monitoring incoming early warning satellite signals, designed to filter out the noise from light bouncing off high altitude clouds, had not been fully tested, and the program let those erroneous, and potentially catastrophic, signals through.
Consistency in Software Testing
Software testing is the activity of verifying that software behaves as expected. That is, given a known set of inputs, testing verifies the software produces expected output. Given such a broad definition of testing, we often find it helpful to categorize discussion along several dimensions. In this post, we discuss one of the most important dimensions: level of consistency – the degree of regularity in testing a piece of software or a software system, and the reliability of executing those tests correctly each time.
Level 0: No Testing
At the lowest level of consistency in testing, we have software with no testing plan. While atypical, clients do occasionally bring projects with code bases lacking any record of testing. For example, the client may have “legacy code” that the original developer has long abandoned. The software may compile. It may seem to run without errors. The client might even use the software in production at the moment. Without a thorough suite of tests, however, every line of code must remain suspect for bugs.
Level 1: Manual testing
Manual testing sits above no testing as the next level of automation. In manual testing, one or more users submits one or more input files to the software, and then inspects the output for correctness. Often, but not always, the developers who wrote the software also hold the responsibility for performing such manual testing. Some projects may have an individual, or even an entire team, devoted to manually testing the software under development.
Manual testing can become expensive in terms of both time and money. Manual testing scales poorly with software complexity. Each new feature grows the number of test cases combinatorially, but increasing the number of testers can only reduce testing time linearly at best. When developers themselves must perform the tests, projects suffer an additional penalty because development of new features must halt to test existing features.
Manual testing relies on humans to perform testing tasks consistently across multiple test cases, between multiple releases, etc. As time progresses on the project, the probability of that even the most meticulous tester will make a mistake during testing approaches certainty. In the best scenario, this lapse leads to additional costs in time and resources chasing down a bug that doesn’t exist. In the worst scenario, the software ships with bugs because a test seems to pass when it shouldn’t have. This can happen when a tester misinterprets results or the tester has accidentally run the test with the wrong set of inputs. Running a test with incorrect inputs is not unlikely when a tester has to enter round after round of inputs for different test scenarios.
Level 2: Automated testing
Automated testing provides the highest level of consistency. In automated testing, the software project includes a suite of tests executed as a separate software program. For example, a shell script that invokes the software with the same set of input files and verifies the output files’ contents match expectations can provide a very rudimentary level of test automation.
In projects with the most rigorous levels of test automation, a complete set of verification programs – known as “test code” – accompanies the “production code” of the software itself. Oftentimes, the developers write test code in the same programming language or languages as the production code. This allows them to write very fine-grained tests of individual portions of the code base, test parts in isolation from each other, and make it easy to test the software under extreme conditions that can be hard to replicate or synthesize just through a program’s inputs alone. In the best of these scenarios, every line of production is covered by one or more lines of test code.
While few projects attain 100% code coverage by tests, even more modest levels of automated testing can provide significant increases in stability and reliability of software. Automated testing leverages computers’ greatest strength: following instructions exactly and identically, every single time. The test programs will always invoke the production program on the same set of inputs in the same way, and perform the same set of output verifications the same way, every time they’re run.
5AM Solutions Testing Practices
5AM Solutions has a culture of working smart, and it is hard to get much smarter than automation. Automation can produce orders of magnitude improvements in speed, cost, and reliability; and automated tests are no different. It takes only a few lines of code to write an automated test, and that test is likely to execute in a fraction of a second without ever getting tired or making a mistake.
There are also quite a few additional benefits of automating tests beyond automation’s traditional perks. Automation makes it easy to run tests at regular intervals such as when a developer adds a new feature or fixes a bug. Running the tests can indicate whether changes to the code worked and whether those changes had any negative repercussions on other parts of the code base. In this way, developers can avoid disasters by failing early in development rather than late in the production cycle, or worse: after the software is released.
Automated testing also gives more power to 5AM’s developers. In contrast to waiting on armies of software testers to push and pull the program until it breaks and report back, developers at 5AM can be responsible for writing tests for their own code. This provides us with an immediate feedback cycle.
Similarly, in contrast to manual testing, automated testing can scale with software projects as they grow. Each test our developers write remains in place, continually ensuring that the code it tests works as expected, and raising alerts when changes break the expected behavior.
One of the greatest benefits of automated tests to 5AM is that they allow developers to work together as a team. With test code, each developer can develop his or her components in relative isolation. Further, the test code serves as a contract for the production code. It guarantees and documents the production code’s behavior and allows other developers to ascertain the correct use of each component.
5AM Solutions demonstrates its belief in and commitment to automated testing by making it an official part of our “definition of done.” At 5AM, a feature is not done until it has appropriate, high quality, automated tests to go along with it. This is a commitment, to our clients and to ourselves, that we are producing the most maintainable, correct, and well-engineered software possible. Without this commitment, we could not confidently stand by the integrity of our code which would negatively affect the lives of the researchers, clinicians, and individuals who our product reaches everyday.
Testing is important, and that is why we do it.