Author: Jeff McKenna, MCG Software, Inc. (mckenna@acm.org)
Copyright 1995, MCG Software, Inc.
We have been investigating the issues of testing Smalltalk applications fairly seriously for some time now as a result of consulting work. This has resulting in our generating a software product, OTF - An Object Testing Framework, that is currently offered in the marketplace. The following discussion highlights some of the areas that we find important when thinking about testing in Smalltalk. They are not in any particular order.
The concept of use cases or scenarios has been influential in OOA & D techniques. We have been finding that these use cases offer a rich source of test cases. Opportunities for the merging of the use case specifications into test specifications abound.
We have consistently been following a development model that treats all software development as 'maintenance'. By this we mean that once the overall architecture is developed, usually a fairly quick job, all changes can be classified as what is nominally called maintenance: Changes in understanding or requirements that cause changes in the software.
Maintenance carries with it the requirement of 'not breaking the old stuff'. It is not possible to do this without a comprehensive set of test cases to perform regression testing.
The much smaller granularity of Smalltalk objects, both classes and methods, requires a corresponding change in the granularity of testing. We must have tests that test each level of the system: individual methods, classes, packages, subsystems and the 'system' itself. Our estimates are that test bulk should be as much as 4 times or more the code bulk. One of the major failings of the major type of testing utilized these days is that it focuses almost exclusively on the user interface. This does not allow the kind of systematic or deep testing that should be performed. It also provides little feedback to the development process of exactly what is wrong.
The key to successful testing is to focus on the contract of the object at whatever level. This is really the old public/private problem brought to the front. We have said that one person's public is another person's private. At the class level, the public problem seems fairly clear but what does it mean at the package level? It should be sufficient to test the public interface at package level with package level testing provided that that interface is well understood and documented. We have seen little work in this area.
We have found that being able to automatically run regression tests after each build to insure that things aren't any worse than they were before dramatically improves team productivity. This has meant that SQA has had the power to reject builds immediately. Developers have not liked this at first but have found it advantageous in the long run.
When we discuss testing it is important to note that testing is not programming. Testing as programming is rather like asking the fox to guard the chicken house. Adding four times the code bulk to the code does not necessarily make for good testing. Testing tools must be much less intrusive than that. Testing is very procedural (as are use cases) and will remain that way. The tools must reflect that.
Much of testing in Smalltalk isn't any different than testing any other software system. Lots of hard work getting the data, seeing if it is correct, and molding that data into a form suitable for testing. Testing has low status and that is too bad since our industry is suffering from an error rate that many consider criminal. Hopefully objects and Smalltalk and object-centric testing can begin to change that.