Automated Testing in IBM Smalltalk and VisualAge

Authors:

• Michael Silverstein, IBM Corporation RTP, NC (msilverstein@vnet.ibm.com)
• Mark Foulkrod, IBM Corporation RTP, NC (foulkm@carvm3.vnet.ibm.com)

Copyright International Business Machine Corp., 1995 All Rights Reserved.

IBM and VisualAge are trademarks of the IBM Corp.

NOTE: This article was provided untitled; title added for HTML by editor

A few simple tools can make large scale Smalltalk product testing manageable. The Advanced OO Validation group at IBM is responsible for testing releases of IBM Smalltalk base and add-on frameworks (features) as well as the VisualAge product's visual construction tools. The task has become more daunting in the face of product availability across four platforms and counting. Other tasks include verifying prior release fix packages, demonstration versions of the product and regression testing against prerequisite software updates.

In order to address these challenges, the group has started to invest in developing a testing framework. The requirements for the framework were to:

• enable automated test cases to be written in Smalltalk/VisualAge.
• use the same version & release control mechanisms as the product.
• provide for documentation of test cases.
• provide for interactive test case debugging.
• allow for non-automated test cases to be executed consistently.
• report test execution results.

There are several benefits to using Smalltalk itself as a basis for testing a Smalltalk product. First, there is no need to learn a separate test scripting language. In addition, test cases can have intimate knowledge of all of the objects in the system. When problems are uncovered, the native Smalltalk debugging tools may be used to interactively determine the cause.

The test framework that creates, manages, executes, and reports the results of the test cases is comprised of:

• GUI based views for running tests, viewing their results, and accessing their documentation.
• Classes that provide test specific behavior such as execution, condition checking (assertion) and exception handling.
• Extensions to the base classes for random test data generation (random instances).
• Classes that wrapper widget and VisualAge based views to streamline their testing.
• Generic test methods that can be run against any VisualAge parts.

A TestRunner view provides the user interface for test execution and documentation. From this view, available test cases are listed for users to execute. It also allows users to construct, save, and restore customized test case 'buckets' as a fast means of defining multiple tests to run. Test cases may be run in either overnight mode, where all errors are captured to a log and the next test case is resumed, or in interactive mode, where errors may be handled as they occur using Smalltalk debugging tools. Pass/fail results, condition checks, and exception stack captures are logged for each test case and displayed after completion.

The test framework has been implemented mainly at the base Smalltalk layer with extensions for VisualAge. At the this layer, the framework enables test case developers to affect and query the state of classes under test. It also handles many of the details of dealing with widgets by providing simple methods for clicking on pushButtons, retrieving strings from textWidgets, etc. For VisualAge end-user applications, the test framework provides similar APIs for manipulating parts and querying their states as the base layer. Because the visual editors are themselves visual parts, we have added a further layer of wrappering to abstract many of their operations that enables us to write test cases that check the edit-time behavior of VisualAge parts.

We have devised a set of general tests that may be applied to any VisualAge part, allowing us to instantly begin to stress test a newly developed or acquired part without having to create new and specific test cases. Other methods enable us to create use case scenarios that stimulate features of a part and compare the part's reaction to those stimuli against an expected pattern.

Because the group has only recently started to develop and use these tools, they are still new and evolving. Initially, designing and debugging automated test cases has been time consuming. Although the investment in tools has been great, the return has been greater. As the test framework has matured, the following benefits are realized:

• The tools require less attention
• The process for creating tests becomes more streamlined
• Opportunities for reuse increase.
• Overall product test coverage increases
• Temporary testers may be employed without incurring a steep learning curve. In many cases, intimate knowledge of the product is becoming less of a prerequisite to being able to test it.

With the test framework in place, we are now able to explore more advanced testing techniques. The framework began as a basic effort to provide a convenient mechanism to manage test cases and to manipulate objects, with a focus on user interface parts and widgets. Our focus has now turned toward providing generic/abstracted test techniques for entire groups of classes. We have succeeded to some degree by providing the beginnings of these techniques for VisualAge part providers in the form of generic tests described above. At present, we are experimenting with effective techniques for testing with randomly generated instances.