A Client-Based Software Engineering Project Course

 

Peter Smith (psmith@cs.ubc.ca)

The Department of Computer Science,

The University of British Columbia

 

1. Introduction

In order to complete a degree in Computer Science at the University of British Columbia, students are required to complete CPSC319, Software Engineering Project. In this 3 credit, two semester (8 month) course, students work together in a team environment (with roughly 8 people per team) to design and construct a software product.

          Before embarking on their project, students will have taken courses in basic programming, as well as in the use and design of data structures, currently using C++.

In addition, they will have taken a software engineering theory course where they become familiar with the basics of analysis, design, testing, and other software engineering practices.

          In the last 2 years, CPSC319 has undergone dramatic changes. For example, the introduction of client-based projects has added realism to the team-work experience, and the relaxation of the grading system has permitted more freedom in work allocation and scheduling. We will examine these changes and will discuss why they were necessary.

 

2. The Original Course Format

In its traditional format, CPSC319 was structured as a series of assignments (known as milestones or deliverables) intended to guide the students through the steps of completing a software engineering project. These milestones included the analysis and design documents, several versions of the implementation, testing specifications, testing results, and a user manual. Milestones were spread throughout the 8 month period at roughly 2-3 week intervals.

          The project teams were created in the first few weeks of the course. For each milestone, the teams would divide up the necessary work as evenly as possible amongst their members. Total cooperation between team members was strongly encouraged, although cooperation between different teams was not permitted. There were no exams or individual assignments in the course.

          Each student was expected to attend a weekly team meeting (where a teaching assistant would be present) as well as a weekly lecture. The lectures would give the course instructor an opportunity to discuss the current milestone with the entire class. Students would also need to spend as much time as necessary in smaller groups, or on their own, in order to complete the assigned work.

          For many years, the same project, an Airline Reservation System was assigned. This project required the students to implement a database component (written in C), to manage the storage and retrieval of information about flights, booking agents, and passengers. Secondly, a Graphical User Interface was added to provide a user friendly view of the system. Each team would implement exactly the same project at the same time.

 

3. The Motivation for Improvement

In an ongoing effort to improve the Department's Software Engineering curriculum, several problems with the format of CPSC319 were identified. These problems were discouraging students from fully enjoying, or benefiting from the course.

          The Airline Reservation System was an artificial project. Each team would produce exactly the same product, with the knowledge that their work was purely for academic reasons, and that their software would be discarded at the end of the course. Also, since the same project had been offered for many years without being significantly updated, the students believed that the requirements were too inflexible and did not permit them to demonstrate their creativity.

          To simulate a real world software engineering project, each milestone would contain various difficulties. For example, sudden changes in the requirements would be introduced after implementation had started, team members would be moved to other teams, or the students would be asked to use development tools that contained known bugs. Although these situations are very realistic, the students saw them as artificial, and resented any forced changes of this type.

Finally, the project workload was considered to be too high for a 3 credit course. Frequent complaints indicated that either the course should be worth 6 credits, or the workload should be drastically reduced to a 3 credit level.

 

4. The Revised Course Format

Over the last two years, CPSC319 has been updated quite dramatically. We will now discuss the structure of the 1997/98 course offering. The main changes include the introduction of client-based projects, the reduction of workload, the improved focus on project management skills, and the simplification of the grading system.

4.1 Client Based Projects

Perhaps the most important development was the introduction of client-based projects. Rather than assigning a contrived project, the instructor was responsible for locating real clients who had the need for custom-designed software. As this approach was fairly experimental, only members of the UBC campus were approached for ideas. All clients were happy to offer their ideas, on the understanding they would receive the final software product for free, even though it may not be fully functional.

          This year's projects included a room booking database, a project management tool, a problem tracking system, a visitor booking system, a course marking program, a simulation result graphing tool, and a volunteer information database. Each program required the construction of a database component (using C++) and a graphical user interface (using Tcl/Tk). Each of these projects was assigned to only two teams, rather than to the entire class.

          Before releasing the project details, the instructor spent several hours with each client to discuss their requirements. Eventually, each project could be described in approximately 3-4 pages of formal English, in addition to any paper-based documentation, sample file formats, or tables of data that the students would require.

          To ensure that all of the projects were of the same difficulty, the instructor was responsible for reducing or simplifying the requirements on some projects, as well as increasing the level of difficulty on other projects. It was considered important to stretch the abilities of each team, without overloading them with too much work. Finally, although most of the main ambiguities and problems were removed from the requirements, many small problems were left intact for the students to discover and deal with on their own.

4.2 The First Term

In the old format of CPSC319, students would start their project as soon as the teams had been formed. One observation was that many of the students were not prepared for the development task ahead of them. They were required to gain the necessary skills while undertaking the project, which clearly slowed down their performance.

          In the revised course format, the project did not start until the second term. Not only did this reduce the size of the project (and the student workload), but it gave an opportunity to learn the necessary skills in advance. Upon entering the second term, students no longer had to learn the important concepts on their own.

          The lecture topics that were offered in the first term include GUI design concepts, programming in Tcl/Tk, using Unix commands, using the C++ Standard Template Library (STL), debugging, writing a project proposal, interacting in a team, and efficient team management practices. Although only one lecture was offered per week, a more interactive tutorial session (also of one hour per week) enabled students to gain hands-on experience with the concepts taught in lectures.

          To confirm that the students had developed the required skills, each student was asked to complete two programming assignments (on an individual basis, rather than in a team). The first assignment was to construct a Tcl/Tk-based GUI for a Unix command, whereas the second assignment gave them experience in constructing a C++-based package using STL to store data.

          The first term concluded with a two hour examination of all the knowledge acquired during lectures, tutorials, and in the two assignments.

4.3 The Second Term

Shortly after the first term's examination, students were placed into teams and assigned projects. Teams were formed by ranking students by their examination grade and ensuring that each team contained students with a range of abilities. During the Christmas break, the team members were permitted to study their project descriptions, and to form an initial idea of how they would design the software.

          During the first three weeks, the teams were asked to produce a written proposal (approximately 10-15 pages long) of how they to intended to complete the project. The proposal included a description of what the final software would look like, including sample usage scenarios and possibly some hand-drawn screen shots. Next, the team would estimate how long they would require to complete each deliverable (for example, analysis documents or testing specifications) and which team members would be responsible for them. Due to their inexperience in estimation, as much guidance as possible was offered by the instructor and teaching assistants.

          After the first month, each team met with their client and gave a brief presentation of their ideas. This meeting permitted both parties to discuss the proposed view of the software, and to clarify or amend the fine details in the requirements. To reduce the workload of the client, the students were expected to spend the remainder of the term working on the project without further client interaction. However, several of the clients expressed interest in participating more closely by having extra meetings, or by interacting via email.

          Throughout the remainder of the term, the team worked on the project fairly independently. There were regular weekly meetings where the team could discuss their project, and seek guidance and approval from their TA. In some cases, particularly in the early phases of the project, further guidance was requested from the instructor or the client. Generally, the teams were familiar enough with the software engineering process to work independently.

          At the end of the semester, the teams were required to present their product to the client by giving a brief presentation, and by officially handing over the software and documentation. Given the limited time span of a single university course, the students were not required to alter their software after this point in time.

4.4 Encouraged Work Habits and Project Management Skills

In the old format of CPSC319, there was little or no emphasis on project management skills. Teams were simply left to struggle through the project work in the best way they could, rather than be required to follow efficient working guidelines. In the new format of the course, the teaching assistant is treated as a member of each team who offers as much knowledge and advice as possible, without actually doing any of the work.

         

The students were expected to follow some basic guidelines:

    To complete all of the software engineering documents (such as design and analysis models) that were discussed in their previous software engineering theory course. To reduce the time required to produce these documents, hand drawn diagrams were acceptable, as long as only the team members themselves would be reading them.

    To follow their planned schedule as closely as possible. If they failed to keep to schedule, they must have discussed and understood why their estimation was incorrect, possibly updating their schedule or altering their expectation of the workload.

    To predict and deal with risks. For example, to recognize that a particular person was unable to help with part of the project, and therefore work around them.

    To support other members' work, and help them with problems that arose, rather than blaming them for not being able to complete the work.

    To keep accurate records of all decisions that were made. This would normally be in the form of minutes, agendas, and a Gantt chart.

    To submit only software that has been fully tested, although it may be limited in functionality. They were discouraged from including extra features that were not correctly implemented.

    To ensure that the software met the client's requirements. Even though the client did not have much opportunity to comment on whether the software was acceptable, the teaching assistant took careful note of whether the requirements were being followed.

By following these guidelines, the students gained an understanding of how to work efficiently, and to produce an acceptable final product. This provided a great benefit over the old format of CPSC319 where these practices were not enforced.

 

5. The Grading System

In a Software Engineering Project course, it is very difficult to grade students in a fair manner. Given that each piece of work could have been completed by any number of students within the team, it is difficult to accurately determine who should be rewarded for the work.

          When constructing a new grading system, there were several factors that were taken into account. The instructor must ensure that the final grade received by the student is an accurate reflection of the student's ability to complete a software engineering project. Also, the student must receive adequate feedback during the year so they may learn from their mistakes, and potentially improve their grade before the completion of the course. The new grading system for CPSC319 addresses these issues.

5.1 The Old Grading System

In the old format of the course, each deliverable was assigned a percentage weighting (roughly 10% for each of the 10 deliverables). A student's final course grade was based partly on the quality of the team's deliverable, as well as that student's own participation in the work.

          To grade a deliverable, the teaching assistant would review the work and comment on any mistakes they found, or would suggest improvements the team could make for future deliverables. Given that each teaching assistant supervised 5-6 teams who were all submitting the same deliverables at the same time, it was fairly easy to make a comparison of the work. Instead of grading each deliverable on a percentage basis, they were divided into the broad categories of: Excellent, Very Good, Good, Poor or Fail.

          To evaluate a student's individual participation, team members were periodically asked to write an evaluation. These evaluations would contain a half-page description of the work they had completed, as well as some brief comments on the work completed by fellow team members. The teaching assistant would then read these evaluations and assign students to the same broad categories.

          The most notable limitation of this grading system was that it incorrectly awarded grades to the upper or lower students in the class. For example, it was possible for a student to pass the course, based primarily on their team's work, rather than their own. From the other perspective, students with an excellent academic record could be disadvantaged by a low-performing team. In this second case, disputes between students and teaching assistants were common.

5.2 The New Grading System

The revised grading system is based around the idea that CPSC319 should in fact be two separate courses. The first term is similar to any other course with lectures, tutorials, assignments and an exam. The work undertaken by the students in this term is entirely individual work, and constitutes 80% of their final course grade. The second term is entirely team based, although only the student's individual participation in the project is graded. The second term's work is graded with a broad numeric rank from 0-5, and makes up the remaining 20% of the final course grade.

           It is important now to discuss the decisions behind this grading structure, in particular the reason for only assigning 20% to the second term of work, even though it makes up roughly 60% of the course workload. Also, we will discuss the way in which the broad numeric rankings are used to indicate the student's ability in the course, and how it encourages them to learn from their mistakes.

5.3 Pass/Fail or Percentages?

One of the major flaws with the old course format was that the high workload was considered unacceptable, and as a result, students would neglect their other courses. One solution would be to make CPSC319 a Pass/Fail course, rather than assigning percentage grades. This would be beneficial to most students who simply wished to gain the experience of working in a team, without overworking themselves to obtain a high grade. Students who were not completing a satisfactory amount of work would be warned about their risk of failure.

          On the other hand, a Pass/Fail course would not be entirely satisfactory. Firstly, without a more detailed grading system, potential employers have no means of comparing two students who both passed the course. Secondly, the students who wish to maintain their high academic record, or simply to put more effort into the course, would no longer have any incentive to do so.

The solution of assigning 20% of the final course grade for the project can be seen as a compromise. The weighting is not high enough for it to dramatically affect a student's grade, so they are not required to overwork themselves. Conversely, it is high enough to benefit those students who wish to aim for a better standing in the course.

Finally, the primary objection to a 20% weighting is that students quite-rightly believe that the amount of time they spend on course work should be related to the weighting of that work. After explaining this grading rationale, students appear to be content with the system, although we do not currently have enough evidence to support this belief.

5.4 Ranks

The approach taken for assessing a student's grade on the project was designed to follow the model used in the software engineering industry. There were six possible numeric ranks (from 0 to 5), to represent the student's promotion level. As in industry, a student who performs excellent work will be promoted to the next level, whereas a student who does not perform adequately will be moved down a level (similar to a warning that a manager might give an employee).

          At the start of the second term, each student was assigned rank 3. At two week intervals throughout the semester, the instructor and teaching assistants would meet to discuss the progress of each team, as well as the characteristics required for students to be promoted or to be warned. To ensure this was done correctly, the teaching assistants spent a sizeable amount of time reviewing (and offering suggestions about) each team's work, as well as determining which team members had completed that work.

Many students remained at rank 3 for the whole course, since the work they completed was considered acceptable. An equal number of students were moved up to rank 4 since they had surpassed our basic expectations. At either end of the scale, approximately 10% of the students obtained a rank 5 since they demonstrated exceptional qualities, whereas approximately 5% of the class had been demoted to rank 2 due to poor performance.

          It is important to note that only the student's final rank was used in calculating their final course grade. Throughout the course, a teaching assistant could alter a student's rank to give them feedback on their progress. Being promoted to a higher rank indicated to the student that their effort was worthwhile. On the other hand, a rank of less than 3 was seen by the students as an unacceptably low grade, encouraging them to work harder for the remainder of the course. In an extreme case, any student who received the rank of 0 would immediately fail the course.

          In order for a teaching assistant to accurately assign each team member a rank, the students were asked to inform their teaching assistant of how they were progressing. Rather than submitting written evaluations (as with the old course format), students were encouraged to appeal their current rank, or the rank of a fellow team member, if they felt a mistake had been made. At this point, the teaching assistant could either correct the student's rank, or inform the student of how they could improve their work habits for the future. This type of feedback became a very important mechanism for both teaching and grading the students.

         

 

6. The Student Experiences

By updating CPSC319, we have solved the major problems that were discouraging students from fully enjoying, or benefiting from the course. In addition, the students are now gaining several new experiences. The improvements offered by the new course format are:

    The introduction of client-based projects has noticeably increased the morale of the students. Rather than constructing the same project every year, each team feels that their project is worthwhile, given that a real client is expecting to use their product when it is completed. For many students this motivation has extended beyond the desire to simply obtain a high grade, as it has become a personal challenge to complete the software.

    Any problems that arise during the project are considered to be realistic, rather than being introduced by the instructor. For example, students learn to appreciate that clients often provide ambiguous requirements, or change them unexpectedly. In addition, students face difficulties in scheduling their work, especially if team members become unavailable, or are simply uncooperative.

    The requirements presented to each team are flexible enough for the student to incorporate their own creativity. Rather than specifying the requirements down to the procedure-call level (as was the case with the Airline Reservation System), students are encouraged to use whatever methods or tools they believe are appropriate for the job. They are also permitted to make suggestions to the client on how the requirements could be improved, or which extra features might be useful.

    The students are now encouraged to adopt positive working habits, such as keeping records, identifying risks, and supporting their fellow team members. In the past, each team would struggle through the project without these habits, often detracting from the working environment.

    The workload of the course has been noticeably reduced, given that less emphasis has been placed on the grading system. Also, towards the end of the course students were strongly discouraged from overworking themselves. They were explicitly asked to finalize their project as soon as possible, rather than adding new features. Interestingly enough, there were still a few students who insist on overworking themselves in order to complete the project.

    There appears to be fewer in-team disputes than in previous years, since the grading system rewards individuals rather than teams. Because of this, even if one team member fails to complete their assigned task, the remainder of the team simply needs to work around the missing member. They will not lose marks because of it.

    Because of the new format of the first term (with lectures, tutorials and assignments), students were already familiar with the tools (such as STL and Tcl/Tk), and were not required to spend extra time becoming familiar with these topics before embarking on the project. However, those team members who enjoyed learning about new tools had the opportunity to do so, and often their team would benefit.

 

7. Future Plans

Due to the fact that this year's student course evaluations have not yet been returned to the instructor, it is too soon to fully judge how successful the course has been. However, informal discussions with students have indicated that it has been enjoyable and beneficial to most of them. However, the following changes will probably be made for future years:

    The workload of the course still needs to be reduced slightly. One possibility is to reduce the course to one term, although this requires that the material covered during the first term be taught in another course. Alternatively, more time should be spent in encouraging the students to scale-down their product, especially if it won't affect their final grade.

    The weighting of the project (only 20%) was designed to reduce a student's desire to spend too much time on the project. However, if this year's student course evaluations clearly indicate that a higher weighting (such as 50%) would be preferred, this change could be easily made.

    The method used to allocate ranks can be improved, since students need to be promoted in order to feel that their work has been worthwhile. In the current system, only about 50% of the class were promoted to a higher rank, and most of them were only promoted once. In the future, it would be desirable to have a large number of ranks (maybe 7 or 8), and start everybody at the rank of 2. Therefore, more promotions will be needed, and students will feel they are being better rewarded. It is interesting to note that the final course grades will not be any different under this system.

    Now that client based projects have been proven successful, we intend to solicit project ideas from off-campus sources, such as local computer companies. Although both students and employers will benefit from the relationship, more care must be taken when dealing with non-academic clients. For example, particular attention must be put into the quality of their proposal and of how seriously the students view the requirements. No money will change hands under this system.

    To ensure that the quality of the course is maintained from year to year, a guidebook for instructors and teaching assistants should be written. This guidebook will act as a summary of the procedures for running the course on a daily basis, as well as describing the important experiences that students should be exposed to.

 

8. Summary

CPSC319 has successfully been modified to enhance the learning experience, and to better motivate students. The introduction of client-based projects has added realism to the course, and allows students to demonstrate their creativity. Students are now encouraged to follow well-defined project management principles, rather than struggling to complete their project without sufficient guidance. Finally, a simplified grading system allows students to get immediate feedback on how to improve their working habits.