The usual long, hot summer. In Townville, Illinois, manager Joseph D Doe of the Buy-It-Here supermarket chain is getting hot under the collar. For the sixth time in three days he is listening to another complaint from one of his branch managers about ‘poor freezer maintenance service’. There is a pattern in the complaints which come from different areas in the state.
A freezer cabinet or cool room malfunctions. A service engineer arrives, looks, works on the system and gets it going again. Some four hours later the system is again out of operation. Withdrawal of products from freezers and cool rooms in which the temperature has risen above safe limits is beginning to bite into branch profits. Joseph D Doe reaches for the phone…
Incidents like this once prompted a supplier of refrigerants to create a course on the Systematic troubleshooting of refrigeration systems (E I du Pont de Nemours, 1964). The course was constructed around an algorithm (decision-making tree) via which each subsystem in a refrigeration unit — the condenser, evaporator, thermal expansion valve and compressor electrical system — was subjected to a systematic troubleshooting regime. Through this procedure the stop-start-stop problems which were known to plague users of refrigeration and air conditioning installations were minimized.
Imagine you are Joseph D Doe. You have chosen to send your 15 service engineers (in groups of five) on a course which makes use of the procedure referred to above. The course lasts two days. It covers both theory and practice. The post-test consists of a performance test in which course participants have to demonstrate their skill in diagnosing and correcting three malfunctions, using the systematic troubleshooting decision tree taught in the course. Critical theoretical knowledge relating to the procedure is tested with the help of written responses to multiple-choice questions, and an oral test given by an instructor not involved in the teaching of this particular course. As
Figure 7 Results from the Buy-It-Here trouble shooting course
Joseph D Doe you want some guarantee of the efficiency of the course. A pre-test of the service engineers’ individual knowledge and skill is requested. The results of the pre-test and post-test for the three groups of engineers are given in Fig. 17. What do you think of these results? Would you be satisfied with them? Under what conditions would you trust the validity of both tests? Would you recommend this course to other supermarket chains on the basis of the engineer’s end results? Engineers scoring 50 points or more on the combined practice=and-theory tests will have satisfied the course makers’ criterion for a participant’s success.
Case study no.6 generates a lot of questions as well as a lot of answers when it is given in the Think Tank workshop. Discussions, questions and answers tend to run along the following lines. As Joseph D Doe, most Think Tank workshop participants are satisfied — at first glance — with the end results. One hundred per cent of the Buy-It-Here engineers have scored 70 points or more on the final test: 20% the perfect score, 46% 90 points and 34% 70 points. With one exception (an engineer in group B) there is no indication in the results of the pre-test that existing knowledge of and skill in the systematic troubleshooting procedure which the course set out to teach was contributing to these end results.
A condition for trusting the validity of the apparent high success is that both pre- and post-test were so comprehensive that everything the course intended to teach was tested for. Joseph D Doe would need assurance of this from some test specialist (with appropriate technical knowledge of the troubleshooting system being taught) from outside the course.
Twenty per cent of the Buy-It-Here service engineers have a perfect score. What should one think about that? One should be suspicious of such results until there is the assurance (as hinted at above) that the test is comprehensive and therefore valid. But one should not be surprised once the result is known to be a valid one. Short, practical courses which have been developmentally tested (with the help of the jury of eight, for example) before installation should be expected to deliver top end performance results. Joseph D Doe should have no hesitation in recommending the course when the appropriate test conditions exist.
What Joseph D Doe should be interested in is the apparent difference in performance between group B and groups A and C.
The service engineer with the high pre-test score would certainly be worth talking to. What sort of things might account for this group difference? ‘Installation problems’ and ‘motivation problems’ are both brought up as possible causes in our discussions on this point in the Think Tank workshop. Such might well be the problem. Motivation and lack of motivation (possibly on the part of the engineer with the high pre-test score) are both infectious! Especially so in a course with a sharp end performance goal.
The long, hot summer continues. Joseph D Doe relaxes. From time to time he talks with his chief service engineer about the results of the troubleshooting course. As he listens to the most recent report, a contented smile comes over his face. At that moment the telephone rings.
A complaining voice crackles in his left ear…
Everything that shines can’t always shine like gold. Even after an effective and efficient training programme you still have to keep an eye on things!
[By the way, what had the designer of the troubleshooting course used as the REO — the system of stimuli which generated the responses needed for learning? If you have some knowledge of condensers, evaporators, thermal expansion valves and compressor electrical things, what do you think the designer put in the databank (referent 2) when thinking up the plan, structure and strategy of instruction for this course? If you have some technical knowledge, try to fill a referent 2 for yourself.]
In case study no. 6 a performance test, multiple-choice questions and oral questions were mentioned as being used to test the knowledge and skill of the 15 engineers. Essay tests, short answer tests, anecdotal records, rating scales, true—false items and matching items are also used to test what an individual has learned or not learned as a result of the design that has been used. So too are such activities as presentations; solving, carrying out an experiment successfully, writing a poem, making a video product — any activity, in fact, which is appropriate for indicating the individual student’s achievement of the intended learning goal.
In practice, it is not usually possible to make use of a pre-test. You generally have to work only with the results of a post-test, especially when the number of students is large and it is the teacher who has to prepare , administer and grade the pre-and post-test results. Instead, you must rely upon the accuracy of your ‘Go’ decision, needs analysis and needs specification. You must work on the assumption that the students ‘entering profile’ is not contributing significance to the end result. If it does, they don’t need to be on the course.
In addition to knowing the didactical effect of your design, you also need to know how each student experienced the design.