Asking the questions

Making a test can be harder than taking it.

In a recent post, we asserted that learning on one’s own was difficult in part because there are fewer ways to tell whether you really understand the material.  For instance, you don’t have the weekly quizzes and midterm tests of a classroom course.  (There are often problems and self-tests in whatever book or video course you choose, but no one to pressure you for a good grade.)  We assumed that the tests were an accurate way to check your understanding.  But that’s not always the case.  It is much harder to make up a good test than it appears.

We assume that the teacher has already worked out the purpose of the test, and that it’s to measure your understanding rather than (say) to separate the top 1% of students from the rest.  Also, we assume that the standards for each grade (for instance, A: student has assimilated all material and can apply it to new situations; B: student has assimilated all material, etc.) have been set out.  We are at the point of choosing or writing the problems.  It’s here that the subtle difficulties come in.

Our physics tutor, who has taught the subject in the classroom, has dealt with many problems using the venerable block on the inclined plane.  The object of such a problem is to test whether the student understands how to work with forces and accelerations in two dimensions.  But there is a very limited number of variations possible on this theme.  However imaginative the teacher becomes, there are not many different things one can do with a block and a plane.  Hence many students work a few examples in practice, and thereby develop patterns to follow for all later problems.  They can be very successful in getting good grades, but the test only measures how well they are at pattern-matching, not any skill with Newtonian physics.  They may not themselves be aware of the difference.

Mathematics becomes increasingly important in higher-level Physics courses.  Students in the major are often introduced to a mathematical technique only shortly before its application to a problem, or even as part of a physics problem.  It is common for students to lose track of the physics because they’re trying to work out how the teacher got from one step to another in the mathematics.  A problem-oriented test at this level may turn out to measure the students’ math ability rather than any physical understanding or skill.

In other subjects similar things happen.  What is intended to be a test on grammar in a foreign language may hinge on the students’ vocabulary; no doubt this also happens in English and literature courses.  And our tutor has many stories of students who are very competent in mathematics but, because their English is not quite that of a native speaker, are often stumped by word problems.

So the conscientious teacher (the overwhelming majority of them) is faced with a difficult task in constructing a test.  It should not contain difficulties foreign to the purpose, like mathematics or vocabulary in physics problems; but should require the knowledge and skills one is looking for, rather than things like pattern-matching.  Our consultants are sometimes amazed that it can be done at all.