If you are designing a multiple-choice or true-false quiz to assess students' knowledge and mastery of some material, there is a chance that part of what you are assessing is not knowledge of the content but rather how good students are at taking tests (test-wiseness). Basically, students can guess the correct answer or narrow down the options based on things such as wording and grammar (secondary cues) rather than knowledge of the topic being assessed.
For example, in answering the nonsensical multiple-choice question below, you can infer the correct answer by matching the end of the question stem ("an") with the beginning of the choices. The correct answer must be the one that starts with a vowel:
A zieb always comes after an: a) burb b) kond c) arph d) rhoo
- Categorical Exclusive
- Distracters contain words such as "all", "always", or "every." If a choice contains the word "all" or or "every", a student may more easily rule it out. Try not to use those terms in the question or in the choices.
- The correct answer contains a key sound, word, or phrase that is contained in the question’s stem.
- Absurd/Implausible Answer
- Distracters are unrelated to the stem. Some incorrect choices may be more obviously wrong and easier to eliminate.
- Correct answer is more precise, clear, or qualified than the distracters. It is very detailed and provides no room for ambiguity. That suggests it is the correct answer.
- Correct answer is longer than the distracters. Try to make each of the choices relatively equal in length.
- Distracters do not match the verb tense of the stem, or there is not a match between articles ("a", "an", "the").
- Correct answer is given away by another item in the test. Sometimes the answer to question 2 can be found in the stem of question 1, for example.
- Order of Answer
- Don't use a predictable pattern for the answers (for example, making the correct answer usually be C).
- Number of Options and Guessing
- Out of a hundred multiple-choice items with three answer options each, a student with no knowledge of the content will get 33 correct simply by guessing blindly. However, the more choices you add, the more susceptible the question becomes to test-wiseness, hence four items is most common.
- Odd Man Out
- If you make two or more choices difficult to distinguish from one another conceptually, that usually helps you rule them out. If three answers are very similar and one is very different, the different one may likely be the correct answer.
- Another one mentioned is that test authors are more likely to make spelling errors on incorrect choices, since they mainly review the correct answers.
- Robert Runt has a webpage with tips for students on taking tests. At the bottom of the page is a sample multiple-choice quiz illustrating many of the above principles.
- Gibb's 1964 dissertation contains a 70 item Experimental Test of Testwiseness.
- Mahamed et al. created a 20 item test-wiseness questionnaire for pharmaceutical students.
Thorndike (1951) first conceptualized "test wiseness" as a variable that can affect test scores. Thorndike viewed test wiseness as a general and lasting characteristic of the individual and a prominent source of score variance on cognitive tests second in its influence only to individual cognitive ability. More specifically, Millman, Bishop, and Ebel ( 1965 , p. 707) defined test wiseness as the "subject's capacity to utilize the characteristics and formats of the test and/or the test taking situation to receive a high score. Test wiseness is logically independent of the examinee's knowledge of the subject matter for which the items are supposedly measures."
- Gibb B.G. (1964). Test-wiseness as secondary cue response. Doctoral Dissertation, Stanford University.
- Allen, Kirk; Stone, A., Rhoads, T.R., & Murphy, T.J. (2004). "The Statistics Concepts Inventory: Developing a Valid and Reliable Instrument" (PDF). Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition. Retrieved on 2007-12-01.
- Runt, Robert. Tips for Taking Multiple Choice Tests. Retrieved on 2007-12-01.
- Mahamed, Anisah; Gregory, P.A.M., & Austin, Z. (December 2006). ""Testwiseness" Among International Pharmacy Graduates and Canadian Senior Pharmacy Students". American Journal of Pharmaceutical Education 70 (6): 131. Retrieved on 2007-12-01.
- Kuntz, Patricia (March 1982). "Test-Wiseness Cues in the Options of Mathematics Items". Annual Meeting of the American Educational Research Association (66th, New York, NY). Retrieved on 2007-12-01.
- Thorndike, R.L. (1951) Reliability. In Lindquist, E.F. (ed.), Educational Measurement. ACE, Washington DC, pp. 560–620.
- Millman J., Bishop, H., & Ebel, R. An analysis of test-wiseness. Educational and Psychological Measurement, 1965, 25, 707-726.
- Borrello, Gloria M.; Thompson, B. (1985). "Correlates of Selected Test-Wiseness Skills". Journal of Experimental Education 53. Retrieved on 2007-12-01.