Unit 6 : Assembling, Administering and Appraising Achievement Tests.
ASSEMBLING, ADMINISTERING AND APPRAISING ACHIEVEMENT TESTS
Guidelines for Assembling Achievement Tests
- Review test items and assessment tasks. Where possible, fellow teachers or colleagues can review the test items or tasks. The following points should be considered.
- Test format should be appropriate for the learning outcome being measured.
- Knowledge, understanding or thinking skill required by the item or task should match the specific learning outcome and subject-matter content being measured.
- The item or task should not be excessively wordy.
- The point of the item or task as well as the desired response should be clear.
- A scoring rubric or scoring guide should be available.
- The item or task should be free from technical errors and irrelevant clues.
- The item or task should be free from racial, ethnic and gender bias.
- Test items should be typed or written neatly. Writing items on the chalkboard or dictating them must be done with utmost care since it may cause problems for students especially those with visual, listening comprehension or hearing problems.
- Arranging test items.
- Items should be sequenced (especially objective-type tests) such that they appear in the order of difficulty with the easiest ones placed first.
- Items should also be arranged in sections by item-type. The sections should progress from easier formats to more difficult formats. Within each section, group items such that the easier ones come first. For example, all true-false items should be grouped together, then all matching items and so on.
- Items can also be arranged according to the order in which they are taught in class or the order in which the content appeared in the textbook.
- Sequencing is not necessary for essay-type tests where optional choices are made. All items of this nature should however equal difficulty levels.
- Provide directions to students. Directions should include the amount of time allowed to complete the test, where and how answers should be written, number of points for each test item, what should be done about guessing (on selection-type items). Each item format should have a specific set of directions.
- Reproducing the test.
- Items must be spaced and arranged so that they can be read and scored (for objective-type tests) with the least amount of difficulty. Cramming too many tests on to a page is poor economy.
- Multiple-choice items should have the alternatives listed vertically below the stem of the item rather than across the page.
- Items should not be split with parts of the item on two different pages. All items should be numbered consecutively.
- All illustrative material should be clear, legible and accurate.
- Proofread the entire test or assessment before it is finally reproduced.
Guidelines in Administering Achievement Tests
- Prepare students for the test. The following information is essential to students’ maximum performance.
- When the test will be given (date and time).
- Under what conditions it will be given (timed or take-home, number of items, open book or closed book, place of test).
- The content areas it will cover (study questions or a list of learning targets).
- Emphasis or weighting of content areas (value in points).
- The kinds of items on the test (objective-types or essay-type tests).
- How the assessment will be scored and graded.
- The importance of the results of the test.
2. Students must be made aware of the rules and regulations covering the conduct of the test. Penalties for malpractice such as cheating should be clearly spelt out and clearly adhered to.
- Avoid giving tests immediately before or after a long vacation, holidays or other important events where all students are actively involved physically or psychologically/emotionally.
- Avoid giving tests when students would normally be doing something pleasant e.g. having lunch etc.
- The sitting arrangement must allow enough space so that pupils will not copy each others work.
- Adequate ventilation and lighting is expected in the testing room.
- Provision must be made for extra answer sheets and writing materials.
- Pupils should start the test promptly and stop on time.
- Announcements must be made about the time at regular intervals. Time left for the completion of the test should be written on the board where practicable.
- Invigilators are expected to stand a point where they could view all students. They should once a while move among the pupils to check on malpractices. Such movements should not disturb the pupils. He/she must be vigilant. Reading novels, newspapers, grading papers are not allowed.
- Threatening behaviours should be avoided by the invigilators. Speeches like ‘If
you don’t write fast, you will fail’ are threatening. Pupils should be made to feel at ease.
- The testing environment should be free from distractions. Noise should be kept to a very low level if it cannot be eliminated or removed. Interruptions within and outside the classroom should be reduced. It is helpful to hang a “Do not DISTURB – TESTING IN PROGRESS” sign at the door.
- Test anxiety should be minimized. .
- Things that create excessive anxiety are (1) warning students to do their best ‘because the test is important’, (2) telling students that they must work fast in order to finish on time, (3) threatening dire consequences if they fail, and (4) threatening students with tests if they do not behave.
- Teachers and invigilators should not walk around looking over students’ shoulders while they are responding to assessments.
- Before assessments, teachers should convey a sense of confidence about student’s performance in the upcoming assessment.
- Do not talk unnecessarily before letting students start working. Remarks should be kept to a mi nimum and related to the test.
- Avoid giving hints to students who ask about individual items. Where an item is ambiguous, it should be clarified for the entire group.
- Expect and prepare for emergencies. Emergencies might include shortages of answer booklets, question papers, power outages, illness etc.
Appraising Achievement Tests (Item Analysis)
Item analysis is the process of collecting, summarizing, and using information from students’ responses to make decisions about each test item. It is designed to answer the following questions:
- Did the item function as intended?
- Were the test items of appropriate difficulty?
- Were the test items free of irrelevant clues and other defects?
- Was each of the distracters effective (in multiple-choice items)?
Benefits of item analysis
- It helps to determine whether an item functions as intended. It provides information on whether an item assesses the intended learning targets, whether it is of the appropriate level of difficulty or whether it distinguishes between high achievers and low achievers and whether the options are working.
- Item analysis data provide a basis for efficient class discussion of the test results. Difficult items can be identified and discussed. Misinformation and misunderstanding of distracters can be corrected.
- Item analysis provides feedback to the teacher about pupil difficulties. It brings to light general areas of weakness that require more attention.
- Item analysis data provide a basis for the general improvement of classroom instruction. It assists in evaluating the appropriateness of the learning outcomes and course content of the particular students being taught.
- Item analysis procedures provide a basis for increased skill in test construction. Item analysis reveals ambiguities, clues, ineffective distracters and other technical defects that were missed during the test preparation. Information revealed provides experience for future writing of tests.
- It helps to create item banks for use in future tests.
Steps in doing item analysis of objective tests
- Arrange the marked test papers from the highest score to the lowest score.
- Create three groups – upper, middle and lower groups using the top 27% and the bottom 27% if the total number of students is more than 40. Where the number of students is between 20 and 40, select the top 10 students and the bottom 10 students. For fewer than 20 students, create only two groups.
- For each item summarize the number of students in each of the upper and lower groups who selected each option.
- Calculate the difficulty index , i.e the percentage of the total number of students who got the item correct. The difficult index by convention is written as a decimal.
- Compute the discrimination index, i.e. the difference between the percentage of students in the upper and lower groups who got the item correct. The discrimination index is often written as a decimal fraction.
- Evaluate the effectiveness of options for multiple-choice tests (distracter analysis).
- Every distracter should have at least one lower group student choosing it, and more lower group students than upper group students should choose it.
- Every correct option should be selected by more students in the upper group.
- Options are ambiguous if upper group students are unable to distinguish between the correct response and one or more of the distractors.
- If a large number of upper group students select a particular wrong response, check to be sure the answer key is correct.
Options Upper Group Lower Group
A 0 2
B 2 4
C* 15 5
D 3 9
- Ambiguous alternative
Options Upper Group Lower Group
A 1 4
B* 10 5
C 9 5
D 0 6
Options B and C seem equally attracted to the high achiever. Option C should be checked as well as the test item for ambiguities.
- Miskeyed Item
Options Upper Group Lower Group
A 13 7
B 6 6
C 0 3
D* 1 4
Majority of the upper group selected A. Option A might be the correct response and not D.
- Poor distracter
Options Upper Group Lower Group
A 2 6
B* 12 6
C 0 0
D 6 8
Option C attracted no student. It is a poor distracter and has to be replaced.
Using Difficulty and Discrimination Indices
Points to note
- A low index of discriminating power does not necessarily indicate a defective item. They could be examined, however, for the possible presence of ambiguity, clues, and other technical effects especially if they are selection-type items.
- Negatively discriminating items should be avoided and not used in test construction.
- Discrimination indexes of 0.30 and above are more desirable in test construction. However, items with high, positive discrimination indices are used mostly by test developers on standardized tests.
- It is sometimes necessary to retain items with low discriminating power in order to measure a representative sample of learning outcomes and course content.
- Items with a 50% level of difficulty make maximum discrimination possible.
In norm-referenced testing, difficulty indices of between 0.16 and 0.84 are used to select items where the test represents a single ability. If performance on the test represents several different abilities, difficulty indices between 0.40 and 0.60 are used.