Repair Kit Fix 10: quality assessments

Fix 10: Don’t rely on evidence gathered using assessments that fail to meet standards of quality, rely only on quality assessments

This is a fix I’m particularly interested in reading because I’m not sure I write quality assessments and I want to know what the author suggests the identifiers are of a quality assessment.

“To be quality an assessment must be accurate.  Accurate assessment requires attention to three questions:

1) Why are we assessing?

2) What are we assessing?

3) How will we assess it?” (page 82)

Four criteria/features of test design quality are listed in the chapter.

In short:

1) Match question style to context, make sure questions will gather the evidence needed to show student achievement.

2) Tests must have good questions (specifically multiple choice) not bad ones.

This particular criteria could stand to be more specific.  I attended at least two different workshops where I learned to write quality  multiple choice, well at least for DOK (Depth of Knowledge) level 2.  I never could find anyone who could explain to me how to write a good DOK 3 level question.  The best answer I ever got was “Write a DOK level 3 constructed response then write answers to turn it into a multiple choice/selected response.”  The problem was every time I wrote the answers it seemed to become a DOK 2.  Without further help I think many teachers could struggle with writing a good question.  It’s been a long time since I’ve been in college but I don’t ever remember being taught how to write assessment questions, most teachers just used the tests that came with the textbooks.  That being said I made it a habit to review the tests I had given (and yes by that time I was writing my own using various sources) looking for questions missed by a high percentage of students to eliminate bad questions.  When I came across those questions I discounted those points from the exam so as not to penalize students.  If a student had happened to answer correctly I let them keep the points as extra credit.

3) Gather enough evidence to make valid judgements of proficiency. “We know we have enough evidence when we can confidently say that, if we gathered one more item it would simply confirm what we know now.”

Ok so is this done all on one test or through number of exams?  Does this depend on the student?  I need more information here or maybe I just need to put this fix into practice and see it work for myself, that might get me the answer I’m looking for.

4) Avoid bias that can distort results.  There can be problems with the student, the assessment setting, the scoring process, or the assessment itself can cause the score to misrepresent student achievement.

I’d say to the best of your ability because there are things you can’t control like students coming to school sick or without eating breakfast. I guess though this is where chances for retakes come in and the part from Fix 3 about sufficient evidence.

Teacher Vignette:

“My tests are composed of 10-12 high quality multiple choice questions that follow all conventions of multiple choice items designed to pinpoint where the errors are in student understanding.”

Ok this is all well and good but if teachers don’t know/aren’t taught those conventions how do they write quality multiple choice questions.  It might have been nice to get a reference to a good place to learn about that.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s