Posts tagged ‘High Stakes Assessment’

Society for Industrial and Organizational Psychology “SIOP” discussion of the US Supreme Court case Ricci v. DeStefano

I wanted to share some well informed points of view around testing in the workplace and commentary on the recent Ricci v. DeStefano case from the SIOP web site.

SIOP is a well respected Division of the APA with a mission to enhance human well-being and performance within work settings by promoting the science, practice, and teaching of industrial-organizational psychology.

Industrial-organizational (I-O) psychology is the scientific study of the workplace. Rigor and methods of psychology are applied to issues of critical relevance to business, including talent management, coaching, assessment, selection, training, organizational development, performance, and work-life balance.

Excerpt from article by Clif Boutelle and Stephany Schings:

With its recent narrow 5-4 decision in the Ricci v. DeStefano case, the Supreme Court ruled that results of a valid, job-related test cannot be thrown out simply because the test may result in adverse impact. The decision has underscored the importance of valid, job-related tests, and for many I-O psychologists, it has reaffirmed their status as central and vital players in the testing process.

Read the complete SIOP Article here

My original blog posting on the Ricci v. DeStefano

July 16, 2009 at 4:26 pm Leave a comment

Distinguishing Low, Medium and High Stakes Assessments

Trying to classify an assessments into a Low, Medium and High stakes category has some pros and cons.  On the plus side we can quick “range “ the time to create, deliver and report on an assessment, and range the impact to the person taking the assessment, and it can help conversations about an assessment.  All conversations have to be considered in context.  If you were having the conversation in the context of Psychometrics you would probably be distinguishing between low stakes and high stakes exams; if you were having the conversation with instructors and training you might be distinguishing assessments prior to a course, during a learning experience, tests after and course evaluations.

image

This chart can help us think about Low, Medium and High Stakes assessments in both context but without specific measure. Essentially it aids the conversation by promoting some distinctions and a vocabulary rather than providing a measurable outcome. That is in itself amusing because the world of assessments is all about the theory and technique of educational and psychological measurement.

In higher stakes assessments we tend to talk about candidates, in medium and lower stakes assessments we talk about students, employees and/or learners and in low stakes assessments we might talk about respondents. And so in our vocabulary, in the context of low, medium and high stakes assessments, we’ll talk about Participants who are the people that answer the questions in our assessments; they participate.

There are six terms that can help us provide some distinctions:

  1. Consequences to the Participant

    If the consequences to the participant are low then that helps us classify into a Low Stakes assessment but if the consequences are great (affecting Lives, Limbs, and/or Livelihoods) then it would be a High Stakes assessment.

  2. Legal Liabilities

    When stakes are high, consequences are high, and so in come the lawyers. I don’t want to turn this into a political debate but laws are written to protect our rights and lawyers help us understand the laws to do the right thing. All stakeholders in the assessment process have rights and responsibilities. Often this debate is taken from the side of the Participant but other Stakeholders have rights too. Assessments must be fair and reliable and fit for purpose. And we can’t go around certifying people to fix gas leaks that aren’t qualified.

  3. Proctoring/Invigilation

    When the stakes are high people are more motivated to cheat which requires that the assessment process is invigilated to prevent this and to promote trust in the assessment process.  As I fly around the world I woudl liek to know that my pilot and the air traffic controllers didn’t cheat on their exams. With certain kinds of, low stakes, assessments invigilation would provide an unwanted and undesirable level of supervision. Here’s some examples of assessments that we should probably not proctor/invigilate:

    1. Novice student taking an assessment where the goal is to provoke intrigue before a learning experience.
    2. Course evaluation where the moderator might choose to influence the outcome.
  4. Validity and Reliability

    Ideally all assessments should be reliable and valid.  They should work consistently over time and they should align with the subject matter that you are assessing. However, if we applied the same standards to low, medium and high stakes assessments we might never justify the costs for say Formative assessments or Course Evaluations.  We must always work ethically but we don’t need a $25,000 study on the validity and reliability of the assessment. When conducting a High Stakes test or exam we need to be sure that the assessment aligns with the topics that it is assessing and it is fair to all participant.

  5. Planning

    If you are administering a High Stakes test or exam to 5,000 Participants you’ll use a different plan than for 10 people evaluating a course. Planning will include considering how you’ll develop your assessment, how you’ll have expert(s) review, how will you maintain confidentiality and security, how will you deal with delivering the assessment, how will you provide accommodations for those with special needs, how will you report on the results to the Participant and Stakeholders.

  6. Psychometrician Involvement

    A Psychometrician is a Psychometrics professional familiar with the theory and techniques to measure knowledge, skills, abilities, attitudes, and personality traits.  Psychometricians are often involved with the development of High Stakes assessments such as tests and exams, and analyze the results to ensure that the assessments are valid and performing consistently and reliably.

It is worth reminding you that all assessments can be very valuable, within their context, regardless of how you categorize them.  Just because quizzes, designed to strengthen memory recall, and course evaluations, designed to measure in order to improve the learner’s environment, are low stakes does not mean that they provide low value. But those types of distinctions will follow in another blog entry!

March 22, 2009 at 1:26 pm 2 comments


Add to Technorati Favorites

Recent Posts

Categories