Fidelity of an Assessment

May 20, 2009 at 6:13 pm Leave a comment

imageFrom the calculation shown above, you can see that a worthy goal of creating, delivering and reporting on assessments is to minimize the Error of Measurement. However, this competes with another worthy goal; ensuring that assessments are affordable.

imageThe more we can simulate the actual performance environment and the actual knowledge to be recalled, and/or the skill and/or ability to be used the smaller the error of measurement.   So if we were assessing your driving performance we should put you in a car and go driving with you.

Simple enough! Driving without crashing is a worthy goal but hardly the bar that we should set for our driving tests!  We need to ask ourselves what real performance looks like, what are the behaviors that can predict good performance, and what is the appropriate ways to measure these.

So what are the performance characteristics of a good driver?  Good eye sight, ability to control the car, spatial awareness, understanding and obedience to road signs, signaling intent to others, etc.  It might be better to assess some of these attributes before we jump in a car!  So let’s determine the criteria for successful performance and then assess each attribute (sight, road signs, rules of the road, etc) with one form of assessment before we start witnessing someone’s actual performance.

imageWe would start by determining the knowledge, skills and abilities required to perform a task in a real world situation.  Then we would use an appropriate assessment and select less expensive assessments to start with and then progress closer to real world environments.

Now the question is, how do we assess someone’s potential performance within a low cost simulated environment?  The key is to place someone into the context of the performance environment. We can do this by using scenario questions with the stimulus that we can afford/justify:

  • Low cost
    Tell a story about a situation and ask questions related to that story.  “You travelled to Morocco with your friend who rented a car and you came upon a road sign blah, blah, blah. When would you turn left?”
  • Medium cost
    Produce pictures and sounds and let the pictures and sounds tell the story. “Please follow along with the pictures and answer the question when would you turn left?”
  • High Cost
    Show a video that simulates driving. The person could either interact with the simulation or answer multiple questions about the video and or simulation as they progress through the experience.

The closer the simulation is to the real world (sounds, smell, sight, danger, etc), the more accurate the measurement.  When we simulate something to 100% we are in the real world!

In certain situations we want to measure your performance whilst your adrenaline levels are high or when your fight or flight mechanisms are kicking in. To do that we have to raise the fidelity of the simulation without causing harm.

For instance, suppose we want to learn how you would react in a crash.  We can’t go around having you crash things as that would be a danger to yourself and others. However, by using low and high fidelity simulations we could produce pretty accurate predictors on how you would act during a crash. 

imageThe  picture above tries to illustrate this idea of fidelity. The higher the fidelity of the stimulus the more accurate the measurement can be.  However, maybe we don’t need or can’t afford high fidelity. So we have some options:

  • Text stimulus

    Advantages: Easy, inexpensive, and appropriate for many situations.
    Disadvantages: Might inappropriate assess reading skills. Rarely stimulates the body to react with fight or flight mechanisms which might cause a different outcome in real life than during an assessment. 

  • Picture stimulus

    Advantages: Easy, inexpensive, images can convey a real world situations. Can focus on assessing the topic rather than also assessment language skills.
    Disadvantage: Rarely stimulates the body to react with fight or flight mechanisms.

  • Video stimulus

    Advantages: Videos can repeatedly convey real world situations, Can focus on testing on the topic rather than also assessment language skills. Can stimulate more emotional reactions.
    Disadvantages: More expensive to produce.

  • Interactive stimulus (Gaming)




    Advantages: Games can convey real situations. Can focus on testing on the topic rather than also assessment language skills. Can stimulate more emotional reactions.
    Disadvantages: More expensive to produce.

I hope that you find this helpful.


Entry filed under: Assessment, Learning, Professional Life, Talent Management. Tags: , , , , , , , .

Blooms Taxonomy Questionmark Live – Story Behind the Story

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed

Add to Technorati Favorites

Recent Posts


%d bloggers like this: