Posts filed under ‘Talent Management’

Assessment Maturity Model

Over the last few years I have been working with customers and friends on an Assessment Maturity Model. It started with brainstorming, it was developed using a wiki and it has been tested by representing the idea to a number of groups around the world.  Now I feel that it is ready to take to the next step.

Over the last few weeks I have been formalizing the model to make it easy to understand and building a web site http://www.assessmentmaturitymodel.org/ 

The premise behind the Assessment Maturity Model is that if you can’t measure it you can’t manage it. But to measure it you need to know what "it" is. 

image

 

The Assessment Maturity Model proposes that there are six key performance indicators, known as Measures, within the three key Areas of Assessments, namely:

  • Assessment Development
    This area includes all aspects of authoring (creating and maintaining) the items and assessments.
  • Assessment Delivery
    This area includes all aspects of administering the assessment to candidates, respondents, participants, etc.
  • Presenting Results
    This areas deals with presenting results in a trustworthy way to the stakeholder in meaningful context.

Six key performance indicators, known as Measures, are tracked to provide an indication of maturity and these are:

  • Stakeholder Satisfaction
  • Security
  • Strategic Goals
  • Processes
  • Data Management
  • Communications with Stakeholders

image By tracking these Measures an organization can determine where they are and can plan for where they want to be. These can be tracked by a single Area, as shown in the graphic to the left, or with the three Areas combined for an overview. The graphic to the left shows how an organization can track by area based on "Quality" and "Efficiency".

If you manage any type of assessment program I’d encourage you to take time to learn more about the Areas, Measures and Phases of the Assessment Maturity Model.  or just

Please feel free to link to the Assessment Maturity Model web site at http://www.assessmentmaturitymodel.org/, cross link from other web sites and blogs, twitter about it, email me about it, comment here about it and/or even tell your friends about it! 

Watch out for more – this isn’t finished yet!

Advertisements

August 14, 2009 at 12:14 am Leave a comment

Types of Assessments – Learn, Qualify, Perform

image I wanted to share some work with you on how to logically organize assessments and came up with this simple model which seems to resonate with the people that I present it to.  You might also be interested in my previous article on Types Of Assessments (Formative, Diagnostic, Summative, and Surveys)..

There are assessments for learning and assessments to qualify people for certain activities or job roles; then they perform tasks. Throughout the process of learning, qualifying and performing, we accumulate evidence and feedback that help us make decisions. 

The definitions and distinction of the various assessment types that I refer to in this article are documented within the Assessment Maturity Model at http://assessmentmaturitymodel.wikispaces.com/Assessment+Solutions:

Learn

image Assessments Through The Learning Process can be distinguished and linked together by thinking of:

  • Pre-Learning Assessments
  • In-Learning Assessments
  • Of-Learning Assessments

Pre-Learning Assessments

The following assessments are used prior to learning events:

  • Job Task Analysis
  • Needs Analysis Surveys
  • Diagnostic assessments
  • Pre Tests
  • Placement tests
  • Self-diagnostic tools
  • In-Learning Assessments
  • Of-Learning Assessments

In-Learning

The following assessments are used within learning events to assist the learner learn:

  • Formative assessments
  • Quizzes during Learning
  • 360 Learner Peer Review
  • Practice Tests

Of-Learning

The following assessments are used to detect what a learner has learned:

  • Level 1 Surveys
  • Course evaluations
  • 360 (Level 3) Surveys
  • Post Course Tests
  • Graduation Exams
  • Internal exams
  • Open exams
  • Licensing exams

Qualify

image I struggled with this for a long time on whether qualify should be its own category but finally I came to the conclusion that it should because it provides sufficient distinctions on its own and could be separated from learning experiences. 

There are three basic categories of Assessments To Qualify and these have various styles of assessments associated with this:

  • Academic: Graduation Exams
  • Pre-Employment: Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • Certification & Licensing: Graduation exams, Internal exams, Open exams, Licensing exams

Perform

imageAfter learning and qualifying we end of up performing manual and intellectual tasks. And as we do things evidence surfaces that can be referenced to help us make decisions about more learning, more qualifications and our performance.

This evidence can be used and assessed within an organizations’ Performance and Talent Management processes to help people become more successful in their work 

Performance and Talent Management

image Performance Management is a method used to influence and manage behaviors and results with the goal of bringing out the best in people.  Talent Management refers to the phases of finding, developing and retaining people to perform activities. 

Within these phases people engage in learning and qualifications and their activities tends to accumulate evidence that assists them and others assess their performance.

Talent Management assessments can be categorized with the three phases of Talent Management:

  • Talent Acquisition – finding and employing the right people
    • Job Task Analysis
    • As per Pre-Employment assessments i.e. Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • On-Boarding – orienting new people to the workplace
    • As per Assessments Through The Learning Process
  • Performance, Talent and Team Management – to help people be successful
    • As per Assessments Through The Learning Process
    • Appraisal (360s)
    • Employee Attitudes
    • Opinion Surveys

I hope you have found this posting useful in showing how the types of assessments relate to Learning, Qualifying and Performing as well as getting things done!

August 3, 2009 at 12:09 pm Leave a comment

Learn, Qualify, Perform

This original article was enhanced, after feedback from customers and colleagues in the industry, around the concept of “Performing”.  I updated this article and will repost to save confusion.

imageI wanted to share some work with you on how to logically organize assessments and came up with this simple model which seems to resonate with the people that I present it to.  You might also be interested in my previous article on Types Of Assessments (Formative, Diagnostic, Summative, and Surveys)..

There are assessments for learning and assessments to qualify people for certain activities or job roles; then they perform tasks. Throughout the process of learning, qualifying and performing, we accumulate evidence and feedback that help us make decisions. 

The definitions and distinction of the various assessment types that I refer to in this article are documented within the Assessment Maturity Model at http://assessmentmaturitymodel.wikispaces.com/Assessment+Solutions:

Learn

image Assessments Through The Learning Process can be distinguished and linked together by thinking of:

  • Pre-Learning Assessments
  • In-Learning Assessments
  • Of-Learning Assessments

Pre-Learning Assessments

The following assessments are used prior to learning events:

  • Job Task Analysis
  • Needs Analysis Surveys
  • Diagnostic assessments
  • Pre Tests
  • Placement tests
  • Self-diagnostic tools
  • In-Learning Assessments
  • Of-Learning Assessments

In-Learning

The following assessments are used within learning events to assist the learner learn:

  • Formative assessments
  • Quizzes during Learning
  • 360 Learner Peer Review
  • Practice Tests

Of-Learning

The following assessments are used to detect what a learner has learned:

  • Level 1 Surveys
  • Course evaluations
  • 360 (Level 3) Surveys
  • Post Course Tests
  • Graduation Exams
  • Internal exams
  • Open exams
  • Licensing exams

Qualify

image I struggled with this for a long time on whether qualify should be its own category but finally I came to the conclusion that it should because it provides sufficient distinctions on its own and could be separated from learning experiences. 

There are three basic categories of Assessments To Qualify and these have various styles of assessments associated with this:

  • Academic: Graduation Exams
  • Pre-Employment: Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • Certification & Licensing: Graduation exams, Internal exams, Open exams, Licensing exams

Perform

image After learning and qualifying we end of up performing manual and intellectual tasks. And as we do things evidence surfaces that can be referenced to help us make decisions about more learning, more qualifications and our performance.

This evidence can be used and assessed within an organizations’ Performance and Talent Management processes to help people become more successful in their work 

Performance and Talent Management

imagePerformance Management is a method used to influence and manage behaviors and results with the goal of bringing out the best in people.  Talent Management refers to the phases of finding, developing and retaining people to perform activities. 

Within these phases people engage in learning and qualifications and their activities tends to accumulate evidence that assists them and others assess their performance.

Talent Management assessments can be categorized with the three phases of Talent Management:

  • Talent Acquisition – finding and employing the right people
    • Job Task Analysis
    • As per Pre-Employment assessments i.e. Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • On-Boarding – orienting new people to the workplace
    • As per Assessments Through The Learning Process
  • Performance, Talent and Team Management – to help people be successful
    • As per Assessments Through The Learning Process
    • Appraisal (360s)
    • Employee Attitudes
    • Opinion Surveys

I hope you have found this posting useful in showing how the types of assessments relate to Learning, Qualifying and Performing as well as getting things done!

July 28, 2009 at 12:15 pm Leave a comment

Assessment Solutions – Vocabulary

I have recently brought some work on Assessment Solution vocabularies into the public domain and enhanced it to be useful in the context of the Assessment Maturity Model.

You can reference this work from http://assessmentmaturitymodel.wikispaces.com/Assessment+Solutions

Please feel free to comment here, on the AMM wiki or email me directly. 

Thank you for all of your comments so far!

Eric

July 28, 2009 at 11:59 am Leave a comment

US Supreme Court ruled that results of a valid, job-related test cannot be thrown out

The US Supreme Court recently ruled that white and Latino firefighters in New Haven, Connecticut were discriminated against when the city failed to certify the results of an exam.  The full text of the ruling can be found at:
    http://www.supremecourtus.gov/opinions/08pdf/07-1428.pdf 

The city followed well established guidelines and best practices for test design but became nervous when one racial group was disproportionally impacted. This resulted in a heated public debate and a lawsuit that’s travelled a rocky road all the way to the Supreme Court.

I am not a lawyer and do not wish to discuss the merits of this particular case. Dramatically smarter people have gotten involved and the Supreme Court opinion is well worth a read.

I do believe that here are some lessons that we can learn from this case:

  • Valid and reliable exams assist with promoting a meritocracy.  This is very important when life, limb and livelihoods are on the line. The union’s guidelines and city processes promoted a meritocracy.
  • Job analysis to ensure that the test is aligned with the job is important and was effectively conducted in this case. As the court stated “There is no genuine dispute that the examinations were job-related and consistent with business necessity.”
  • Establishing processes and following them is important. Processes had been established and followed by the city except that when there was a dispute the city broke the process by not certifying the test results.

The city’s failure to certify the results, despite effective job analysis and an effective exam, caused this protracted legal process. Unions often get a bad rep, but in this case–as firefighters have to trust each other when their lives are on the line–they were the ones fighting for a meritocracy and approved the examination processes. Hats off to them for that.

The “fairness” or “bias” of high-stakes tests are common themes when results are challenged/appealed. So validity, reliability and cut scores require meticulous attention. Although this ruling raises the bar for challenges based on racial bias, it does not eliminate the need for employers to be vigilant about test quality. Questionmark’s white paper on defensibility addresses  test reliability, validity and other essentials.  It’s available for download at: www.questionmark.com/whitepapers

I’m happy that the US Supreme Court has ruled in favor of exams and demonstrated their appreciation for best practices in the areas of job analysis, unbiased development, expert review, peer review, and agreeing, documenting and following processes.

June 30, 2009 at 8:15 pm Leave a comment

Fidelity of an Assessment

imageFrom the calculation shown above, you can see that a worthy goal of creating, delivering and reporting on assessments is to minimize the Error of Measurement. However, this competes with another worthy goal; ensuring that assessments are affordable.

imageThe more we can simulate the actual performance environment and the actual knowledge to be recalled, and/or the skill and/or ability to be used the smaller the error of measurement.   So if we were assessing your driving performance we should put you in a car and go driving with you.

Simple enough! Driving without crashing is a worthy goal but hardly the bar that we should set for our driving tests!  We need to ask ourselves what real performance looks like, what are the behaviors that can predict good performance, and what is the appropriate ways to measure these.

So what are the performance characteristics of a good driver?  Good eye sight, ability to control the car, spatial awareness, understanding and obedience to road signs, signaling intent to others, etc.  It might be better to assess some of these attributes before we jump in a car!  So let’s determine the criteria for successful performance and then assess each attribute (sight, road signs, rules of the road, etc) with one form of assessment before we start witnessing someone’s actual performance.

imageWe would start by determining the knowledge, skills and abilities required to perform a task in a real world situation.  Then we would use an appropriate assessment and select less expensive assessments to start with and then progress closer to real world environments.

Now the question is, how do we assess someone’s potential performance within a low cost simulated environment?  The key is to place someone into the context of the performance environment. We can do this by using scenario questions with the stimulus that we can afford/justify:

  • Low cost
    Tell a story about a situation and ask questions related to that story.  “You travelled to Morocco with your friend who rented a car and you came upon a road sign blah, blah, blah. When would you turn left?”
  • Medium cost
    Produce pictures and sounds and let the pictures and sounds tell the story. “Please follow along with the pictures and answer the question when would you turn left?”
  • High Cost
    Show a video that simulates driving. The person could either interact with the simulation or answer multiple questions about the video and or simulation as they progress through the experience.

The closer the simulation is to the real world (sounds, smell, sight, danger, etc), the more accurate the measurement.  When we simulate something to 100% we are in the real world!

In certain situations we want to measure your performance whilst your adrenaline levels are high or when your fight or flight mechanisms are kicking in. To do that we have to raise the fidelity of the simulation without causing harm.

For instance, suppose we want to learn how you would react in a crash.  We can’t go around having you crash things as that would be a danger to yourself and others. However, by using low and high fidelity simulations we could produce pretty accurate predictors on how you would act during a crash. 

imageThe  picture above tries to illustrate this idea of fidelity. The higher the fidelity of the stimulus the more accurate the measurement can be.  However, maybe we don’t need or can’t afford high fidelity. So we have some options:

  • Text stimulus

    Advantages: Easy, inexpensive, and appropriate for many situations.
    Disadvantages: Might inappropriate assess reading skills. Rarely stimulates the body to react with fight or flight mechanisms which might cause a different outcome in real life than during an assessment. 

  • Picture stimulus

    Advantages: Easy, inexpensive, images can convey a real world situations. Can focus on assessing the topic rather than also assessment language skills.
    Disadvantage: Rarely stimulates the body to react with fight or flight mechanisms.

  • Video stimulus

    Advantages: Videos can repeatedly convey real world situations, Can focus on testing on the topic rather than also assessment language skills. Can stimulate more emotional reactions.
    Disadvantages: More expensive to produce.


  • Interactive stimulus (Gaming)

     

     

     

    Advantages: Games can convey real situations. Can focus on testing on the topic rather than also assessment language skills. Can stimulate more emotional reactions.
    Disadvantages: More expensive to produce.

I hope that you find this helpful.

May 20, 2009 at 6:13 pm Leave a comment

7 Talent Management Strategies for Transformational Change – Bersin & Associates

I learned this week about the 7 Talent Management Strategies for Transformational Change as represented by Josh Bersin that I could paraphrase as follows:

  1. Talent Engagement – a business strategy
  2. Build Learning Environments – enable prosumer mentalities
  3. Deep specialization – have networked experts
  4. Focus on First Line Management 
  5. Design organization for Talent Mobility – ability to learn is a key skill
  6. “We” learning – easily harvesting wisdom from crowds in a YouTube/Wiki world
  7. Do less with less – focus on what’s important and build self-learning organizations

You’ll find more on this and other Talent Management ideas at www.bersin.com and Josh Bersin has indicated that he’ll be publishing papers on this in 2009.

April 17, 2009 at 7:37 am Leave a comment

Older Posts


Add to Technorati Favorites

Recent Posts

Categories