Posts filed under ‘Soapbox’

US Supreme Court ruled that results of a valid, job-related test cannot be thrown out

The US Supreme Court recently ruled that white and Latino firefighters in New Haven, Connecticut were discriminated against when the city failed to certify the results of an exam.  The full text of the ruling can be found at: 

The city followed well established guidelines and best practices for test design but became nervous when one racial group was disproportionally impacted. This resulted in a heated public debate and a lawsuit that’s travelled a rocky road all the way to the Supreme Court.

I am not a lawyer and do not wish to discuss the merits of this particular case. Dramatically smarter people have gotten involved and the Supreme Court opinion is well worth a read.

I do believe that here are some lessons that we can learn from this case:

  • Valid and reliable exams assist with promoting a meritocracy.  This is very important when life, limb and livelihoods are on the line. The union’s guidelines and city processes promoted a meritocracy.
  • Job analysis to ensure that the test is aligned with the job is important and was effectively conducted in this case. As the court stated “There is no genuine dispute that the examinations were job-related and consistent with business necessity.”
  • Establishing processes and following them is important. Processes had been established and followed by the city except that when there was a dispute the city broke the process by not certifying the test results.

The city’s failure to certify the results, despite effective job analysis and an effective exam, caused this protracted legal process. Unions often get a bad rep, but in this case–as firefighters have to trust each other when their lives are on the line–they were the ones fighting for a meritocracy and approved the examination processes. Hats off to them for that.

The “fairness” or “bias” of high-stakes tests are common themes when results are challenged/appealed. So validity, reliability and cut scores require meticulous attention. Although this ruling raises the bar for challenges based on racial bias, it does not eliminate the need for employers to be vigilant about test quality. Questionmark’s white paper on defensibility addresses  test reliability, validity and other essentials.  It’s available for download at:

I’m happy that the US Supreme Court has ruled in favor of exams and demonstrated their appreciation for best practices in the areas of job analysis, unbiased development, expert review, peer review, and agreeing, documenting and following processes.


June 30, 2009 at 8:15 pm Leave a comment

Conversations On My Journey

As I have travelled for the last few weeks I have been asked, in many different ways, how to create valid and reliable assessments.  I refer people to great authors and my mentors but I also take time to explain some of the basics to help folks get started. 

Some of the ‘basics’ quickly resonate with people and they smile and appreciate the direction; at other times I am greeted with blank stares!  I have a guiding principle that I don’t really understand something until I can explain it and the other person can explain it back to me.  Only then do I know that I “get it”  My many conversations around the world have given me experience of when a person “gets it” and when they don’t, regardless of location or cultural background. 

Here are some things that I now “get.” I’ll be happy to share more details as I develop future articles:

  1. Our assessments are not the assessments of our grandfathers. With the aid of high speed, high fidelity, and sophisticated technologies we can deliver and analyze assessments results to yield positive benefits for individuals, groups and society.  We can do so much more with less.
  2. Not all assessments are used for the same purposes.  Most assessments are designed to measure knowledge, skills, attitudes and abilities; some have byproducts of improving memory recall and/or changing opinions and attitudes.
  3. Blooms Taxonomy helps people understand distinctions between knowledge, comprehension, and higher levels of thinking.  
  4. It’s now cool to possess knowledge, skills, abilities and healthy attitudes (KSAAs). In the past some people would get on by turning up but not really contributing.  With the recession real contributions are now valued. Unfortunately harvesting KSAA (i.e. studying) is not so cool in cultures of instant gratification, however, our flatter world will surely cause that to change too.
  5. It’s now easy to crowd source learning content. So many people want to contribute their knowledge and experience to make the world a better place and now we have the tools (wikis, blogs, and social networking) to enable this.
  6. Enable learning environments and motivated learners will learn. Edutainment (to educate using entertaining movies, sound and fun interactions) has its place for non-motivated learners. However, when people are motivated to learn, wikis, blogs, social networking, and search can be less expensive and more important than course based instruction.
  7. Seek and ye shall find.  We are entering an era when people who gain clarity on what they want to learn and take responsible for their learning will prosper more than those who don’t. 
  8. Knowing how to accept feedback is a valuable skill. As an imperfect individual I receive lots of feedback and sometimes this is packaged up in a way to help me and sometimes it is ugly. It appears to me that knowing how to find the “diamonds in the rough” is a valuable skill.

And yes, feel free to comment and provide feedback!

May 18, 2009 at 12:59 pm Leave a comment

How is the Economy viewed through the Lens of Learning and Assessments Professionals

There are professionals that understand the economy better than I do and so don’t be a fool and take anything that I say or blog about to be advice!  I’m happy to be able to pass along my current knowledge, understanding and interpretations of the marketplace and I hope that my point of view helps you. 

Here is an interview hosted, directed, and produced by  Dr. Will Thalheimer of Work-Learning Research in March 2009 entitled “How the Bad Economy is affecting Learning Professionals as viewed through the lens of Learning Assessments”.  I hope that you find some value here:

Dr. Will Thalheimer is a great guy and one of the world leading experts and meta researchers in the field of memory recall. 

April 15, 2009 at 12:12 pm Leave a comment

What Does “Open” Mean?

After talking with my “open” mentors Steve Lay and Tom King, and then discussions with our customers I have started to think about how we might define “open” in the context of assessment and technology. In this post I have collated the intellectual effort of others, that I have learned from over the years, and more recently from Tom and Steve.  I’m posting to see how on or off target I am/we are so please feel free to comment.

Open Agenda

An Open Agenda is the opposite of a hidden agenda!  An Open Agenda allows people to know where you are going and why you are going there. 

Open Process

With an open process people understand where you have been, where you are how you determine direction.

Open To New Ideas

A closed dogmatic approach to problems solving tends not to yield solutions that are maintainable, or long lasting, and as such inhibits the creative process. 

Low Barriers …to Witness

This requires that the information, such as the agenda, mission, process, and deliverables, are easy to discover, see, and use for your own benefit.

In the world of technology this normally means a quick, simple and free registration process, requiring no more than an email address, and making overview and technical documentation available as well as source code.

Low Barriers …to Participate

If you wish to participate there might be small fees or qualification requirements (such as being a citizen of a country to vote), but in the context of your participation the barriers are low. In the world of open participation you might share your ideas and be involved in discussions and priority setting.

Low Barriers …to Contribute

It may be that you contribute to the process but without transparency regarding how your contributions are evaluated, prioritized and/or acted upon. In the world of technology this normally means allowing source code to be available for others to contribute to on a known licensing model. Typically this means that the author contributes their work to the community at large.


People have an insight into each step of the process, which provides information on how decisions are reached, deliverables created and buy-in and/or consensus was achieved. Having a transparent process typically accelerates the use of open standards and best practices as different communities can share ideas throughout the process.

Some processes do not benefit from being transparent and others do. For instance, I can hardly imagine that negotiating the end of a war would benefit from having the discussions televised on CNN live!  But in contrast I enjoy watching and knowing that my City Government is open and televised; it’s the best sleep aid I know!

The “Tools” of Openness

Here are some tools that can help organizations be open:

  • Wiki – a tool that allows us to document a body of work in an cooperative and collaborative way
  • Forums – normally threaded discussions that can be accessed by web browsers and email clients, allowing us to build consensus by discussing issues
  • Issue tracker – a tool that allows us to examine and participate in the process of prioritizing and resolving issues.
  • Source code repository – a place where source code is deposited and shared that allows us to examine the output of the development process along with a detailed audit trail of who contributed to it and when.
  • Blogs and micro-blogs – where ideas can be easily shared and responses gathered


Being “Open” is multi-facetted which I have tried to capture above.  But I’m open to being wrong and learning more.

April 10, 2009 at 9:39 am Leave a comment

As Learning and Assessment Professionals What Could We Have Done to Prevent the Financial Crisis

As I prepare my various presentations for our Questionmark Users Conferences in Memphis, Tennessee in the context of an economic downturn but a thriving assessment market I can’t help thinking about how we as learning and assessment professionals could have done more to prevent the economic crisis.

A radical thought but take this journey with me.  The current crisis stems from:

  • Unethical behaviors in the mortgage sales process
    • Where were our regulators?  Did they know what they were meant to be doing? Were they assessed?
    • Were these mortgage brokers tested on ethics?
    • Was the chain of command trained on how to spot and prevent fraud and deceptive practices?
    • Do our certification methodologies need to move into the 21st century? We have so many tools to raise people’s levels of consciousness and thereby their accountability. But are we using them as effectively as we could?
  • Lack of Consumer Education
    • Businesses are out to make money. Credit helps us borrow against our future in order to have something now but borrowing excessively causes stress while making others wealthy. How many borrowers fail to understand this and why?
    • I wish I had learned fiscal responsibility earlier on in my life. Did I miss that class at school or college or did someone forget to put that into the curriculum?
    • Should/could there be elearning courses and assessments that could be part of the lending process to ensure that consumers are competent to take on their responsibilities?
  • Lack of understanding of new (unstable) financial instruments (e.g. Credit Default Swaps)
    • It seems to me that the financial instruments, or weapons of mass financial destruction, were poorly understood.  We don’t let children play with fireworks for very good reasons!  Many good people’s financial future has been wrecked by financial expert’s incompetence.
    • Just as we don’t let our pilots fly complex machines without educating them and testing them I don’t think that we should have amateurs pretending that they understand these complex instruments.
    • Is there any process to confirm that complex financial instruments are understood by the people who construct and trade them? Are they competent?
    • As assessment professionals we know that when someone says “Trust me I know” it often means that they don’t!
    • Should there have been more effective learning around these financial instruments?
    • Should they have tested to see if they had sufficient competence to play the high-risk financial games that they were playing?
  • C level executives exposing their organizations to unacceptable levels of risk
    • Some organizations were so focused on training and certifying new employees being on-boarded to handle their growing businesses that they forgot about educating and assessing middle managers and above.
    • I’m CEO of Questionmark.  I go on training courses. I do tests. I have to take our internal tests.
    • Should we not have batteries of test instruments to confirm that our C levels have the business understanding and ethics to run the organizations (banks, etc) that we as a society depend on?

Okay, so you now get what I’m talking about and you get where I am going!

So Learning and Assessment professionals stand up and be counted!  You have knowledge, skills, abilities and attitudes that can help us learn from recent experiences and help prepare us for a more ethical and stable future.  Your organization employs you for your expertise and so I ask that you use your knowledge and skills to help and benefit us all.

Anyone that knows me gets that I am passionate about assessments and I love learning. I especially love learning from our customers.  At this year’s User Conference I’d love to learn what we could have done to prevent the unethical/uneducated behaviors and questionable practices that led us on this journey of mass financial destruction not to agonize over what has happened but to help our profession be a part of the solution.

Lets make a difference and stand up for our profession!

March 28, 2009 at 3:37 pm 6 comments

Add to Technorati Favorites

Recent Posts