Posts filed under ‘Learning’

Learning Mashups

Every day I am asked to explain mashups and their impact on the world of learning and assessments and I felt that it was about time that I shared something more widely.  In the context of learning and assessments mashups can be categorized as follows:

  1. Portlet – configured by an end user
  2. Page (Web, Wiki or Blog) – configured by a web author
  3. Data – configured by a programmer

Portlet Based Mashup

image Portlet based mashups provide different “windows” that interact with different and separate applications and users can move these portlets around their screen to meet their own personal requirements.

This is explained in the YouTube video below:

How is this applicable to learning and assessment?

As organizations standardize and align on their IT architecture there will be a framework in which to view applications in the context of someone’s job role.  The job roles for CLOs, Training Managers, Instructors, Teachers, Professors, Instructional Designers, Psychometricain, students and other stakeholders are different and have different preferences, priorities and organizational styles.  Portlet based user interfaces allows users to view their screens in the way that they find them most useful to perform their tasks.

Page Based Mashup

Page based mashups are configured by the author of the web page, wiki and or blog to determine where application data and interactions might occur.  As an author of a blog I have inserted a YouTube video above and another below that explains how to embed a video and assessment into the same wiki page:

To learn more about inserting assessments into web pages, blogs and wikis check out: http://questionmarkdemo.wikispaces.com/

Back to mashups…

This blog posting is presented as an HTML web page, with HTML text and a video feed from YouTube and a assessment feed from a Questionmark server.

image The diagram to the left shows this diagrammatically.  The main page contents are coming in from the web server, and in this case, a Questionmark assessment is coming in from another server.  The author of this web page is the person that chose how this design would be rendered.

How is this applicable to learning and assessment?

Documentation, learning content and assessments are often available throughout an organization’s infrastructure but challenging to find and they are especially difficult to find in context.  Wikis, blogs and web pages can be used to expose existing resources, wrap context, to provide a meaningful learning and/or assessment experience. Learning mashups can improve learner experiences and learning outcomes.

Data Mashups

image

A data mash up is where data is drawn from different data sources and displayed within a single page.  In this style a programmer constructs a program to retrieve the data, from different data sources, and then display it within a web page.

image

The diagram to the right illustrates the data that is mashed up to provide a presentation of the price of taxis in New York City.

This form of data mashup used to require that the application had direct access to the database.  These days, with web-services, it is possible to read and write data anywhere in the world with a single application.

How is this applicable to learning and assessment?

With more data being accessible via web-services, applications can now provide powerful reports to assist decision making to improve learning outcomes, drive qualifications when required and boost performance.

Conclusion

Standard tools and technologies supporting blogs, wikis and web pages can provide new styles of easy-to-maintain stimulating and powerful learning environments.

Did you learn something from this mashup?

September 1, 2009 at 1:40 pm 3 comments

Assessment Maturity Model

Over the last few years I have been working with customers and friends on an Assessment Maturity Model. It started with brainstorming, it was developed using a wiki and it has been tested by representing the idea to a number of groups around the world.  Now I feel that it is ready to take to the next step.

Over the last few weeks I have been formalizing the model to make it easy to understand and building a web site http://www.assessmentmaturitymodel.org/ 

The premise behind the Assessment Maturity Model is that if you can’t measure it you can’t manage it. But to measure it you need to know what "it" is. 

image

 

The Assessment Maturity Model proposes that there are six key performance indicators, known as Measures, within the three key Areas of Assessments, namely:

  • Assessment Development
    This area includes all aspects of authoring (creating and maintaining) the items and assessments.
  • Assessment Delivery
    This area includes all aspects of administering the assessment to candidates, respondents, participants, etc.
  • Presenting Results
    This areas deals with presenting results in a trustworthy way to the stakeholder in meaningful context.

Six key performance indicators, known as Measures, are tracked to provide an indication of maturity and these are:

  • Stakeholder Satisfaction
  • Security
  • Strategic Goals
  • Processes
  • Data Management
  • Communications with Stakeholders

image By tracking these Measures an organization can determine where they are and can plan for where they want to be. These can be tracked by a single Area, as shown in the graphic to the left, or with the three Areas combined for an overview. The graphic to the left shows how an organization can track by area based on "Quality" and "Efficiency".

If you manage any type of assessment program I’d encourage you to take time to learn more about the Areas, Measures and Phases of the Assessment Maturity Model.  or just

Please feel free to link to the Assessment Maturity Model web site at http://www.assessmentmaturitymodel.org/, cross link from other web sites and blogs, twitter about it, email me about it, comment here about it and/or even tell your friends about it! 

Watch out for more – this isn’t finished yet!

August 14, 2009 at 12:14 am Leave a comment

Types of Assessments – Learn, Qualify, Perform

image I wanted to share some work with you on how to logically organize assessments and came up with this simple model which seems to resonate with the people that I present it to.  You might also be interested in my previous article on Types Of Assessments (Formative, Diagnostic, Summative, and Surveys)..

There are assessments for learning and assessments to qualify people for certain activities or job roles; then they perform tasks. Throughout the process of learning, qualifying and performing, we accumulate evidence and feedback that help us make decisions. 

The definitions and distinction of the various assessment types that I refer to in this article are documented within the Assessment Maturity Model at http://assessmentmaturitymodel.wikispaces.com/Assessment+Solutions:

Learn

image Assessments Through The Learning Process can be distinguished and linked together by thinking of:

  • Pre-Learning Assessments
  • In-Learning Assessments
  • Of-Learning Assessments

Pre-Learning Assessments

The following assessments are used prior to learning events:

  • Job Task Analysis
  • Needs Analysis Surveys
  • Diagnostic assessments
  • Pre Tests
  • Placement tests
  • Self-diagnostic tools
  • In-Learning Assessments
  • Of-Learning Assessments

In-Learning

The following assessments are used within learning events to assist the learner learn:

  • Formative assessments
  • Quizzes during Learning
  • 360 Learner Peer Review
  • Practice Tests

Of-Learning

The following assessments are used to detect what a learner has learned:

  • Level 1 Surveys
  • Course evaluations
  • 360 (Level 3) Surveys
  • Post Course Tests
  • Graduation Exams
  • Internal exams
  • Open exams
  • Licensing exams

Qualify

image I struggled with this for a long time on whether qualify should be its own category but finally I came to the conclusion that it should because it provides sufficient distinctions on its own and could be separated from learning experiences. 

There are three basic categories of Assessments To Qualify and these have various styles of assessments associated with this:

  • Academic: Graduation Exams
  • Pre-Employment: Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • Certification & Licensing: Graduation exams, Internal exams, Open exams, Licensing exams

Perform

imageAfter learning and qualifying we end of up performing manual and intellectual tasks. And as we do things evidence surfaces that can be referenced to help us make decisions about more learning, more qualifications and our performance.

This evidence can be used and assessed within an organizations’ Performance and Talent Management processes to help people become more successful in their work 

Performance and Talent Management

image Performance Management is a method used to influence and manage behaviors and results with the goal of bringing out the best in people.  Talent Management refers to the phases of finding, developing and retaining people to perform activities. 

Within these phases people engage in learning and qualifications and their activities tends to accumulate evidence that assists them and others assess their performance.

Talent Management assessments can be categorized with the three phases of Talent Management:

  • Talent Acquisition – finding and employing the right people
    • Job Task Analysis
    • As per Pre-Employment assessments i.e. Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • On-Boarding – orienting new people to the workplace
    • As per Assessments Through The Learning Process
  • Performance, Talent and Team Management – to help people be successful
    • As per Assessments Through The Learning Process
    • Appraisal (360s)
    • Employee Attitudes
    • Opinion Surveys

I hope you have found this posting useful in showing how the types of assessments relate to Learning, Qualifying and Performing as well as getting things done!

August 3, 2009 at 12:09 pm Leave a comment

Learn, Qualify, Perform

This original article was enhanced, after feedback from customers and colleagues in the industry, around the concept of “Performing”.  I updated this article and will repost to save confusion.

imageI wanted to share some work with you on how to logically organize assessments and came up with this simple model which seems to resonate with the people that I present it to.  You might also be interested in my previous article on Types Of Assessments (Formative, Diagnostic, Summative, and Surveys)..

There are assessments for learning and assessments to qualify people for certain activities or job roles; then they perform tasks. Throughout the process of learning, qualifying and performing, we accumulate evidence and feedback that help us make decisions. 

The definitions and distinction of the various assessment types that I refer to in this article are documented within the Assessment Maturity Model at http://assessmentmaturitymodel.wikispaces.com/Assessment+Solutions:

Learn

image Assessments Through The Learning Process can be distinguished and linked together by thinking of:

  • Pre-Learning Assessments
  • In-Learning Assessments
  • Of-Learning Assessments

Pre-Learning Assessments

The following assessments are used prior to learning events:

  • Job Task Analysis
  • Needs Analysis Surveys
  • Diagnostic assessments
  • Pre Tests
  • Placement tests
  • Self-diagnostic tools
  • In-Learning Assessments
  • Of-Learning Assessments

In-Learning

The following assessments are used within learning events to assist the learner learn:

  • Formative assessments
  • Quizzes during Learning
  • 360 Learner Peer Review
  • Practice Tests

Of-Learning

The following assessments are used to detect what a learner has learned:

  • Level 1 Surveys
  • Course evaluations
  • 360 (Level 3) Surveys
  • Post Course Tests
  • Graduation Exams
  • Internal exams
  • Open exams
  • Licensing exams

Qualify

image I struggled with this for a long time on whether qualify should be its own category but finally I came to the conclusion that it should because it provides sufficient distinctions on its own and could be separated from learning experiences. 

There are three basic categories of Assessments To Qualify and these have various styles of assessments associated with this:

  • Academic: Graduation Exams
  • Pre-Employment: Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • Certification & Licensing: Graduation exams, Internal exams, Open exams, Licensing exams

Perform

image After learning and qualifying we end of up performing manual and intellectual tasks. And as we do things evidence surfaces that can be referenced to help us make decisions about more learning, more qualifications and our performance.

This evidence can be used and assessed within an organizations’ Performance and Talent Management processes to help people become more successful in their work 

Performance and Talent Management

imagePerformance Management is a method used to influence and manage behaviors and results with the goal of bringing out the best in people.  Talent Management refers to the phases of finding, developing and retaining people to perform activities. 

Within these phases people engage in learning and qualifications and their activities tends to accumulate evidence that assists them and others assess their performance.

Talent Management assessments can be categorized with the three phases of Talent Management:

  • Talent Acquisition – finding and employing the right people
    • Job Task Analysis
    • As per Pre-Employment assessments i.e. Pre-employment Screening, Pre-employment Skills Tests, Personality assessments, Psychological assessment
  • On-Boarding – orienting new people to the workplace
    • As per Assessments Through The Learning Process
  • Performance, Talent and Team Management – to help people be successful
    • As per Assessments Through The Learning Process
    • Appraisal (360s)
    • Employee Attitudes
    • Opinion Surveys

I hope you have found this posting useful in showing how the types of assessments relate to Learning, Qualifying and Performing as well as getting things done!

July 28, 2009 at 12:15 pm Leave a comment

Wikis and Social Constructivism: “Learning is best when learners collaborate to create…”

I wanted to share an article with you, that is short, meaningful and useful, on how wikis not only provide the benefits of knowledge sharing but also assists learners learn: Practical Applications of Research: Wikis for learning – what to expect by Dr. Irene Boland

From Dr. Boland’s research the key learning benefits of wikis appear to me to be:

  • Receive immediate feedback from others
  • Curiosity about how others are responding to, or modifying, their work causes learners to return to the wiki and refresh their memory regularly.
  • Learners can chart their own course for learning based on their interests.
  • Learn by reading other’s postings even if the leaner does not post.
  • Develop their own methods of classifying content in ways that make sense to them, rather than using existing systems of classification.
  • Help their peers find relevant content by assigning tags or keywords to pages

Enjoy!

July 6, 2009 at 9:24 am 1 comment

Learning Environments

This year I have been witnessing a change in thinking and common practices around learning environments. I’d like to share my point of view with you because I’m convinced that a significant shift is starting to occur. Let me start by giving a contextual overview in the form of a few tables:

Learning Delivery Systems

image

Assessment Systems

image

Tracking Systems 

image 

Marketing Hyper vs. IT’s Point of View

The stock market’s exuberance in the late ‘90s generated venture capital funding of learning environments and management systems start ups.This resulted in marketing hype that led to unrealistic expectations. Although vendors tried hard to meet these unrealistic expectations, customers were frustrated and the market was altogether unhappy and unhealthy.  This became obvious to me when customers would explain that they were on their third LMS. Clearly things had to change.

One strong benefit that came from the late ‘90s early ‘00s era was the clear separation that was drawn between “management systems” and “content”.  SCORM and AICC helped provide us with these clear distinctions, however, we didn’t manage to achieve sufficient distinctions between the key modules of a learning environment as web-services had not matured quickly enough.  Consequently, large monolithic systems tended to capture buyers’ imaginations as being the big pill to swallow to solve the learning problem.

From my point of view, IT departments initially pandered to users’ requirements and assisted with the deployment of customized and dedicated learning systems. However, as costs of tightly integrated systems increased and the number of dissatisfied users increased, IT departments started to look at alternative systems and architectures that could both meet the users’ requirements and align with organizations’ overall IT infrastructure.  IT started to look at learning from a users and performance perspective and started to apply IT methodologies to the issue.

Users’ requirements varied on one side from structured learning (course based), mostly used in schools, colleges and on-boarding new employees, through to self-service, where a motivated learner sought out the information that they needed in order to perform their tasks or gain qualifications.  Deploying only one learning methodology (course based vs. self-service) does not fit all requirements. Knowing the context of the user is key to providing a great user experience!

So what are we seeing now?

Authentication and Single Sign On (“SSO”) Portals

Learning materials, documents and content live in many systems. The issue is access control to these systems for the right person at the right time. This motivates a need for federated searches of multiple content repositories and to mashed up user interfaces that allow users to view multiple systems.

imageAs I have illustrated to the left, you can now provide access to multiple applications with a common look and feel using a portal such Microsoft’s SharePoint.  Each application is “skinned” by the portal’s feel and is presented in a “portlet”.  Depending on the user’s privileges they can add, change, delete, and move around portlets to suit their working style and job role.image

Behind the portal sits a number of systems that a user will use to perform their tasks. 

The portal looks after two important functions:

  • Authentication of the User
    This is usually performed by way of user names and passwords but could also be achieved via biometrics.
  • Access Control (Privileges)
    To limit access to the portlets that a user can actually see and use.

We have to be careful that we don’t return to dispersed SSO portals with one portal for learning, one for accounting, etc.; that would give us Multiple Sign On Portals which would be a retrograde step. 

True Single Sign On Portals provide IT departments with a single but centralized access control system and allow the user to define their portal to accommodate their style of working.

Authorization

imageEach application has its own unique set of privileges that would become difficult for a centralized IT team to control. For instance, in the context of testing, we might only allow users to take a test between certain hours and maybe require a proctor/invigilator. Whilst it would be possible for all of these user privileges to be stored more centrally and associated with the portal, this becomes impractical and slows the upgrade process.  However, it is common for application systems to receive data from the portal and then derive privileges based on the user’s associations.

Islands of Data

imageThe challenge with this architecture is that each application maintains its own databases for its operational needs such as storing course evaluation data and tests results.

We are seeing leading edge employers look at employee life cycle data in order to improve their talent management systems and processes.  From recruiting to on-boarding, appraisals, formal/structured learning, informal learning, career progression, and exit interviews we see present and future requirements for viewing and correlating data from multiple courses to help people understand the dynamics of their talent.

So learning environments are now being built in such a way that their data is consumable by web-services and a data warehouse.

Data Warehouses

imageOrganizations maintain Data Warehouses (DWs) for several reasons:

  • DWs can  be structured to quickly access data for reporting applications to consume rather than focus on efficiently collecting and maintaining data. 
  • DWs allow for data to be connected and correlated despite the fact that they are generated by different systems potentially using different types of databases.
  • DWs can insulate reporting systems from application upgrades which would otherwise necessitate reporting systems to be updated at the same time that the application upgrade is performed.

Wikis and Blogs

Wikis and blogs have become very important and popular tools for harvesting knowledge from subject matter experts (SMEs). With succession planning being important for many organizations that have aging work forces, harvesting knowledge from SMEs is a key initiative. 

Conclusion

When we combine IT department support for standardization on portals, wikis, blogs, data warehouses and reporting systems we can see that the time is right for a revolution within our learning environments.

The data and anecdotal evidence that I have access to makes me believe that learning environments will change and become more aligned with standard and supported IT systems.

We are now in an era where resonating with IT, their requirements, and their systems, will assist us in rapidly deploying powerful, integrated, and scalable environments for our learners, and performers.

June 16, 2009 at 8:33 am 1 comment

Fidelity of an Assessment

imageFrom the calculation shown above, you can see that a worthy goal of creating, delivering and reporting on assessments is to minimize the Error of Measurement. However, this competes with another worthy goal; ensuring that assessments are affordable.

imageThe more we can simulate the actual performance environment and the actual knowledge to be recalled, and/or the skill and/or ability to be used the smaller the error of measurement.   So if we were assessing your driving performance we should put you in a car and go driving with you.

Simple enough! Driving without crashing is a worthy goal but hardly the bar that we should set for our driving tests!  We need to ask ourselves what real performance looks like, what are the behaviors that can predict good performance, and what is the appropriate ways to measure these.

So what are the performance characteristics of a good driver?  Good eye sight, ability to control the car, spatial awareness, understanding and obedience to road signs, signaling intent to others, etc.  It might be better to assess some of these attributes before we jump in a car!  So let’s determine the criteria for successful performance and then assess each attribute (sight, road signs, rules of the road, etc) with one form of assessment before we start witnessing someone’s actual performance.

imageWe would start by determining the knowledge, skills and abilities required to perform a task in a real world situation.  Then we would use an appropriate assessment and select less expensive assessments to start with and then progress closer to real world environments.

Now the question is, how do we assess someone’s potential performance within a low cost simulated environment?  The key is to place someone into the context of the performance environment. We can do this by using scenario questions with the stimulus that we can afford/justify:

  • Low cost
    Tell a story about a situation and ask questions related to that story.  “You travelled to Morocco with your friend who rented a car and you came upon a road sign blah, blah, blah. When would you turn left?”
  • Medium cost
    Produce pictures and sounds and let the pictures and sounds tell the story. “Please follow along with the pictures and answer the question when would you turn left?”
  • High Cost
    Show a video that simulates driving. The person could either interact with the simulation or answer multiple questions about the video and or simulation as they progress through the experience.

The closer the simulation is to the real world (sounds, smell, sight, danger, etc), the more accurate the measurement.  When we simulate something to 100% we are in the real world!

In certain situations we want to measure your performance whilst your adrenaline levels are high or when your fight or flight mechanisms are kicking in. To do that we have to raise the fidelity of the simulation without causing harm.

For instance, suppose we want to learn how you would react in a crash.  We can’t go around having you crash things as that would be a danger to yourself and others. However, by using low and high fidelity simulations we could produce pretty accurate predictors on how you would act during a crash. 

imageThe  picture above tries to illustrate this idea of fidelity. The higher the fidelity of the stimulus the more accurate the measurement can be.  However, maybe we don’t need or can’t afford high fidelity. So we have some options:

  • Text stimulus

    Advantages: Easy, inexpensive, and appropriate for many situations.
    Disadvantages: Might inappropriate assess reading skills. Rarely stimulates the body to react with fight or flight mechanisms which might cause a different outcome in real life than during an assessment. 

  • Picture stimulus

    Advantages: Easy, inexpensive, images can convey a real world situations. Can focus on assessing the topic rather than also assessment language skills.
    Disadvantage: Rarely stimulates the body to react with fight or flight mechanisms.

  • Video stimulus

    Advantages: Videos can repeatedly convey real world situations, Can focus on testing on the topic rather than also assessment language skills. Can stimulate more emotional reactions.
    Disadvantages: More expensive to produce.


  • Interactive stimulus (Gaming)

     

     

     

    Advantages: Games can convey real situations. Can focus on testing on the topic rather than also assessment language skills. Can stimulate more emotional reactions.
    Disadvantages: More expensive to produce.

I hope that you find this helpful.

May 20, 2009 at 6:13 pm Leave a comment

Older Posts


Add to Technorati Favorites

Recent Posts

Categories