Summary: Learning evaluations are the criteria you will analyze to determine the success of your learning experiences. There are a few theorists who have developed different learning evaluation models which suggest key evaluation criteria relating to the micro-level as well as the macro-level.
Companies investing in online learning often don’t have the tools or practices to effectively evaluate their learning experiences. They tend to focus on shallow metrics such as course completion or the number of questions answered correctly in activities. While students may complete a course, there’s no telling if there’s behavior change, which should be the goal of the learning experience, and without the full picture of the impact of the learning experience, there’s no knowing its return on investment.
Successful learning experiences should increase skills and competencies in employees and teams, as well as contribute to an organizational learning culture, all while increasing revenue and decreasing costs. Measuring completion rates and final grades, particularly in self-directed online courses, will only result in poorly perceived adoption rates and in all likelihood, waning support from budget holders.
So, how do you measure the impact of learning?
This is where we can draw on the principles of monitoring and evaluation and a few well-known learning evaluation models.
Monitoring and evaluation is an approach predominantly used to measure the effectiveness of programs or initiatives in the development sector with the intention of having maximum impact. The principles of this approach can also be translated to measuring online learning experiences. Here are a number of ways you can apply M&E principles to learning:
With a monitoring and evaluation mindset, you’ll be able to more effectively implement and design criteria for evaluating learning effectiveness.
Learning evaluations are the criteria you will analyze to determine the success of your learning experiences. There are a few theorists who have developed different learning evaluation models, which suggest key evaluation criteria relating to the micro-level (the learning experience itself) as well as the macro-level (its impact on your business and even society). For example, Kirkpatrick’s model founded by training expert Donald Kirkpatrick indicates that a learning experience should be evaluated across the following four levels: reaction, learning, behavior, results.
Roger Kaufman, an expert in evaluation and needs assessment, expanded upon Kirkpatrick’s model to include an evaluation of a learning experiences’ societal impact in what is termed Kaufman’s Five Levels of Evaluation.
Knowing your general evaluation criteria introduced above is a good start, but you also need to identify the specific metrics that indicate whether learning was successful or not.
Begin with tracking the achievement of the learning objectives. At the start of the learning design process, you should have decided on objectives and competencies that you would like learners to develop through the learning experience. These not only help to clarify learning and motivate learners toward specific goals but are also useful for measuring whether the learning experience achieved the objectives or not.
Other metrics for evaluation include:
Using digital data, through xAPI, is also a good indicator to note how learners interact with your course and other online platforms in your organization. xAPI, or experience application program interface, allows you to track and view how learners use your course, what they click, how much time is spent on which activities or content, what was revisited, and so forth. This data is invaluable when analyzing the success of individual components of your course. In the case of informal workplace learning where employees are largely self-led in their learning, xAPI can also track engagement with different browser tabs and applications, giving you a sense of the employees' commitment to continual learning.
Ultimately, having an understanding of the various learning evaluation models and identifying useful metrics is the first step toward the effective evaluation of your online learning experiences. Coupling this with understanding your learning analytics will result in more enhanced and accurate learning return on investment and more improved learning experiences for your learners.
Elevate Learning is a full-service learning design consultancy. We partner with you to design and implement online and blended learning experiences, organisation-specific learning programmes, and experiential content and workshops.
