How to Measure the Efficacy of elearning / learning Part 2


How to Measure the Efficacy of elearning / learning Part 2

Evaluating eLearning / learning

Introduction to Evaluation of elearning / learning

The field of evaluation is well established and as a result, there are a number of different approaches and theories. In order to develop a practical, non-labour intensive and therefore cost-effective tool, we (Higher Level Learning) opted for a goal-based approach (see 2. below) whilst making certain assumptions regarding environmental variables.

Evaluation Theories

Robinson (2002) lists the five main approaches developed by different evaluation theorists, below. ‘While the different approaches are all attempting to answer similar questions (about the worth of programmes and elements of them), the emphasis on various aspects (purpose, focus and evaluator’s role) varies, as does the underpinning theory. However, all

share at least one feature: a concern for rigor in the conduct of the evaluation and a concern for reliable and systematic evidence to support any conclusions.’


Approach Emphasis Focusing issues Evaluator’s role Specific information needs
1. Experimental Research design What effects
result from
activities and
can they be
Expert/ scientist Outcome measures.
Learner characteristics.
Variation in treatment.
Other influences on learners.
Availability of a control group.
2. Goal oriented Goals and
What are the
goals and
objectives, and
how can they be measured?
Specific programme
outcome measures.
3. Decision focused
Which decisions
need to be
made and what
information will
be relevant?
support person,
provider of
Stage of programme
Cycle of decision making.
Data gathering and reporting.
4. User-oriented Information
users or clients
Who are the
users and what
information will
be most useful?
Collaborator Personal and
Group information
Programme history.
Intended uses of information.
5. Responsive Personal
Which people
have a stake in
the programme
and what are
their points of
Variation in individual and group perspectives.
Stakeholder concerns and participation in
determining and framing the data.
Programme history.
Variation in measures and sites.

How to measure the efficacy of elearning / learning Part 1

How to measure the efficacy of elearning / learning

Whether you’re an elearning designer or developer or anyone working in the field of learning, measuring the efficacy of learning is sometimes regarded as a difficult or almost impossible task especially measuring the transfer of skills in the workplace.  Consequently it is rarely attempted or only ‘surface scratched’. We’ve done some work on this at Higher Level Learning and can hopefully provide one method that works and has been tried and tested in the work environment. We call it SeET – Snapshot elearning Effectiveness Tool.

Our criteria for success was:

  • A practical solution that can be used by staff with little experience of evaluation
  • A tool that steps beyond Kirkpatrick level one and measures behavioural changes in the workplace based on completed elearning (or just learning)
  • A solution that can be quickly deployed and easily measured in the workplace
  • A cost-effective solution
  • the tool should not be limited to evaluating elearning and could be adapted for use in other areas

One question before we go any further:

Why measure e-learning or learning at all?

In the current financial climate, it’s more important than ever for elearning and learning programmes to demonstrate their value and the benefits they provide to an organisation. This means creating programmes and initiatives clearly linked to business objectives and having effective tools in place to measure their success (hence the creation of SeET).


Scenario Based Learning

elearning developers, designers – download your free scenario based learning template (Word format) scenariobasedlearning

Scenarios can be used to immerse learners in a work related situation and expose them to a given situation and ask them to apply knowledge relevant to that situation. They can be developed to address single or multiple learning objectives.

  • SBL presents learners with:
    • Issues
    • Challenges
    • Dilemmas
    • Emotional attachment

This type of learning, also known as experiential learning, occurs when we immerse ourselves in a situation in which we are forced to make a decision.

Our Expertise


Persuading subject matter experts to write for scenarios

elearning developers

Working as elearning designers and / or elearning developers, we come across many different types of business who want online learning creating or face to face training, converting to online learning.  From what I’ve seen the message still isn’t getting through that people learn from involvement and decision making and not from reading pages and pages of Content. I saw it yesterday (literally) and I’ll see it tomorrow. So how do we persuade subject matter experts (SMEs) to write differently?

Every training intervention has a reason and it’s usually a problem. Businesses don’t spend money on training unless they have to. E.g. A food manufacturing manager needs to educate staff on food hygiene – otherwise customers will more than likely acquire food poisoning. A NHS manager has to train 500 nurses to use a new type of thermometer – otherwise patients may be ill and go undetected. Everything’s a problem and problems need to be fixed.

So, tell your SME that.  In fact first, ask them what problems will be solved by the training – why are they being asked to write the training Content? Then tell them to write as though they are going to fix the problems.

For example as a food hygiene employee, what problems are likely to emerge? Tell them to write a scenario to fix each potential problem, for example,  a customer has become ill. How can that of been prevented? What should be done to ensure it never happens again? What decisions should be made – what are the right and wrong decisions? Ask them to take the learner through the process but ask them to write the Content as a scenario in a non-linear fashion. To fix the problem, what is the correct route – the right decisions, and what is the incorrect route – the wrong decisions. Ask the to write it like a branch of a tree but ask them never to create more than 3 branches – from experience, after 3 or 4 it becomes too complicated to design and build.

Our Expertise


Articulate Storyline Short Cuts

e learning Developers

e learning developers

e learning developers

Download this Articulate Storyline keyboard shortcuts document courtesy of elearning Uncovered. Storyline-2-Keyboard-Shortcuts



Start the game design in Storyline

Bespoke eLearning Development

eLearning Developer

Now we have some theory to underpin the game design, it’s time to start the practical game design. As an elearning developer / designer you will have a preferred rapid development tool. Some of the obvious ones being Articulate Storyline 2, Captivate 9, Lectora. All of these have their various advantages and disadvantages. For a good assessment of them all see: I had to choose one to create a game so I went with Storyline 2 for no other reason than, this is the last tool I used to create an elearning course, but I’ll try and keep the instructions as generic as possible. I’ll provide a link to the finished course at the end so you can download it and look at the structure/actions/code etc. I’ll also try and break down the elearning development into salient sections – one post a section. Keep the Octalysis gamification framework at hand to check what you need to think about as you start to develop:

elearning developer

elearning developer


Learning Design Documentation

Bespoke eLearning Development

eLearning Developer

It would be remiss of me to go any further without mentioning some kind of learning design documentation – useful for the elearning developer / designer. A design document sets out what form the game will take, how learning will occur and generally gives you something to work from and refer to. As a serious games’ designer I like to mix the Learning Design with the game design document – how does the game meet the learning objectives – remember this is not just about Gamification, it’s about creating a game to help people learn – a Serious Game: differences between Gamification and Serious Games

You can call it what you like, I tend to call it a Learning Design Document.

If you’re developing a Serious Game for a client they will expect to know what you are going to produce and why you are producing it in that way. They will want to know how your game will meet their learning objectives. It should form something along the lines of:

Learning Design documentation

  • Concept document:
    Describes the idea of the game and proposed game genre, target audience; most compelling features; how it will meet the learning objectives; cost and time to develop. It defines the concept, scope, worthiness and feasibility; sells the idea to your client. You may need a prototype of some kind.
  • Design Document:
    Describes the nuts and bolts of the entire project, with all the details and the method by which each element will be implemented. It describes how learning will occur for each of the learning objectives – in detail. It ensures that what is produced is what you want to produce and what your client wants you to produce.
  • Production Documents:
    Project and time-management details; task database; budget spreadsheet; technical specifications; Q/A database. It implements the design document on time and within budget. This may be the job of the project manager (if you have one). But they will require your input  on the learning design aspect of the project so you will need to know this in some way or another.

Creating Learning Objectives

Bespoke eLearning Development

eLearning Developer

Let’s start by creating learning objectives – useful for the elearning developer / designer – well basically anyone that’s trying to create some kind of learning!. The objectives should be based on the problems the business is going to solve. For example let’s say that the Customer Support Team at Rob’s carpets are making the following common mistakes when logging new customers on the system over the phone:

  • The full name and address including postcode is not being recorded
  • A landline and mobile number are not being logged correctly
  • The ‘new customer’ tick box is not being ticked before saving
  • How the customer heard about the business is not being added
  • The details of ‘Rob’s Carpets’ website is not being given to customers over the phone

So our learning objectives should be something along the lines of:

By the end of this course you should be able to:

  • Enter the customer’s full name and address including postcode
  • Enter the customer’s landline and mobile telephone numbers
  • Ensure that the customer is logged as a ‘new customer’ before saving
  • Log how the customer heard about the business
  • Ensure that the details of ‘Rob’s Carpets’ website is given to customers over the phone

You can make your objectives ‘SMART’ objectives but I’m going to keep them ‘unsmart’ as not to over complicate the design at this stage. If you want to learn more about creating ‘SMART’ objectives and also download a handy PDF, try this: Creating SMART objectives

In the next post I’ll explain how the objectives will form the basis of the quest game.


Intrinsic vs extrinsic motivation

Bespoke eLearning Development

As an elearning developer / designer, It is important to distinguish between these two types of motivation:

  • extrinsic motivation – where you are engaged because of a goal or reward;
  • and intrinsic motivation – where the activity itself is fun and exciting, with or without a reward.

Extrinsic motivation has it’s place in gamification and learning however the goal of any designer should be to tap into the intrinsic motivation – it lasts longer and ultimately is more productive for the designer, the game and the learning.


Week 1 Storyline Development


Week 1 Aims


Bespoke eLearning Development

Week 1 will be our ‘defining week’ for the elearning developer to begin work. We will identify a problem that we need to solve with some kind of elearning game. We will also define the motivation of the ‘players’ as without motivation they will not play and consequently they will not learn. And finally we will decide on the two main game elements 1) the fun aspect 2) how learning will occur as a result of playing the game.

bespoke elearning problem

So let’s invent a potential bespoke elearning problem and then see how we can develop a game using Storyline to resolve it…
Staff in a business (let’s call it Rob’s Carpets) have recently received training on a new system that allows them to log how new customers made first contact with the business. However many of the staff are making mistakes and need some kind of refresher training.
So to provide the extra training the staff need, we will create a game. This game will provide the refresher training in order to reduce the mistakes made.

Let’s give the game a name: Customer Quest (it’s probably a good idea to keep the name in the positive sphere – I wouldn’t recommend for example calling it – ‘Stop making bloody mistakes!’. The trick is to train staff without them realising it)
Fun element: The game will be created as a simple quest type game.
Learning element: The game will introduce questions and feedback which will help the ‘players’ to learn as they progress through the quest.