The field of evaluation is well established and as a result, there are a number of different approaches and theories. In order to develop a practical, non-labour intensive and therefore cost-effective tool, we (Higher Level Learning) opted for a goal-based approach (see 2. below) whilst making certain assumptions regarding environmental variables.
Robinson (2002) lists the five main approaches developed by different evaluation theorists, below. ‘While the different approaches are all attempting to answer similar questions (about the worth of programmes and elements of them), the emphasis on various aspects (purpose, focus and evaluator’s role) varies, as does the underpinning theory. However, all
share at least one feature: a concern for rigor in the conduct of the evaluation and a concern for reliable and systematic evidence to support any conclusions.’
|Approach||Emphasis||Focusing issues||Evaluator’s role||Specific information needs|
|1. Experimental||Research design||What effects
can they be
|Expert/ scientist||Outcome measures.
Variation in treatment.
Other influences on learners.
Availability of a control group.
|2. Goal oriented||Goals and
|What are the
how can they be measured?
|3. Decision focused
need to be
made and what
|Stage of programme
Cycle of decision making.
Data gathering and reporting.
users or clients
|Who are the
users and what
be most useful?
Intended uses of information.
have a stake in
and what are
their points of
|Variation in individual and group perspectives.
Stakeholder concerns and participation in
determining and framing the data.
Variation in measures and sites.
Whether you’re an elearning designer or developer or anyone working in the field of learning, measuring the efficacy of learning is sometimes regarded as a difficult or almost impossible task especially measuring the transfer of skills in the workplace. Consequently it is rarely attempted or only ‘surface scratched’. We’ve done some work on this at Higher Level Learning and can hopefully provide one method that works and has been tried and tested in the work environment. We call it SeET – Snapshot elearning Effectiveness Tool.
Our criteria for success was:
One question before we go any further:
In the current financial climate, it’s more important than ever for elearning and learning programmes to demonstrate their value and the benefits they provide to an organisation. This means creating programmes and initiatives clearly linked to business objectives and having effective tools in place to measure their success (hence the creation of SeET).
Scenarios can be used to immerse learners in a work related situation and expose them to a given situation and ask them to apply knowledge relevant to that situation. They can be developed to address single or multiple learning objectives.
This type of learning, also known as experiential learning, occurs when we immerse ourselves in a situation in which we are forced to make a decision.
Working as elearning designers and / or elearning developers, we come across many different types of business who want online learning creating or face to face training, converting to online learning. From what I’ve seen the message still isn’t getting through that people learn from involvement and decision making and not from reading pages and pages of Content. I saw it yesterday (literally) and I’ll see it tomorrow. So how do we persuade subject matter experts (SMEs) to write differently?
Every training intervention has a reason and it’s usually a problem. Businesses don’t spend money on training unless they have to. E.g. A food manufacturing manager needs to educate staff on food hygiene – otherwise customers will more than likely acquire food poisoning. A NHS manager has to train 500 nurses to use a new type of thermometer – otherwise patients may be ill and go undetected. Everything’s a problem and problems need to be fixed.
So, tell your SME that. In fact first, ask them what problems will be solved by the training – why are they being asked to write the training Content? Then tell them to write as though they are going to fix the problems.
For example as a food hygiene employee, what problems are likely to emerge? Tell them to write a scenario to fix each potential problem, for example, a customer has become ill. How can that of been prevented? What should be done to ensure it never happens again? What decisions should be made – what are the right and wrong decisions? Ask them to take the learner through the process but ask them to write the Content as a scenario in a non-linear fashion. To fix the problem, what is the correct route – the right decisions, and what is the incorrect route – the wrong decisions. Ask the to write it like a branch of a tree but ask them never to create more than 3 branches – from experience, after 3 or 4 it becomes too complicated to design and build.
Now we have some theory to underpin the game design, it’s time to start the practical game design. As an elearning developer / designer you will have a preferred rapid development tool. Some of the obvious ones being Articulate Storyline 2, Captivate 9, Lectora. All of these have their various advantages and disadvantages. For a good assessment of them all see: http://elearninguncovered.com/2013/01/e-learning-authoring-tool-comparison/. I had to choose one to create a game so I went with Storyline 2 for no other reason than, this is the last tool I used to create an elearning course, but I’ll try and keep the instructions as generic as possible. I’ll provide a link to the finished course at the end so you can download it and look at the structure/actions/code etc. I’ll also try and break down the elearning development into salient sections – one post a section. Keep the Octalysis gamification framework at hand to check what you need to think about as you start to develop: http://yukaichou.com/gamification-examples/octalysis-complete-gamification-framework/
It would be remiss of me to go any further without mentioning some kind of learning design documentation – useful for the elearning developer / designer. A design document sets out what form the game will take, how learning will occur and generally gives you something to work from and refer to. As a serious games’ designer I like to mix the Learning Design with the game design document – how does the game meet the learning objectives – remember this is not just about Gamification, it’s about creating a game to help people learn – a Serious Game: differences between Gamification and Serious Games
You can call it what you like, I tend to call it a Learning Design Document.
If you’re developing a Serious Game for a client they will expect to know what you are going to produce and why you are producing it in that way. They will want to know how your game will meet their learning objectives. It should form something along the lines of:
Let’s start by creating learning objectives – useful for the elearning developer / designer – well basically anyone that’s trying to create some kind of learning!. The objectives should be based on the problems the business is going to solve. For example let’s say that the Customer Support Team at Rob’s carpets are making the following common mistakes when logging new customers on the system over the phone:
So our learning objectives should be something along the lines of:
By the end of this course you should be able to:
You can make your objectives ‘SMART’ objectives but I’m going to keep them ‘unsmart’ as not to over complicate the design at this stage. If you want to learn more about creating ‘SMART’ objectives and also download a handy PDF, try this: Creating SMART objectives
In the next post I’ll explain how the objectives will form the basis of the quest game.
Extrinsic motivation has it’s place in gamification and learning however the goal of any designer should be to tap into the intrinsic motivation – it lasts longer and ultimately is more productive for the designer, the game and the learning.
Bespoke eLearning Development
Week 1 will be our ‘defining week’ for the elearning developer to begin work. We will identify a problem that we need to solve with some kind of elearning game. We will also define the motivation of the ‘players’ as without motivation they will not play and consequently they will not learn. And finally we will decide on the two main game elements 1) the fun aspect 2) how learning will occur as a result of playing the game.
So let’s invent a potential bespoke elearning problem and then see how we can develop a game using Storyline to resolve it…
Problem: Staff in a business (let’s call it Rob’s Carpets) have recently received training on a new system that allows them to log how new customers made first contact with the business. However many of the staff are making mistakes and need some kind of refresher training.
So to provide the extra training the staff need, we will create a game. This game will provide the refresher training in order to reduce the mistakes made.
Let’s give the game a name: Customer Quest (it’s probably a good idea to keep the name in the positive sphere – I wouldn’t recommend for example calling it – ‘Stop making bloody mistakes!’. The trick is to train staff without them realising it)
Fun element: The game will be created as a simple quest type game.
Learning element: The game will introduce questions and feedback which will help the ‘players’ to learn as they progress through the quest.