The question of the effectiveness of learning programmes has probably been asked for as long as structured learning programmes have existed. How do you know if a learning programme is meeting its objectives? It is easiest to answer this question in retrospect, and there is much thoughtful and helpful theory and research on this evaluation of learning programmes, e.g. distinguishing between different levels of evaluation, from participant satisfaction to actual achievement of organisational goals through new thinking and action by learners.
You can tell the effectiveness of learning programmes by their architecture
We have always been fascinated by the question of whether it is possible to assess the effectiveness of learning programmes before they are implemented, by looking at their design. Can you assess whether a programme will be more or less effective just by looking at its "architecture", its design? - We think you can, with some reservations.
The caveats first: a large part of the effectiveness of learning programmes depends on specific factors that only manifest themselves in the implementation: The quality of the facilitators, their daily form, the motivation of the participants, their daily form, for example. A good learning design will always try to influence these factors positively, but the extent to which this succeeds cannot be definitively determined in advance.
Accepting these trade-offs, we believe there are a number of criteria that can be used to assess the design of a learning programme to see how likely it is to be effective for learners. In other words, certain decisions in the design of the learning programme increase the likelihood of its effectiveness, while others reduce that likelihood.
Six criteria of effective learning architectures
From our experience, these criteria are:
- Integration: Is learning organised largely separately from the daily work process, or largely integrated?
- Individualisation: Does the programme take general theories as a basis, or the individual cases of the participants?
- Self-responsibility: To what extent does the programme allow participants to take responsibility for their own learning?
- Flow: Do the different learning interventions build on and reinforce each other (the learning programme as a "line"), or are they individual moments that stand relatively independent of each other (the learning programme as "points")?
- Beauty: How much radiance, aesthetics and surprise does the programme have, all of which increase participants' motivation and willingness to engage in learning?
- Consistency: How much internal and external consistency does the learning programme offer? In other words, are the different elements of the learning programme based on a recognisable set of design principles (= internal consistency)? Are these design principles aligned with the context and goals that apply to the learning programme, i.e. does the form support the content (= external consistency)?
Why is our experience that these criteria have a high influence on the effectiveness of learning programmes? - We describe this in the next chapters, criterion by criterion.
These criteria are neither exclusive nor do they form an exact science. But, in our experience, they are a good yardstick for making an initial assessment of a learning programme's likely effectiveness based on its design. Or, to put it another way: from our perspective, anyone who designs learning programmes would do well to use the above criteria and base the design on them.
But this is anything but self-evident. The classical learning programme is still strongly designed based on old-fashioned school experiences:
- It is designed and structured from some universally valid content (what subjects are in school, topics such as "communication", "decisions", "conflicts", etc. are in learning programmes in the work context);
- it takes place largely in seminar rooms (the adult classroom), i.e. outside the work context;
- it prescribes a lot for the learners instead of relying on self-responsibility (this is how many experienced it as students);
- and communication about and within the programme is mainly functional, unsurprising and not necessarily highly aesthetic (this is also in line with many people's school experiences).
"Integration" does not always apply as a criterion
Can these criteria be applied to all learning? - In principle, yes. Except for the first criterion - integration - the criteria mentioned apply to all learning designs, regardless of the learning topic, be it to varying degrees.
Integration is especially important for learning topics that deal with interaction between people - leadership, cooperation, communication, dealing with clients. In fact, all those issues where there are others who will notice something about new action, and where a change of action or learning itself potentially involves social tension. For learning topics that have a focus on cognitive understanding, that have little potential "embarrassment", whose execution is primarily individual, classrooms or e-learning are excellent, for example for learning how to use a new software.
Instructional and experiential learning should not be confused
Yet another distinction is relevant to the effectiveness of a learning programme: that between instructional learning and experiential learning. This distinction is not black and white, and it has implications for the role of the 'trainer' between 'instructor' (I know something that I pass on) and 'facilitator' (I guide learners in their individual learning process). How appropriately this distinction is made has a major impact on learning outcomes.
Instructional learning tends to be the more "hands-on". When there is a well-researched best way to solve a questionable task. How to operate a machine, how to have a good feedback conversation are examples of different topics that have a large craft part.
Experiential learning tends to be helpful when the more 'artistic' issues are involved. When there are many ways to approach a task, when it is about one's own deeper insight, one's own style. Examples: How to make (one's) emotions productive in the work context, or how to use one's talents well. Even tricky craft questions sometimes require practice (= experience) in addition to instruction, e.g. how to drive a car or how to bake a loaf of bread using only water, flour and salt.
As a learner, you notice immediately when instructional learning and experiential learning are not used effectively: For example, you find yourself confronted with a question that you can't answer at all, but for which the teacher clearly knows the right answer. Or you try around with something and just feel lost. Or you get lots of instructions, while you really want to think or try things out for yourself.
Is learning organised largely separately from the daily work process, or largely integrated?
In our view, the integration of learning into the work context is probably the most underestimated design criterion for learning. Hordes of learning professionals are still chasing the "transfer problem" that is considered inevitable: the question of how to ensure that what is learned (in the seminar room) is subsequently applied in work practice. However, the moment one sees working and learning as ultimately identical activities, the moment one lets learning take place as far as possible in the everyday social context, there is no longer any need for "transfer" - because learning takes place during work, with the colleagues it concerns.
Especially for learning topics where application in everyday work is fraught with hurdles - a new way of conducting conversations or leading meetings, for example - learning architectures that strongly situate learning in everyday work are more effective than those that outsource learning to other places and moments.
We distinguish four levels of integration of a learning architecture:
- (Almost) all learning interventions take place outside or without involvement of the work context, e.g. in seminar rooms, or as e-learning. The future application of the learning content in everyday work plays a subordinate role in the implementation of the learning intervention, apart from perhaps a final examination.
- The focus of the learning interventions is "classroom-based", but there are initial interventions for the time between the classrooms, e.g. tasks, reflections, instructions to try something out in everyday life.
- The programme includes a variety of learning interventions that take place in everyday life, but is still built on and around classroom modules. The content thread still follows the logic of these modules.
- The programme is detached from classroom modules, which now only make up the smaller part of the total learning time for participants. The learning is organised around the learners' practice, there are different learning interventions that take place in the learners' daily work.
A learning programme with a high degree of integration into the daily work routine looks completely different from a traditional seminar and thus takes some getting used to for many participants. Classical integrated formats are oriented, for example, towards "coaching on the job": the learner is accompanied internally or externally in a moment that is relevant to him or her, and during or only after the moment there is feedback and concrete tips for other actions. Variations of experimentation are also usually well integrated into the daily work routine; learners try out new strategies for action, accompanied in different ways, at relevant moments.
All forms of learning that involve the learners' social environment indicate a high level of integration: When colleagues take on support roles, interlocutors learn something of the learner's learning goal, managers are consciously invited to create a space for the learner.
Does the programme take general theories as a basis, or the individual cases of the participants?
The more learners recognise their individual learning questions and situations, the easier it is to apply what they have learned. We once encountered this basic question of individualisation in the redesign of a training for nurses.
Traditional training is oriented towards classical school subjects - these are usually the different expertise of the teachers, e.g. pharmacy, practice, maths, communication. These subjects are then taught as different "subjects". This is easy for the teachers because they can tell the same thing repeatedly with relatively little effort. The effort to apply what they have learned concretely to their own work is up to the learner.
When we redesigned the training, we based it on key situations ("critical incidents" - link). A training unit then deals, for example, with giving an injection - a concrete action that is immediately recognisable for the learners from their own everyday life. The teachers then have to bring together their different expertise as it interacts in the concrete moment: The conversation with the patient about the injection, the pharmaceutical check, if necessary, the calculation of the right amount, the practical setting of the injection.
We distinguished four levels of individualisation of a learning programme:
- Almost exclusively input-based learning, the general subjects and theories are the starting point and determine the rhythm of the programme. Logic of subjects, different teachers all "teaching" their respective expertise.
- First explicit translations of the general themes to the individual working situation of the participants. There is room to bring in individual cases and to apply the theory to these cases together with the teachers.
- General theories / topics are equal to the individual cases of the participants. The programme is based on key situations, for example, which have a strong recognition effect.
- The structure and rhythm of the programme is based on key situations or the individual questions or the individual learning process of the participants. Theories and models are adapted to the individual cases/key situations in their selection and in the way they are introduced.
To what extent does the programme enable participants to take responsibility for and shape their own learning?
Joseph Kessels, one of our founders, coined the phrase: "You can't be smart against your will". What sounds funny at first has radical consequences: Forced learning only works to a very limited extent. And yet there are still learning programmes to which participants are sent, learning content is ordered. Prescribing learning content can sometimes make sense - for example, if I want to learn a new profession but have no idea what it takes. But this predefinition then takes place within the framework of voluntariness, because I have decided to learn the new profession.
What works is when learners take responsibility for their own learning. And it is precisely this responsibility that a programme design can support - or exclude from the outset. Learners who feel responsible for their learning are more motivated to learn, are more likely to ensure that learning is tailored to their needs, and are much more likely to apply what they have learned. At the same time, taking responsibility in this way may be extremely unusual for learners in a programme - and they may still need to learn how to do it.
We distinguish four levels of personal responsibility in a learning programme:
- The learning programme and its contents are fixed from the outset. Participants are sent and go through the pre-determined programme.
- Learners come voluntarily. At the same time, the programme is largely fixed, but there are a few points where you can influence it or decide on certain elements yourself.
- The learners make a conscious decision for the learning programme. There is a fixed structure with content, but within the structure there are choices for the learners as to which learning interventions they want to use.
- Learners make a conscious choice to use the programme to address their own learning question. The structure offered is primarily aimed at guiding learners in dealing with their learning question (process orientation), content is largely integrated into the learning process as required and is only generally fixed in advance.
Do the different learning interventions build on each other and reinforce each other (the learning programme as a "line"), or are they individual moments that stand relatively independently of each other (the learning programme as "points")?
What image do we actually have of "learning"? - One possible image is the learning process as a sequence of relevant "aha moments". Each of these moments holds an insight, and this increasing insight is then learning. For some cases this may be true. But on the one hand, this picture is limiting; learning here is primarily cognitive-content-related. On the other hand, the picture implicitly assumes that insight is the ultimate goal of learning. Especially in the work context, however, insight is often not enough - what is relevant is the action that one hopes will follow the insight gained. Unfortunately, psychology has found that insight does not necessarily, and even shockingly often, does not lead to different action.
For us, a more appropriate image of "learning" is that of a developmental process in which experiences, insights, experiments, practice, reflection all play a role, build on each other, reinforce each other in ways that cannot always be planned.
The design of learning programmes often implicitly reflects one or the other of these two images. Our experience shows that programmes designed as a coherent process have a higher chance of changing actions and thus learning effectiveness than programmes consisting of individual "points".
We distinguish four levels of "flow" in a learning programme:
- The learning programme consists of independent, individual learning interventions, each of which typically focuses on one content topic. The main focus of the interventions is cognitive insight. The order of the learning interventions does not matter in principle.
- The learning programme still consists of individual learning points, but there are small efforts to build bridges between the points. The sequence of learning interventions still does not follow an explicit logic, but elements of one point are repeated in a following point, for example. The focus is still strongly cognitive.
- The learning programme consists of strongly salient learning points, but there is a logic in the sequence (what comes first, what after) and connection between the individual learning points. This structure makes the focus more holistic, also practising, experiencing, reflecting are part of the learning programme.
- The learning programme is a learning process. Visible above all is this line, the individual learning interventions are increasingly difficult to delineate and depend on each other. The focus and methodology are holistic, each learning intervention builds on the previous one.
How much radiance, aesthetics and surprise does the programme have, all of which increase participants' motivation and willingness to engage in learning?
At first glance, beauty does not come naturally as a criterion for effective learning programmes. In our experience, however, beauty is one of the most misunderstood variables in learning architectures. The essence of beauty in learning programmes is that they radiate attractiveness, consistency and harmony in all their manifestations. Beauty is evidence of attention paid by learning programme providers to participants and thus invites commitment. Beauty entices learners to invest in the programme.
We distinguish four levels of "beauty" in a learning programme:
- Here is no discernible effort at aesthetics, the programme's expressions seem random, have little recognition value, nothing indicates extra effort on the part of the providers beyond what is absolutely necessary.
- Communication with learners shows traces of aesthetics, e.g. there is a logo, emails are no longer purely functional, thought has been given to texts, locations are deliberately chosen.
- The programme looks attractive. Different elements of the programme are aesthetically coordinated, communication with learners is deliberately designed. Locations are inviting. Everything makes people want to learn.
- Aesthetics permeate all aspects, there are wow moments, surprises. You can tell that beauty is a conscious design criterion.
How much internal and external consistency does the learning programme offer? Where internal consistency means that the different elements of the learning programme are based on a recognisable set of design principles, and external consistency means that these design principles are oriented towards the context and goals that apply to the learning programme (the language of form supports the content).
"Form follows function" also applies to the design of learning programmes. Consistency" is about this aspect. Internal consistency means how the different elements of the programme are "of one piece", fit together, follow the same philosophy. Internal consistency makes it easier for learners to understand the function of different forms of learning through recognition.
External consistency means the extent to which the methods and forms used fit the (learning) objective of the programme. If one of the programme objectives in a leadership development programme is e.g. taking responsibility, then forms that aim at a strong regulation of the learning process would be externally inconsistent (because this regulation is the opposite of taking responsibility). A 2-hour lecture on "interactive didactics" is thus also externally inconsistent. External consistency increases the credibility of the intended goals.
We distinguish between levels of consistency in learning programmes:
- There are gross internal and / or external inconsistencies, little thought seems to have been given to the formal language of the programme.
- The majority of the programme is inconsistent, where goals and methods match, it seems to be coincidence rather than intention.
- The majority of the programme is consistent, methods seem to be deliberately chosen and fit together.
- The programme is almost entirely consistent, there are explicit design principles that guide the different methods.