How Do We Measure the Impact of Training?

How do you evaluate the effectiveness of our training?  This has been a question that we are still seeking to answer in SEND U.

One realization that I have come to in the past few years is that asking the participants how well they liked the training is not enough of a measure. We regularly receive positive comments about the various training initiatives we do for our mission members, either on evaluation sheets hurriedly filled out in the last 5 minutes of the training event, or in conversations with people walking out the door. But while it is gratifying to hear that they enjoyed the workshop, what we want to see happen is for the participants to change their behaviors as result of what they have learned.

The training evaluation guru is Donald Kirkpatrick, and his legacy is continued by his son, Jim Kirkpatrick.  Kirkpatrick identified four levels of evaluation, and all of them are valuable and necessary.

The four levels of Kirkpatrick’s evaluation model:

1. reaction of participant- what they thought and felt about the training
2. learning – the resulting increase in knowledge, capability or confidence
3. behavior – extent of application of new skills and knowledge in the work
4. results – the impact on the work or environment resulting from the trainee doing things differently

Each of these supposedly can be measured, although admittedly the last one is the most difficult to measure.   As believers, we can never attribute any fruit to our efforts alone, because God’s intervention is absolutely crucial to any lasting impact.  Furthermore, we recognize that each of us is only part of God’s overall plan of building His church (see 1 Cor 3:5-7).   Complicating Level 4 evaluation even further is that the typical missionary has participated in multiple training events (college, seminary, internships, short-term experiences, mentoring, multiple pre-field training events, etc) before they arrive on their field of service.    How can we possibly identify what each training program contributed to the overall impact of the missionary’s ministry?

I am not about to give up in attempting to measure the impact of our training (Level 4).   But Level 3 evaluation is somewhat easier to do.  Definitely we should be able to observe and evaluate the degree to which our training has been applied in the missionary’s life and ministry, even if we are not always sure that the training we are evaluating was the sole contributor to the change in behavior.

But to do Level 3 evaluation well means that the evaluation cannot be done immediately or even soon after the completion of the seminar.   Sometimes we have to wait years before the participant has had a full opportunity to apply the learning.    This is far from an ideal situation, because the longer the delay, the more that is forgotten. Unfortunately, too often the time lag between pre-field training and when a missionary begins full-time cross-cultural ministry is far too long.   For many, waiting for the remainder of their financial support to be pledged delays departure for the field.   To reduce this wait, SEND is now requiring our new missionary appointees to have 66% of their support raised before they attend our Member Orientation Program.   But the biggest reason for the delay between learning and application is language learning.  Reaching a level of proficiency that allows for a full-time ministry assignment generally takes two years, and sometimes longer with the more difficult languages.

So, for a full evaluation of what behaviors were changed as a result of pre-field training, we typically need to wait 3 or more years. When asking a first-term missionary what was most helpful about what they learned in pre-field training, it is more than a little disconcerting to hear the response, “Tell me again what we learned at MOP!”With this length of time between the learning event and the application, not only do we need to evaluate whether anything that was learned was applied in their cross-cultural context.   Just as importantly, we need to refresh the memories of our new missionaries, reminding them what they learned back in pre-field training.  Or did they unlearn it?

SEND has instituted an on-field training program for new missionaries that seeks to do exactly that.  We call it MOP-up (Member Orientation Program under pressure).   MOP-up training gives us an opportunity both review the key principles we studied in pre-field training and observe how well our first-term missionaries are applying these principles.  It happens over a 3-month period and is required for all first-term missionaries in SEND.   MOP-up is essential an individualized self-review, combined with coaching and a couple of assessment tools.   During these three months, the first-term missionary reviews the pre-field training notes (their own notes or those on the SEND U wiki).   Throughout this time, they meet regularly with a trained coach who will guide them through the process of reviewing and evaluating themselves as to how well they have applied what they learned in pre-field training.  They then write up a self-assessment of their application in 4 key areas and send this to their coach.

This on-field program only began a little over a year ago, so we are still evaluating the MOP-up program itself and making course corrections along the way.   But despite its imperfections, this training program is helping us evaluate the effectiveness of a prior training program (the pre-field Member Orientation).   Some initial observations are encouraging.  As I have coached various missionaries through MOP-up, I have been quite impressed with the intentional measures taken by our new missionaries to maintain their spiritual vitality and adopt an incarnational lifestyle and learner attitude toward the culture in which they are living.    These are major emphases of our pre-field program, so it is gratifying to see the application.

But there is much more that we can learn about the impact of our pre-field training program by observing and asking good questions of our on-field missionaries – both the new arrivals on the field who are seeking to implement what they learned and also their co-workers and team leaders who are seeing their struggles and successes in making that transition.    This feedback loop is necessary if we are going to make adjustments to the training that will result in more fruitful missionaries over the long term, not just happy and satisfied participants in a training exercise.