HRD in Practice: A U.S. Department Uses Level 2 Evaluation
The U.S. Department of Transportation uses oral quizzes or tests for level 2 evaluation. Oral quizzes or tests are most often given face-to-face and can be conducted individually or in a group setting. Here is a typical example of the department’s level 2 oral quizzing:
1. When it comes to Highway Safety tell me two safety challenges you are facing right now in your state or region.
2. What are “special use” vehicles and what is special about them? 3. What type of crossing is required for train speeds over 201 km/h (125 mph)? 4. Identify the following safety device? … 5. Define what a passive device is? Can anyone give me an example of a passive device? 6. What are three types of light rail alignments? 7. Why is aiming of roundels so critical? (p. 4)
Source: US Department of Transportation. (2004). Level II evaluation. Washington, DC: Author. Retrieved from https://www.nhi.fhwa.dot.gov/resources/ docs/Level%20II%20Evaluation%20Document.pdf
Consider This 1. Do you think this is a good way to evaluate trainees’ knowledge? Why or why not? 2. Do you think it is better to conduct this oral quiz in a group or individually. Explain your
reasoning. 3. What suggestions could you provide to improve the level 2 oral quizzes for the U.S.
Department of Transportation?
Kirkpatrick’s Four-Level Evaluation Framework Chapter 7
Level 3—Behavior: Did They Apply It?
A level 3 evaluation assesses the transfer of training; that is, do the participants of the train- ing program apply their new learning, transferring their skills from the training setting to the workplace, and as a result, did the training have a positive effect on job performance? Level 3 evaluations specifically focus on behavioral change via the transfer of knowledge, skills, and attitudes from the training context to the workplace.
However, before assessing skills transfer to the job, let us consider a practicality to the trans- fer of training evaluation: We must allow trainees a sufficient amount of time and opportu- nity to apply the training skills in the workplace (Piskurich, 2010). The amount of time will depend on numerous factors, including (ASTD, 2013; Cohen, 2005; Morrison et al., 2012; Noe, 2012; Wan, 2013):
• the nature of the training, • the opportunity available to implement the new KSAs, and • the level of encouragement from line management.
Typically, we can confirm transfer by observing the posttrained participants and conduct- ing work sampling (Kirkpatrick, 2009; Noe, 2012; Wan, 2013); evaluation can occur 90 days to 6 months posttraining (Kirkpatrick, 2009; Tobias & Fletcher, 2000). Figure 7.5 shows an example of level 3 training results.
Furthermore, as we will discuss in more detail in Chapter 8:
• positive transfer of training is demonstrated when we observe positive changes in KSAs, and
• negative transfer is evident when learning occurs, but we observe that KSAs are at less-than-pretraining levels (Noe, 2012; Roessingh, 2005; Underwood, 1966).
As discussed in Chapter 2, a trainee may have learned from the training but not be willing to apply the training to the workplace for several reasons. It may sound something like, “Oh, I know how to do it, but I am not doing it for you.” This is known as zero transfer of training, in which learning occurs, but we observe no changes in trainee KSAs. So, and perhaps not surprisingly, there is not a strong positive correlation between level 2 learning and level 3 behavior (Kirkpatrick & Basarab, 2011). That is, just because trainees learn something does not mean they will necessarily apply it. As discussed in previous chapters, irrespective of learning the new KSAs and being able to apply them to the workplace, the trainee must also be willing to apply them.
Level 4—Results: Did the Organization Benefit?
With a level 4 evaluation, the goal is to find out if the training program led to improved bottom- line organizational results (such as business profits). Similar to the correlation between levels 1 and 2, studies have shown a correlation between levels 3 and 4 (Kirkpatrick, 2009); specifically, if employees consistently perform critical on-the-job behaviors, individual and overall productivity increase.
Kirkpatrick’s Four-Level Evaluation Framework Chapter 7
Level 4 outcomes can include other major results that contribute to an organization’s effec- tive functioning. Level 4 outcomes are either changes in financial outcomes or changes in other metrics (for example, excellent customer service) that should indirectly affect financial outcomes at some point in the future; these are known as performance drivers (Swanson, 1995; Swanson & Holton, 2001). Here are some examples of level 4 performance drivers and outcomes (Cohen, 2005; Kirkpatrick, 2009; Phillips, 2003; Piskurich, 2010):
• Improved quality of work • Higher productivity • Reduction in turnover • Reduction in scrap rate • Improved quality of work life • Improved human relations • Increased sales • Fewer grievances • Lower absenteeism • Higher worker morale • Fewer accidents • Greater job satisfaction • Increased profits
Isolating the Effects of Training A major challenge to evaluation training’s effectiveness is isolating any subsequent perfor- mance improvement to the training itself. That is, improved performance may correspond to the timing of the training but may not be linked to new training itself. Phillips (2003) attributes this to the need for isolation. For example, Cohen (2005) described the following scenario:
Let’s say training was focused on new selling techniques for an organization’s sales reps and the post-training assessment of sales and call volume are found to be significantly better than the pre-training amounts; this change could be as much due to an upward turn in the economy as it is to the training itself. (p.23)
In this case linking the improvement to training would be incorrect, so we must protect against erroneously ascribing performance improvement to nontraining reasons. To mitigate this possibility, along with using pretests and posttests in level 2, Kirkpatrick (1959, 2009) also recommends using control groups to statistically manage and separate the impact of other variables. Control groups do not receive the training, or they go through other train- ing unrelated to the training of interest, so we can assess the unique effect of the training intervention. In Cohen’s example, a control group would include sales reps not subjected to the specific training program, and then the control group’s performance would be com- pared to the trained group (known as the experimental group) of sales reps (Cohen, 2005; Kirkpatrick, 1959; Kirkpatrick, 2009; Phillips, 2003; Piskurich, 2010).
Level 4 outcomes in particular may be difficult to isolate to the training program. This is because in order to assess any of the level 4 outcomes, more time must elapse to make a com- plete assessment. For example, an organization might have to wait 2 or 3 fiscal quarters to see
Kirkpatrick’s Four-Level Evaluation Framework Chapter 7
if decreased turnover or higher productivity follow training on those topics. As a result, by the time of assessment, other factors may have had a chance to affect the level 4 outcomes. This is what Sanders, Cogin, and Bainbridge (2013) called a confounding variable, or another fac- tor that obscures the effects or the impact of the training (Guerra-López, 2012). In sum, not unlike a 7-day weather forecast, a level 4 evaluation—although still valuable data—is usually more difficult to credit to the original training because it is the most removed from the train- ing event (Johnson & Christensen, 2010; Kirkpatrick, 2009; Sonnentag, 2003).
Linking Kirkpatrick Outcome Levels to the Performance Formula
Remember that in Chapter 2, we broke down workplace performance by understanding what components make up job performance; specifically, an outcome of three variables:
• Ability—the employee’s capacity to perform the job; collectively, their KSAs • Motivation—the employee’s willingness to perform the job voluntarily • Environment—anything within the organizational environment (such as the supervi-
sor, systems, and coworkers) that would affect the employee’s job performance
The Performance Formula
Performance = f(KSAs × M × E)
KSAs = Ability; M = Motivation; E = Environment
Using Kirkpatrick’s taxonomy (see Figure 7.5), we can see where summative outcomes are expressed within employee performance (Blanchard & Thacker, 2010; Mitchell, 1982).
Figure 7.5: Synthesizing Kirkpatrick and the performance formula
By synthesizing Kirkpatrick and the performance formula, we can illustrate a training’s impact not only on employee performance, but also on organizational performance in total.
f07.05_BUS375.ai
Level 3 Level 2
Learning Reaction Prior
Level 4
Level 1
Future state of Level 4
∑ Summation of
all trainees Level 3
Performance = f(KSAs × M × E)
4 Results
3 Transfer
2 Learning
1 Reactions
Performance = f(KSAs × M × E)
Kirkpatrick’s Four-Level Evaluation Framework Chapter 7
As Figure 7.5 shows, posttraining employee performance (level 3) is dependent on the effec- tiveness of both levels 1 and 2, reaction and learning. Specifically, the newly learned knowl- edge and skills are in level 2, learning, and the attitudes and motivation toward the new learning are in level 1, reaction. Importantly, posttrained performance is both contingent on and subsequently affects the organizational environment level 4 outcomes. Specifically, post- trained employee performance is subject to the antecedent state of the organizational envi- ronment (for example, the quality and state of the departmental supervision would affect the efficacy of the posttraining employee performance). However, it is also expected that the collective performance from the posttrained employee base would ultimately influence and affect the future state of the organizational environment and organizational outcomes and show itself in level 4 outcomes such as improved customer service, more efficient systems, and reduced error rates.
