Outcome Measurement - Quick Guide
Outcome Measurement - Introduction
As per the data shared by the American Society for Training & Development (ASTD), the companies in the US provide a minimum of 30 hours of training to their employees, which costs them roughly 40 dollars for each hour of employee training. The companies in the US alone end up spending a minimum of 165 billion dollars on providing various kinds of training. Can we prove whether any genuine learning has taken place from all this training? With such huge amounts of money being poured into this sector, it’s a question we no longer can afford to ignore.
In an economy where 17% of the population earns below 1000 dollars annually (counting students and undergrads), a training expense of around 1200 dollars per employee annually as training expenditures is a proper Comparison Chart to understand the investment organizations do in this sector. These numbers make it sufficiently clear that the companies, big or small, do invest large sums of money into training their employees. The issue crops up when we try to organize numbers for the other side of the spectrum, which make it difficult to give solid facts to prove whether any genuine learning has taken place.
What is Outcome Measurement?
There are countless ways in which the word outcome can be interpreted but it’s commonly agreed that Outcome Measurement is the act of assessing the impacts of a decision and check its impact.
Outcome measurement is a systematic method of observation and calculation to determine the success of a training program. The chief areas that it takes notes on are −
Has this program brought any difference to the behavior of the employee?
How are the performances of the employees better after the program?
What changes the program brought in the lives of employees, their families, and the organization?
Although there are many uses for the information generated by outcome measurement, managers often choose to ignore it because they don’t understand the holistic method to approach this process. They need to understand that the act of delivering training is just the means to an end, and not the end deliverable in itself.
The days of evaluating employees and their skill-sets based on tests with multiple-choice questions has now been discarded as a defective method. Such assessments can be easily cleared by employees using short-term retention of knowledge. The new-age testing methodologies require that employees cultivate a long-term ability to apply the knowledge they have gained from their training. The goal of the employers is to see a healthy Return on Investment on their training, which means a significant growth in the performance levels of the people.
Outcome Measurement is NOT Performance Measurement
During the last thirty years or so, most of the organizations were happy focusing on what their employees did, what staff they had, how many hours of work did they put in, which department to recruit more resources for, etc. And that was it.
What has changed now is the implementation of Outcome Measurement, which attempts to make a question out of the statements on performances mentioned above in the following manner −
Does the performance of the employees increase if they are given ten hours of technical assistance on job-related techniques? Is the organization able to save more time in doing the same work?
Does a strategic planning process help the organization save efforts and resources on getting the same workload done? If so, then is the benefit worth investing on in the long run?
Outcome Measurement is mostly mistaken with Performance Measurement. There is a difference in both: Performance Measurement is mostly used in business and profit-making ventures; whereas Outcome Measurement is used in non-profit organizations, where the stress is more on output, rather than revenue generation.
Measuring Training Outcomes
Can we pull out training measurement data, similar to the numbers invested in the training, and say the amount of improvement that employees have brought in their performances because of the training imparted to them?
Measuring training outcomes is fast becoming one of the most widely growing research areas in the world of business. Most organizations implement Outcome Measurement because they are asked to do so by the higher management.
In today’s competitive world, organizations are more interested in hiring employees who can deliver outcomes on the ground, rather than people who are termed good performers because they cleared some test. The doers are valued much higher than theoretical winners; business has become more result-oriented than ever.
Managers who don’t believe in measuring the output of training neglect one of the most important and fundamental functions in Business Administration, and become isolated from the many positives of generating Result Measurement Reports for the profitable functions of the organization. They need to know in periodic intervals if the education they are paying for to improve their employees’ skill-sets are bearing any fruit. The necessity to employ winners makes organizations hungry for outcomes.
Why Measure Outcomes?
The principal purpose of every organization is to bring a team of talented individuals together who will work at the peak of their abilities and produce quality output, in the form of products or services, for the organization which can then be catered to a market. While this expectation lies with every company, it’s only Outcome Measurement that can successfully determine whether these expectations are being successfully met.
When one invests in increasing the knowledge and talent of their workforce, it’s necessary to measure the success of their training as well.
It is important to have a method in which the success of the training can be tested in a holistic and organic manner, for example, by observing the effects of these training programs in their daily functioning in the workplace, instead of a memory-based test conducted at the end of the program.
With the information that you collect from such observations, you can be sure about which programs need to continue, how long they need to continue, and which programs need to be stopped right away without any further effort or resources invested in it.
The Benefits of Measuring Training Outcomes
Measuring Training Outcomes helps managers to first ask and then answer all the tough questions that they ask of themselves and their teams, such as −
- What are the difficulties and challenges of Employee Skill Development?
- How are you responsible for the development of your team?
- What developments in training are facing problems?
- How to initiate training measures?
Measuring Training Outcomes helps managers get a realistic assessment of the talent they have, so that they can either employ them in different assignments, or could change course and provide them with some other training that is more in tune with their skills. This evaluative approach to training was introduced to establish a relation between general training practices and employee development.
Outcome Measurement helps employers get a basic knowledge about Employee Development in organizations. It shows you a close view of the issues that employees face in training, problems in learning areas, human responsibility in training, and the success ratio of programs.
Measuring Outcomes - Factors
Sometimes, a perfectly fine program needs to be modified in order to get different outcomes. As all training programs are outcome-based, Outcome Measurement is one of the most important activities that organizations undertake. In this chapter, we will discuss some of the most important factors of Outcome Measurement.
Measuring the impact of training
Earlier, when the management found that a training is not effective, they used to either stop the training or replace the training team. The recent methods of Measuring Outcomes help identify the areas in which a program requires improvement. The Management has all the right to determine if the services they are paying for is delivering the desired result or not.
Identifying important practices
With the data that you have, now you have the knowledge of the activities that are being pursued. You have the ability to determine which actions can be continued and which can be built upon. With the information you collect, you can determine which activities to continue without any modifications, and which activities to change and replicate in some other form.
Identifying areas of improvement
Many times, a seemingly perfect training regime might fail to excite the trainees, for the simple reason that the participants might not feel any changes coming from the steps taken by the trainer. Also, some participants are forcibly sent to be a part of the training process and they feel there is no scope of improvement in their performances. In such cases, the training program might need slight adjustments.
Proving worth to existing and potential investors
The procedures of funding learning initiatives and Capacity Building has undergone radical changes in the past few years. With the recession and global slowdown of business, the investors are cautious about getting a healthy return on their investments in training their resources.
There has been a marked decline in behavioral changes and an increase in productivity-enhancing training. They are more focused on capacity building, as compared to promoting interpersonal skills. In such a climate, it’s imperative to reciprocate the trust the investors have on you.
Confirming Training Outcomes
The traditional training method asks the learners to demonstrate their gain from the learning in a simulation environment. With the help of technology, we can move even further now. Instead of a paper-based examination that asks us questions on areas that may or may not be relevant to our job profiles, we now have the option to ask trainees to complete a task in real life and show us if their outcomes have been a success.
Visual Confirmation
In Visual Confirmation, we take a real-life example as a model to determine our learning, which can be aided by visual confirmation through video conferencing. A training manager can now use these videos to access examples of real-life and explain different aspects of working to the trainees.
The managers can now tell how the training can be put to productive usage, that too by referring data procured from a real-time workshop. It not only changes how learning should be made measurable, but also teaches us what areas of the training are basically redundant and can be disposed with.
Social Ownership
The ability to have your concept so clear in any area that you can teach another person too, is often considered as the ultimate proof of your education. Social Learning can help learners educate people, who are at a newer learning stage than themselves, by teaching them how to implement their teachings in real world.
It also helps training managers in understanding which concepts work in the real world and which don’t, so that they can be provided a feasible solution to the issues that their teams face. These methods provide newer ways for employees to train and engage in an interactive manner.
Skill Assessments
A visual assessment of someone’s skill-set might take a longer time, because you have to wait for a scenario where the person has to use the skill-sets that he has learnt in training. That being said, it is the only evaluation method that can give a more accurate picture of an employee’s learning from the training, as opposed to some memory-based evaluation model which depends on asking questions out of a textbook.
The video engagements and real-life assessments help in giving a clear picture of the person’s achievements in the training, and the impact the training had on the organization’s efficiency. For example, a sales team could be observed for their performance before a training, and then they are given training, after which their skills sets are once again evaluated to check if there has been any real improvement.
Compliance Monitoring
Compliance Monitoring is a famous phrase today; it refers to the contractual obligations between an organization and its investors over making the use of the funds granted to them in specific areas of operations. Compliance monitoring is essential to keep a record of the amount of money it has spent in the services it has offered its clients and the training the workers and staff members.
Compliance monitoring helps us keep the programs running smoothly. Having said that, compliance monitoring might include the outcomes of the training as a part of the report. The term evaluation in the case of compliance monitoring is used to cover a large range of tasks, including studies where the steps taken for the training can be shown in relation to other factors. It is the arranged application of procedures used for research in gauging the planning, implementation and utility of the entire program.
Steps of Outcome Measurement
The objective of Outcome measurement is to check the veracity of the claims of the training program, and to see if the reported achievement has been really so. will explore what your program provides, what its intended impacts are, and whether or not it achieves them. It compares the final outcome with the promised outcome. What it doesn’t promise to check is whether the changes in the outcomes are because of the steps taken in the training or not.
For example, in a team of 20, there are 10 people who enroll for the training program, however a non-participating person might also get motivated seeing another worker showing interest in the training. As a result, the effective performance of the team will go up, however the training in itself had no effect on the improvement of the performance of that non-participating individual.
Many people get confused with the terminology outcome and want to understand it in a better way. To keep it simple, the term outcome encapsulates the three following terms −
Goal − the overall picture; the final outcome as an organic whole.
Outcome − changes in people, families, and organization after the training program.
Indicator − the specific information gathered to determine the occurrence of a genuine outcome.
Outcome Measurement - Limitations
Program managers, who are working on determining the outcomes of their teams after the training is over, need to remember that outcome measurement is not the objective of the training program, but it’s only an effective evaluation system.
The measurement of outcomes is only done to assist the staff working in programs, gain more access to information in exchange of continuous support and services. There are a few limitations to outcome measurement, which we will discuss in this chapter.
Limitation 1
Relationship building is an extremely complex process to measure. The relations that staff members build within the community and the teams is a critical result of activities taken by the organizations, however this can’t be measured. The hard numberoriented outcome measurement measuring systems might ignore these soft factors altogether.
Limitation 2
Depending on the kind of training and the process for which the training is taking place, the outcomes could take years to be properly evaluated. After all, Outcome Measurement is done on a real-time basis. So an ideal situation has to be in place to test the application abilities of the individuals.
Limitation 3
The outcome measurement process influences potential promotion-seeking workers to change their daily activities to score better points.
For example, a person who has been appointed as a Relationship Manager won’t be able to prove his/her improvement after the measurement process. So they will be more inclined to move towards such a measurable parameter, such as data collection, to prove their success.
Hence, outcome measurement should include factors that are measurable, along with ones that aren’t measurable in the immediate sense, but whose effect takes years to materialize.
Limitation 4
Outcome Measurements are done to evaluate the investment versus improvements in performance ratio. All data collected in these steps will tell us about an event that happened in the past. Hence, outcome measurement shouldn’t be used as an active component during decision-making processes, as decision-making involves the process of selecting actions that going to take place in the future.
Long-range planning for any desirable outcome is not an easy task, specifically because not all factors listed under “the parameters necessary to measure the outcomes” include simple cause and effect equations.
For example, the manager of a bank might successfully handle his customer, who in turn, gives a good word-of-mouth publicity about the manager and the bank, through which five new prospects open accounts with the bank. They might not necessarily open their accounts under the said manager, but the organization benefitted on the whole. However, this doesn’t necessarily percolate into the performance points for the manager; as far as he is concerned, his numbers will reflect one account opening only.
Outcome Measurement - Effective Practices
Outcome measurement should meet certain standards of evaluation. There should be some specific fields based on which performance should be assessed. The following four parameters are considered to be the most important ones to keep in mind while assessing an employee’s performance −
Utility − The information should serve the purpose of effective evaluation.
Practicability − The process should be implementable, realistic, and measurable.
Legality − The evaluators must resort to fair and legal means to collect data.
Accuracy − The outcomes of the measurement must be accurate and well-explained.
Without these four parameters, a training program won’t achieve the success it aimed for. In this chapter, we will discuss the steps that the evaluators need to keep in mind while taking their observations.
Begin with the outcome in mind
Evaluation planning starts when the project planning starts, so there should be no question left in the minds of the evaluators as to what the end deliverables for their organization are. Hence, a proper outlining of desired goals is essential to map the outcomes successfully.
Keep the investors in loop
The investors who have funded the training as well as the outcome measurement process are the ones who will decide the favorable outcomes and the definition of success of the training. Therefore, all the investors and the members of the Outcome Measurement team must be on the same page.
Keep working on the assessment
The assessment process provides a foundation based on which a further assessment can be done regarding the success achieved and the objectives realized. A thorough and correct assessment process will help the organization identify genuine progress in their outcomes.
Keep the measurement in context
Organizations operate in different circles and on different industries, hence there can never be a suitable model that can be used to evaluate the performances of employees in different organizations. To design a suitable Outcome Measurement System, the evaluators need to keep in context the industry the organization is operating in.
Other factors matter too, for example: if it’s a profit-making venture or a non-profit one, whether it’s situated in the rural areas or in an urban locality. Keeping these factors in context will not only keep the data collected relevant, but will also save time by keeping all the non-related calculations out.
Use the evaluation for learning
The outcomes of the Outcome Measurement should be used in the decision-making of future projects as well. Most organizations only use these outcome reports to gauge if the last training was successful or not. However, such important information should teach them what steps they ought to take so that they can either improve or replicate the success of the past training programs.
Areas to Accomplish
The outcomes of the training might have different meanings to different departments of an organization. For example, a person with improved communication skills at the cost of lower sales figures might be an improved performance in the eyes of the Marketing Team, but it won’t be the same emotions coming from the Sales Team.
Once the evaluation team decides what program they are going to choose to evaluate, they need to consider multiple parameters to measure the success percentage of any training program.
We have listed here some of the key parameters that the evaluation team should keep in mind −
Identifying the different definitions of success as per the individual departments, and how the teams are going to get that success. This step is generally taken while the planning of the training is done.
Determining if the expected success levels are feasible and deliverable at the end of the training program. Unrealistic expectations not only give a wrong picture of the training program but also make the entire measurement process futile.
Describing the process in which a program’s success is to be measured. The things also to keep in mind are the process of reporting a program’s success and whether it has met its desired success levels.
In addition to these steps, it’s also important to determine if a program has gotten proper and sufficient funding for all the necessary evaluation steps to be completed, so that the end report can be a comprehensive collection of all the important factors on the program-wide improvement effort.
Outcome Measurement - Data Collection
A data collection method needs to include all the factors, such as available sources, resources, activities done, delivered output, and final accomplishments. Data can be collected using methods such as surveys, interviews, observational sources, and record checking. We need to consider the pros and cons of data collection in the following fields −
Type of information needed − Some outcomes and indicators require a specific method to collect data, as compared to other cases where data can be collected in any method.
Validity and reliability − Some methods are more accurate in their outcomes than others.
Resources available − Resources include employees, time taken, and logistics.
Cultural correctness − Taking into the account the customers, beliefs and other sensitive details of the culture of people whom training is delivered.
Surveys
Surveys are standardized methods that can be used to collect information through mail or in person. The one area where surveys have a clear advantage over any other method of data collection is their cost ratio. You can engage a large number of people and collect data at a low cost.
Surveys, however, have their own share of disadvantages. Survey processes conducted over email are very time-consuming and have very low responses ratio, as people have the choice to opt out of it anytime they want. The slowness of responding to the surveys could also make the data irrelevant by the time it is completed. Any mechanisms employed to increase responses will add up to the cost.
The other big disadvantage is that surveys conducted over emails can also be misinterpreted, with no one to correctly explain the meaning of sentences where a reader gets any doubts. To minimize such incidents, a screening round of the email surveys in conducted where all possible doubts and ambiguous interpretations are taken into note, and corrective work is done to avoid confusions on the meanings of the questions.
Interviews
The advantage of conducting interviews over other methods of data collection is its standardization. Every person is asked the same set of questions and is evaluated based on the same parameters, even if they are conducted person-to-person or via telephonic medium.
In comparison to a written survey, an interview gives an opportunity to ask more complex and deeper questions, and the interviewer can also clarify his question and put it in a simpler way, in case the person doesn’t understand it. The participants can also engage in a broader discussion-based answering style.
Taking interviews to collect data can be very time consuming because one needs to dedicate a certain amount of time for each candidate. Hence, it becomes an expensive method of data collection.
Interviews may end up giving misleading information if the interviewers are not properly trained. They might lead the participant toward the right answers unknowingly, or could take an observation in a different meaning.
Focus Groups
It is a special type of interview conducted to get information in a particular field related the participants’ strengths and abilities. Although interviews are generally conducted to exact information from candidate without any influence from the interviewer, focus groups allow the participants to share their opinions, which sometimes leads to people influencing one another’s thoughts, and leading to debates.
The advantages of focus groups are that because of its debate-oriented nature, it triggers memory recollection and some very relevant and important information about the topic gets collected in a short period, which wouldn’t have been possible through interviews
The disadvantages of a focus group are that the outcomes of such debates or conversations rarely can be applied to the entire collection of participants, hence focus groups are often conducted after surveying a group of people to check for their concepts and beliefs.
Observation
Observations are often associated with the task of recording individual communication between people. These processes need well-trained observers and clear instructions on the process of observing, which includes whom to observe and for how long.
The biggest advantage of the method of observation is that it generates a big collection of information based on first-hand, unbiased information collected from participants. Observations, however, require a lot of time and more dedicated resource deployment than any other method, hence they become costlier as well.
Record Review
Record review is the process of collecting data from internal records of an organization, or from records of other groups, such as financial records, monthly reports, activity logs, purchase receipts, etc.
In this method, the process of collecting information becomes very easy, as the data is already compiled and available. It is a fast, economical, and efficient process of information gathering and saves people a lot of time from arranging and cataloguing data. In addition to that, just a few changes to the existing data collection model will provide us a new information system
Questionnaire for Data Collection
The following questionnaire is designed to finalize a data collection method that is the most suitable according to the needs of your organization. The questions asked here will guide you in finalizing the necessities of your outcome measurement system −
Surveys
1. Should the data be collected from the participant, client, or customers’ point of view?
a. Yes
b. No
2. Is there a method in which the information can be gathered from participant, client, or customers?
a. Yes
b. No
3. Does this data need to be standardized, in order to later make statistical comparisons?
a. Yes
b. No
4. Will participants face hurdles in understanding the survey questions because of their age and culture?
a. Yes
b. No
5. Are the participants educated enough to understand the information that the questions ask them about?
a. Yes
b. No
Conclusion − If your answer was “Yes” for all these above mentioned five questions, then your preferred method of data collection should be surveys.
Interviews
1. Are more detailed answers necessary to properly access information from the participants?
a. Yes
b. No
2. Will it be necessary to have a person-to-person conversation with the person, or a telephonic conversation to collect information, considering the participant’s age, cultural background, etc.?
a. Yes
b. No
Conclusion − If your answer was “Yes” for all these above mentioned two questions, then your preferred method of data collection should be conducting interviews.
Observation
1. Is it difficult to get an accurate depiction of the participant’s thoughts in this process?
a. Yes
b. No
2. Can someone trained to observe people’s behaviors be able to give accurate information about the participants?
a. Yes
b. No
3. Do the staff members know how to organize events, hold interactions, or observe for behaviors?
a. Yes
b. No
Conclusion − If your answer was “Yes” for all these above mentioned three questions, then your preferred method of data collection should be observation-based.
Internal Record Review
1. Does the process need you to access internal documentation and details?
a. Yes
b. No
2. If an information system does exist, is it being updated with newer data on a regular basis?
a. Yes
b. No
3. If a system like this does exist, can information be easily extracted from here?
a. Yes
b. No
Conclusion − If your answer was “Yes” for all these above mentioned three questions, then your preferred method of data collection should be Internal Record Review based.
Official Record Review
1. Are there official records that help you finalize the outcomes and indicators?
a. Yes
b. No
2. Are the official records accessible to you directly, or through intervention of some higher management guy?
a. Yes
b. No
Conclusion − If your answer was “Yes” for all these above mentioned two questions, then your preferred method of data collection should be Internal Record Review based.
Communicating the Outcomes
When a training manager designs a training program, he takes suggestions and inputs from the investors, fund-raisers and the organization before finalizing the objectives of the entire program. Once the training is over and the outcome measurement has been done, the details of this report need to be sent to the management. Three things that the training manager needs to know at this stage are −
- What to report
- How to report
- The target readers of the reports
Depending on the type of organization you are working with, there are numerous ways in which you can generate these reports. The following methods help you in sending the right report to the desired audiences −
Sending a formal report − At the end of the training, the manager should compile a complete report of evaluations, where he mentions the desired outcomes, and uses data collection plan to deliver the outcomes, and takes in recommendations for future assignments.
Providing case studies − Often used as formal reports, case studies are now used largely as marketing tools as well. The case study should discuss only one organization in detail and talk about the achievements of that organization only.
Issuing press releases − The press releases are often done to inform the outcomes of the training to the general public, the stakeholders, and the investors. The press releases contain the strongest learning points of the training.
Creating Postcards − Trying to distill the key outcomes in a postcard and putting it on display can be a great promotional material that can be distributed in meetings, get-togethers, etc.
Use Visual Aids − Using visual aids like motion pictures, graphs, diagrams can greatly enhance the presentation value of the discussion. However, the manager should exercise restraint and not over-indulge in visuals, or else the presentation will look extremely gimmicky and distracting.
In addition to all these above-mentioned methods, interviews are also arranged with the heads of the organizations to discuss how the capacity building exercises conducted during the training has helped improve the organization.
