UNIT8 DISC1 (ALSO SEE DISCUSSION 2 BELOW)
Using a Logic Model as a Strategic Planning Tool
In your readings for this week, Watson and Hoefer (2014) (PROVIDED BELOW)provide a general overview of using logic models to define a problem and identify inputs, activities, outputs, and outcomes.
In your initial post, discuss how application of a logic model would be different for a small nonprofit organization, a large federal agency, or the development of a policy for a specific human service program. How would the leader for each of these organizations engage the participants in the process? Cite examples from the reading.
READING: WATSON AND HOEFER 2014 FROM TEXT BOOK
Logic Models and Program Evaluation
Nonprofit administrators both develop and evaluate programs. A logic model is useful for both, even though development happens before the program begins and evaluation happens after it has been in operation. A good evaluation, however, is planned at the same time that the program is designed so that necessary data is collected along the way, rather than annually or after the program finishes. This chapter first describes the process of logic modeling using an example of the logic model. Then, it discusses how to use the logic model to plan an evaluation.
The idea of logic models as an adjunct to program evaluation extends at least as far back as 2000 when the Kellogg Foundation published a guide to developing logic models for program design and evaluation. According to Frechtling (2007), a logic model is “a tool that describes the theory of change underlying an intervention, product or policy” (p. 1). While one can find many variations on how a logic model should be constructed, it is a versatile tool that is used to design programs, assist in their implementation, and guide their evaluation. This chapter describes one basic approach to logic modeling for program evaluation and links the planning and evaluation aspects of human service administration.
You should understand that not all programs have been designed with the aid of a logic model, although that is becoming less common every year. Federal grants, for example, often require applicants to submit a logic model, and their use throughout the human services sector is growing through academic education and in-service training. If there is no logic model for a program you are working with, it is possible to create one after a program has been implemented. You can thus bring the power of the tool to bear when changing a program or creating an evaluation plan.
Logic model terminology uses system theory terminology. Because logic models are said to describe the program’s “theory of change,” it is possible to believe that this refers to something such as social learning theory, cognitive-behavioral theory, or any one of a number of psychological or sociological theories. In general, though, logic models have a much less grand view of theory. We begin with the assumption that any human services program is created to solve a problem. The problem should be clearly stated in a way that does not predetermine how the problem will be solved. The utility of a logic model is in showing how the resources used (inputs) are changed into a program (activities) with closely linked products (outputs) that then lead to changes in clients in the short, medium, and long terms. The net effect of these client changes is that the original problem is solved or at least made better for the clients in the program. An example of a logic model is shown as Figure 7.1.
The problem being addressed by the example program is, “School-aged youth have anger management problems leading to verbal and physical fights at school and home.” This problem statement is specific about who has a problem (school-aged youth), what the problem is (anger management problems leading to verbal and physical fights), and where it is a problem (school and home). It also does not prejudge what the solution is, allowing for many possible programs to address the problem. An example problem statement that is not as good because it states the problem in a way that allows only one solution is, “There is a lack of anger management classes in schools for school-aged youth.”
Another way to make the problem statement good is to phrase the statement in such a way that almost anyone can agree that it is actually a problem. The example problem statement might make this point more clearly by saying, “There are too many verbal and physical fights at school and home among school-aged youth.” Phrased this way, there would be little doubt that this is a problem, even though the statement is not specific about the number of such fights or the cause of the fights. If the program personnel want to focus on anger management problems, this way of stating the problem might lead to a host of other issues being addressed instead that might be leading to fights—such as overcrowding in the halls, gang membership, conflict over curfews at home, or anything else that might conceivably cause youth to fight at school or home. Be prepared to revisit your first effort at the problem statement and seek input from interested stakeholders to be sure that you are tackling what is really considered the reason for the program. The problem statement is vital to the rest of the logic model and evaluation so take the time to make several drafts to get full agreement.
After the problem statement, the logic model has six columns. Arrows connect what is written in one column to something else in the next column to the right or even within the same column. These arrows are the “logic” of the program. If the column to the left is achieved, then we believe that the element at the end of the arrow will be achieved. Each arrow can be considered to show a hypothesis that the two elements are linked. (The example presented here is intentionally not “perfect” so that you can see some of the nuances and challenges of using this tool.)
The first column is labeled “Inputs.” In this column, you write the major resources that will be needed or used in the program. Generically, these tend to be funds, staff, and space, but can include other elements such as type of funds, educational level of the staff, and location of the space (on a bus line, for example), if they apply to your program. The resource of “staff,” for example, might mean MSW-level licensed counselors. In the end, if only staff members with bachelor degrees in psychology are hired, this would indicate that the “staff” input was inadequate.
The second column is “Activities.” In this area, you write what the staff members of the program will be doing—what behaviors you would see them engage in if you sat and watched them. Here, as elsewhere in the logic model, there are decisions about the level of detail to include. It would be too detailed, for example, to have the following bullet points for the case management activity:
• Answer phone calls about clients
• Make phone calls about clients
• Learn about other agencies’ services
• Write out referral forms for clients to other agencies
This is what you would see, literally, but the phrase “case management” is probably enough. Somewhere in program documents, there should be a more detailed description of the duties of a case manager so that this level of detail is not necessary on the logic model, which is, after all, a graphical depiction of the program’s theory of change, not a daily to-do list.
The other danger is being too general. In this case, a phrase such as “provide social work services” wouldn’t be enough to help the viewer know what the employee is doing as there are so many activities involved in social work services. Getting the correct level of specificity is important in helping develop your evaluation plan here and throughout the logic model.
As you can see from the arrows leading from the inputs to the activities, the program theory indicates that, given the proper funds, staff, and space, the activities of case management and individual counseling will occur. This may or may not happen, however, which is why a process evaluation is needed and will be discussed later in this chapter.
The third column lists “Outputs.” An output is a measurable result of an activity. In this example, the activity of “case management” results in client youth being referred to other agencies for services. The output of the activity “individual counseling” is counseling sessions. It is important to note that outputs are not changes in clients—outputs are the results of agency activities that may or may not then result in changes to clients. The connection between agency activity and outputs is perhaps the most difficult part of putting together a logic model because many people mistakenly assume that if a service is given and documented, then client changes are automatic. This is simply not true.
The next three columns are collectively known as “Outcomes.” An outcome is a change in the client and should be written in a way that is a change in knowledge, attitude, belief, status, or behavior. Outcomes are why programs are developed and run—to change clients’ lives. Outcomes can be developed at any level of intervention—individual, couple or family, group, organization, or community of any size. This example uses a program designed to make a change at an individual youth level, but could also have changes at the school or district level if desired.
Outcomes are usually written to show a time dimension with short-, medium-, and long-term outcomes. The long-term outcome is the opposite of the problem stated at the top of the logic model and thus ties the entire intervention back to its purpose—to solve a particular problem. The division of outcomes into three distinct time periods is obviously a helpful fiction, not a tight description of reality. Still, some outcomes are expected to come sooner than others. These short-term outcomes are usually considered the direct result of outputs being developed. On the example logic model, the arrows indicate that referrals and individual counseling are both supposed to result in client youth better recognizing the role that anger plays in their life. After that is achieved, the program theory hypothesizes that clients will use skills at a beginning level to handle their anger. This is a case where one short-term outcome (change in self-knowledge) leads the way for a change in behavior (using skills).
OUTCOMES AND GOALS AND OBJECTIVES: WHAT’S THE DIFFERENCE?
Logic models use the term outcome, but many people use the terms goals and objectives to talk about what a program is trying to achieve. In the previous chapter, you were told that an outcome objective answers the question, “What difference did it make in the lives of the people served?” In this chapter, you are told that an outcome is a “change in the client.” What’s the difference?
In reality, there is not much difference. Goals and objectives are one way of talking about the purpose of a program. This terminology is older than the logic model terminology and more widespread. But it can be confusing, too, because an objective at one level of an organization may be considered a goal at another level or at a different time.
Outcomes are easier to fit into the logic model approach to showing program theory by relating to resources, activities, and outputs. Systems theory terminology is more widespread than before and avoids some of the conceptual pitfalls of goals and objectives thinking.
We present both sets of terms so that you can be comfortable in all settings. But you should realize that both approaches are ultimately talking about the same thing: the ability of an organization to make people’s lives better.
The element “beginning level use of skills to handle anger” has two arrows leading to medium-term outcomes. The first arrow leads to “higher level use of skills to handle anger.” In this theory of change, at this point, there is still anger, but the youth recognize what is occurring and take measures to handle it in a skillful way that does not lead to negative consequences. The second arrow from “beginning level use of skills to handle anger” indicates that the program designers believe that the skills youth learn will assist them to reframe situations they are in so that they feel angry less frequently. This is a separate behavior than applying skills to handle anger, so it receives its own arrow and box.
The final column represents the long-term outcomes. Often, there is only one element shown in this column, one indicating the opposite of the problem. In this logic model, since the problem is seen to occur both at school and at home, each is looked at separately. A youth may reduce fights at home but not at school, or vice versa, so it is important to leave open the possibility of only partial success.
This example logic model shows a relatively simple program theory, with two separate tracks for intervention but with overlapping outcomes expected from the two intervention methods. It indicates how one element can lead to more than one “next step” and how different elements can lead to the same outcome. Finally, while it is not necessarily obvious just yet, this example shows some weak points in the program’s logic that will emerge when we use it as a guide to evaluating the program.
As you can see from this discussion, we have used a logic model to represent what we believe will happen when the proper inputs are applied to the correct client population. In the end, if all goes well, clients will no longer have the problem the program addresses, or at least the degree or extent of the problem will be less.
Evaluation is a way to determine the worth or value of a program (Rossi, Lipsey, & Freeman, 2003). There are two primary types of evaluation: process and outcome. The first, process evaluation, examines the way a program runs. In essence, a process evaluation examines the first three columns of a logic model to determine whether required inputs were available, the extent to which activities were conducted, and the degree of output accomplishment. Another aspect of a process evaluation, called fidelity assessment, examines whether the program being evaluated was conducted in accord with the way the program was supposed to be conducted. If all components of a program are completed, fidelity is said to be high. Particularly with evidence-based and manualized programs, if changes are made to the program model during implementation, the program’s effectiveness is likely to be diminished.
The value of the logic model for evaluation is that most of the conceptual information needed to design the evaluation of a program is in the logic model. The required inputs are listed, and the evaluator can check to determine which resources actually came into the program. Activities are similarly delineated, and an evaluator can usually find a way to count the number of activities that the program completed. Similarly, the logic model describes what outputs are expected, and the evaluator merely has to determine how to count the number of completed outputs that result from the program activities.
Looking at the example logic model shows us that we want to have in our evaluation plan at least one way to measure whether funding, staff, and space (the inputs) are adequate; how much case management occurred and individual counseling was conducted (the activities); and the extent to which referrals were made (and followed up on) and the number of individual counseling sessions that happened (the outputs). This information should be in program documents to compare what was planned for with what was actually provided. Having a logic model from the beginning allows the evaluator to ensure that proper data are being collected from the program’s start, rather than scrambling later to answer some of these basic questions.
As noted earlier, this is not a perfect logic model. The question in the process evaluation at this stage might be to determine how to actually measure “case management.” The output is supposed to be “referrals to other agencies,” but there is much else that could be considered beneficial from a case management approach. This element may need careful delineation and discussion with stakeholders to ascertain exactly what is important about case management that should be measured.
The second primary type of evaluation examines program outcomes. Called an outcome evaluation, it focuses on the right half of the logic model, where the designated short-, medium-, and long-term outcomes are listed. The evaluator chooses which outcomes to assess from among the various outcomes in the logic model. Decisions need to be made about how to measure the outcomes, but the logic model provides a quick list of what to measure. In the example logic model, the short-term outcome “better recognition of the role anger plays in their lives” must be measured and could be accomplished using a set of questions asked at intake into the program and after some time has passed after receiving services. One standardized anger management instrument is called the “Anger Management Scale” (Stith & Hamby, 2002). A standardized instrument, if it is appropriate for the clients and program, is a good choice because you can find norms, or expected responses, to the items on the instrument. It is helpful to you, as the evaluator, to know what “average” responses are so you can compare your clients’ responses to the norms. Sometimes, however, it can be difficult to find a standardized instrument that is fully appropriate and relevant to your program.
Another way of measuring is to use an instrument you make up yourself. This has the advantage of simplicity and of being directly connected to your evaluation. In this case, for example, you could approach this outcome in at least two ways. First, you could request a statement from the case worker or counselor indicating that the client has “recognized the role that anger plays” in his or her life, without going into any detail. A second approach would be to have the client write a statement about the role anger plays in his or her life. Neither of these measurements will have a lot of practical utility. Going through the logic model in this way actually shows that this link in program logic is difficult to measure and may not be totally necessary.
WHAT IS AN UNANTICIPATED OUTCOME?
Outcome evaluations also sometimes include a search for unanticipated outcomes. An unanticipated outcome is a change in clients or the environment that occurs because of the program, intervention, or policy, but that was not thought would result and so is not included in the logic model.
While it may seem startling to have an example in a text that shows a less-than-perfect approach, it is included here to show that using a logic model is very useful in showing weak spots in the program logic. This link to “better recognition” is not a fatal problem, and may indeed be an important cognitive change for the client. The issue for evaluation is how to measure it, and whether it really needs to be measured at all.
Of more importance is the next link, which leads to “learn skills to handle anger.” The evaluation must ensure that clients understand skills to help them handle anger and so document these skills. It is not enough to indicate that skills were taught, as in a group or individual session. Teaching a class is an activity and so would be documented in the process evaluation portion of the overall evaluation, but being in a class does not guarantee a change in the client. In this evaluation, we would like to have a measure of skill that can show improvement in the ability to perform the anger management skill. This attribute of the measure is important because we expect the clients to get better in their use over time and include more skillful use of the techniques as a medium-term outcome in the logic model.
The other medium-term outcome expected is that clients will be able to reframe situations so that they actually get angry less frequently. The program logic shows this outcome occurring as a result of both beginning and higher level use of skills. Because this element is broken out from the use of skills to “handle anger,” it will need a separate measure. As an evaluator, you can hope that an established, normed instrument is available, or that this is a skill that is measured by a separate item on a longer scale. If not, you will need to find a way to pull this information from staff members’ reports or client self-assessments.
The final links in the logic model connect the medium-term outcomes to the long-term outcomes of fewer fights at school and fewer fights at home. Because youth having too many fights was identified as the problem this program is addressing, we want to know to what degree fights decreased. The measure here could be client self-reports, school records, or reports from people living in the home.
Implicit in the discussion of the use of this logic model for evaluation purposes is that measurements at the end will be compared to an earlier measure of the same outcome. This is called a single group pretest-posttest evaluation (or research) design. It is not considered a strong design due to the ability of other forces (threats to internal validity) to affect the results. The design could be stronger if a comparison group of similar youth (perhaps at a different school) were chosen and tracked with the same measures. The design could be much stronger if youth at the same school were randomly assigned to either a group that received the program or a different group that did not receive the program. It is beyond the scope of this book to cover in detail all the intricacies of measurement and evaluation design, but we hope this brief overview whets your appetite to learning more.
Measurement of outcomes, while alluded to earlier, is an important part of any evaluation effort. If measures are not appropriate or have low validity and reliability, the value of the evaluation will be seriously compromised. It is suggested that anyone designing an evaluation look at a book on research methods such as Rubin and Babbie (2012), and also have access to books about measures, such as Fischer and Corcoran (2007). (The cost of a new book on research methods may be pretty high, but used editions contain much the same information and can be found for much lower prices.)
Using an example, this chapter has covered the components of a logic model and how to develop one. It also demonstrates how to use a logic model to design an evaluation plan, including how it raises issues of comprehending program logic, measurement, and evaluation design.
In the article by Weber (2015), there is a statement about the importance of engaging those who will be impacted by change in the process of implementing change. The article further relates that initial efforts to foster change in the organization’s strategic planning process were unsuccessful in part because of lack of engagement. Resistance to change and lack of trust are common experiences when an organization or community engages in change efforts.
In your initial post, discuss the principles for dealing with opposition or resistance in change efforts for the Riverbend City Strategic Planning Committee(INSERTED BELOW), as described in the media piece in the Studies for this unit. Is resistance a normal part of the process? Can it be used in the effort to promote change? How? Discuss the resistance presented by one of the members of the Riverbend City Strategic Planning Committee and critique the response or method of addressing the resistance that was used by the group leader.
RIVERBEND CITY TRANSCRIPT
In this scenario, you will resume your role as a case worker at the Riverbend City Boys’ and Girls’ Club.
The club is partnering with a number of other local organizations for the Northside Youth Rising Initiative, aimed at preventing youth violence and keeping kids in school in Riverbend City’s underprivileged communities. To kick off the initiative and raise funds for it, the associated organizations are planning the Northside Blues Blowout, a large concert. You have been tasked to be the Boys’ and Girls’ Club’s point person on the Northside Blowout planning.
The director of the Boys’ and Girls’ Club, Jayme Young, has convened a meeting with leaders from some of the partner organizations to talk about logistics for the Northside Blues Blowout.
Riverbend Services Consortium Rep
OK, before I start being negative, I do want to make clear: I’m very excited about the Initiative in general and the Northside Blues Blowout in particular. Really! And so is everyone I’ve talked to at the Riverbend Services Consortium!
That said, I’m really nervous about some of our nuts and bolts as we approach the Blowout. As a Human Services nonprofit, we were really excited by the idea of the initiative, and I pushed hard for RSC to get involved. My staff was enthusiastic, my board much less so. But they came along. This was before we’d all made the decision to kick things off with the Blowout. Now I find myself in choppy water. I’m empowered by the RSC board to spend the organization’s money for Northside Youth Rising Initiative program activities. But that’s it. My board really, really doesn’t like the idea of our funds being spent on a concert, even when I explain to them again and again that the concert is itself a fundraiser, and this is a case of investing money to make money.
Since all of this is outside of the budget we’d already approved for the year, every expenditure that RSC makes for this is going to have to be approved by our board, and they’re all convinced that I’m asking them to give me a pile of RSC’s money so that I can light it on fire. One of our board members left me a really angry voicemail last night, asking me how she’s supposed to be asking donors for contributions when those contributions are just going to be turned around to pay for music. I know, I know. She’s old-fashioned, but she’s been a great friend to RSC and she’s donated and solicited a lot of money for us. I’m not sure where this leaves me to be honest. I’m trying to figure it out. At the very least, as we work out the budget for the Blowout and figure out how much everyone’s organization needs to kick in, I’m going to need a hell of an airtight case for every line item I take back to them.
Father Junot Rivera
St. Francis Borgia Catholic Church
I also have some qualms about the Blues Blowout. I don’t have a nonprofit board of directors to answer to, but I do have a parish with a healthy mix of very progressive, forward-thinking people and some people who, though wonderful, are not as accepting of new and different things. And I have to respect the views of everyone in my parish.
Everyone at St. Francis Borgia loves the idea of our community getting involved with the Northside Youth Rising Initiative. Everyone. People ask me about it after mass every single week, and I always enjoy talking to them about it. But the Blowout has some of my parishioners very concerned. There’s a very strong Legion of Decency in the parish, and its members don’t want parish funds going to a concert—even one in the service of good works— with performers they consider immoral. This isn’t a concern that I personally have; but like I said, I have to respect the views of my parishioners.
They’ve asked me to ask if a member of the parish can sit on the subcommittee that chooses the acts for the Blowout; they’ve also asked if I can arrange for them to have veto power. I know we haven’t even started talking about how we’re going to choose our acts, and I know that individual veto power probably isn’t a reasonable request. But I wanted to put it out there that these are conditions I need to work within as we move forward.
Franzen is excited to be partnering with all of these great, great organizations for this important initiative. We’re thrilled!
Now, I’m afraid there’ll be a little bit of a delicate dance when it comes to our participation in the Blues Blowout. I know that you all think it’s important that we kick this thing off in style, and you’re all convinced that this’ll be a big fundraiser. I hate to be a broken record, but I have to say that I’m just not convinced. This isn’t my first time at the rodeo, I’ve just been involved in way too many fundraising galas and what have you where everyone thought they were going to raise a fortune but instead just wound up breaking even. It’s just such an easy trap to fall into! Everyone says no, it’s not going to happen this time, we’ve learned all the lessons, and then BAM! You’re giving tickets away because they didn’t sell and you just hope people buy enough drinks to cover the costs of putting the thing on.
Now, I know I’ve already lost the argument on having the Blowout. And that’s OK. But I’d like to explore the possibility of having Franzen’s contribution to the Blowout being just of the in-kind nature. We can have staff work the event, we can put our marketing department at the disposal of the committee to help publicize it, stuff like that. But we’re having a very lean year, and what funds we do have available, well, I’d like to keep them earmarked for the Initiative’s actual program activities. Look at it this way: if I’m right and the Blowout doesn’t make any money, this leaves some operating cash from Franzen InterTech on the table for the Initiative to use.
I understand the logistical difficulties everyone is worried about, and I hope we can figure out ways through all of them. I think we will! We’d better, because I have to tell you, the Northside Youth Rising Initiative is a big, big deal, and it can’t happen without the Blues Blowout, and so we’d darned well better figure all of this out.
Crandall’s had a plant on the north side of Riverbend City for a long, long time, and we’ve got deep roots in that part of town, a lot of those people work in our factory. And it just breaks my heart, seeing the troubles that people have. I know that Hispanics really value family, and they just don’t do well without a good family structure. But you get these deadbeat dads, these absentee fathers, and it just cripples a whole generation. We see that in our plants, let me tell you. Thirty years ago, Hispanics living on the north side were some of our best employees. Now we have all these problems. We’ve got to get the Youth Rising Initiative going to help them out, and if you ask me we need to change the program a little bit so that we’re doing more about these absent fathers.
Sorry, I get carried away. Anyway, if other partner orgs are having trouble with finances for the Blues Blowout, Crandall can probably adjust our contribution.
Director of Outreach, Riverbend City Schools
Hate to say it, but if we’re talking about the nuts and bolts of putting on the Blues Blowout, I’m another person bringing some complications to the table. The school district is ecstatic about the initiative overall— I mean, how could we not be? And I think the Blues Blowout sounds like a great event both to raise the profile of the initiative and to let the community have some fun. I’m looking forward to being there and rocking out.
But here’s the bad news: the district operates under some pretty broad ethical guidelines that got put into place after some stuff that happened in the 90s. And those guidelines weren’t really worded as well as they could be, but they’re still binding. Long story short, district funds can’t be used to procure food or drink for outside events. I know, I know. We can negotiate for food donations or gifts in kind by vendors that the district already has a relationship with, so that might be a way to deal with this.
Hennsey County Juvenile Justice System
I know one way we can save a little bit of money on the Blues Blowout! My little brother has a band, and I swear, they’re really good. You’d be amazed! I know I am. They play a couple of times a month at the Bunker downtown, also Whiskey Lane out in Hoskins. They do a bunch of great classic rock songs, some of their own material, too. They always drive the crowds nuts, get people dancing like crazy. I think they’d be great for the Blowout! They’re called Spoilsport.
We haven’t really talked about what our budget is for talent, but I know that my brother would talk the rest of Spoilsport into playing for a really low fee. I bet we could get them for $2000. That’s less than they make at a weekend at Whiskey Lane, I can tell you that! I think this’d be a really exciting way to stretch our event dollars-planning and get the show some really exciting top-level entertainment.
It’s funny, my brother always gives me guff for my boring job in juvenile justice; I’m pretty excited to have him help us bring some rock and roll to it. What do you guys think?