Vocabulary

• Evaluators: People who judge or assess the quality or effectiveness of something, like a public health program or intervention.
• Workgroup: A small team of people who work together on specific tasks or projects related to public health topics, like preventing diseases or educating communities.
• Sustainability: The ability of a public health program or intervention to continue functioning effectively over time, often focusing on long-term impact and resource management.
• Stakeholders: People or groups with an interest or concern in a public health issue or program, including community members, policymakers, healthcare providers, and advocacy organizations.
• Feasibility: The likelihood or practicality of successfully implementing a public health intervention or program, considering factors like resources, time, and community support.
• Prioritization: The process of deciding which public health issues or interventions are most important or urgent, often based on factors like severity, impact, and available resources. 
• Downstream: Referring to the later stages or consequences of a public health issue or intervention, such as the impact on individuals, communities, or healthcare systems over time.

(Centers for Disease Control and Prevention, 2011)

Who is the Audience for this Workbook?

The purpose of this workbook is to help public health program managers, administrators, and evaluators develop a joint understanding of what constitutes an evaluation plan, why it is important, and how to develop an effective evaluation plan in the context of the planning process. This workbook assists in developing an evaluation plan; it is not a complete resource on how to implement program evaluation. Use it with other evaluation resources, such as those listed in the Resource Section of this workbook. The workbook was written by the staff of the Office on Smoking and Health (OSH) and the Division of Nutrition, Physical Activity, and Obesity (DNPAO) at the Centers for Disease Control and Prevention (CDC). However, the content and steps for writing an evaluation plan can be applied to any public health program or initiative. Part 1 of this workbook defines and describes how to write an effective evaluation plan. Part 2 of this workbook includes exercises, worksheets, tools, and a Resource Section to facilitate program staff and evaluation stakeholder workgroup (ESW) thinking through the concepts presented in Part 1 of this workbook. 

What is an Evaluation Plan?

An evaluation plan is a written document that describes how you will monitor and evaluate your program and how you intend to use evaluation results for program improvement and decision-making. The evaluation plan clarifies how you will describe the What, the How, and the Why It Matters for your program. 

• The What reflects the description of your program and how its activities are linked with the intended effects. It serves to clarify the program’s purpose and anticipated outcomes. 

• The How addresses the process for implementing a program and provides information about whether the program is operating with fidelity to the program’s design. Additionally, the How (or process evaluation), along with output and/or short-term outcome information, helps clarify if changes should be made during implementation. An evaluation plan is a written document that describes how you will monitor and evaluate your program so that you can describe the What, the How, and the Why It Matters for your program and use evaluation results for program improvement and decision-making. 

• The Why It Matters provides the rationale for your program and the impact it has on public health. This is also sometimes referred to as the so what question. Being able to demonstrate that your program has made a difference is critical to program sustainability. 

An evaluation plan is similar to a roadmap. It clarifies the steps needed to assess the processes and outcomes of a program. An effective evaluation plan is more than a column of indicators added to your program’s work plan. It is a dynamic tool (for example, a living document) that should be updated on an ongoing basis to reflect program changes and priorities over time. An evaluation plan serves as a bridge between evaluation and program planning by highlighting program goals, clarifying measurable program objectives, and linking program activities with intended outcomes. 

Why Do You Want an Evaluation Plan?

Just as using a roadmap facilitates progress on a long journey, an evaluation plan can clarify what direction your evaluation should take based on priorities, resources, time, and skills needed to accomplish the evaluation. The process of developing an evaluation plan in cooperation with an evaluation workgroup of stakeholders will foster collaboration and a sense of shared purpose. Having a written evaluation plan will foster transparency and ensure that stakeholders are on the same page with regards to the purpose, use, and users of the evaluation results. Moreover, use of evaluation results is not something that can be hoped or wished for but must be planned, directed, and intentional (Patton, 2008). A written plan is one of your most effective tools in your evaluation tool box. A written evaluation plan can do the following: 

• Create a shared understanding of the purpose(s), use, and users of the evaluation results. 

• Foster program transparency to stakeholders and decision makers, 

• Increase buy-in and acceptance of methods. 

• Connect multiple evaluation activities—this is especially useful when a program employs different contractors or contracts. 

• Serve as an advocacy tool for evaluation resources based on negotiated priorities and established stakeholder and decision maker information needs. 

• Help to identify whether there are sufficient program resources and time to accomplish desired evaluation activities and answer prioritized evaluation questions. 

• Assist in facilitating a smoother transition when there is staff turnover. 

• Facilitate evaluation capacity building among partners and stakeholders.

• Provide a multi-year comprehensive document that makes explicit everything from stakeholders to dissemination to use of results.

• Facilitate good evaluation practice. 

There are several critical elements needed to ensure that your evaluation plan lives up to its potential. These elements include ensuring the following:

• Your plan is collaboratively developed with a stakeholder workgroup. 

• It is responsive to program changes and priorities. 

• It covers multiple years if your project is ongoing. 

• It addresses your entire program rather than focusing on just one funding source, objective, or activity. 

You will, by necessity, focus the evaluation based on feasibility, stage of development, ability to consume information, and other priorities that will be discussed in Steps 3 and 4 in this workbook. However, during the planning phase, your entire program should be considered by the evaluation group.

How Do You Write an Evaluation Plan?

This workbook is organized by describing the elements of the evaluation plan within the context of using the CDC’s Framework for Program Evaluation in Public Health and the planning process. The following elements of an evaluation plan will be discussed in this workbook:

• Title page: Contains an easily identifiable program name, dates covered, and basic focus of the evaluation.

• Intended use and users: Fosters transparency about the purpose(s) of the evaluation and identifies who will have access to evaluation results. It is important to build a market for evaluation results from the beginning. Clarifying the primary intended users, the members of the stakeholder evaluation workgroup, and the purpose(s) of the evaluation will help to build this market. 

• Program description: Provides the opportunity for building a shared understanding of the theory of change driving the program. This section often includes a logic model and a description of the stage of development of the program in addition to a narrative description. 

• Evaluation focus: Provides the opportunity to document how the evaluation focus will be narrowed and the rationale for the prioritization process. Given that there are never enough resources or time to answer every evaluation question, it is critical to work collaboratively to prioritize the evaluation based on a shared understanding of the theory of change identified in the logic model, the stage of development of the program, the intended uses of the evaluation, as well as feasibility issues. This section should delineate the criteria for evaluation prioritization and include a discussion of feasibility and efficiency. 

• Methods: Identifies evaluation indicators and performance measures, data sources and methods, as well as roles and responsibilities. This section provides a clear description of how the evaluation will be implemented to ensure credibility of evaluation information. 

• Analysis and interpretation plan: Clarifies how information will be analyzed and describes the process for interpretation of results. This section describes who will get to see interim results, whether there will be a stakeholder interpretation meeting or meetings, and methods that will be used to analyze the data.  

• Use, dissemination, and sharing plan: Describes plans for use of evaluation results and dissemination of evaluation findings. Clear, specific plans for evaluation use should be discussed from the beginning. This section should include a broad overview of how findings are to be used as well as more detailed information about the intended modes and methods for sharing results with stakeholders. This is a critical but often neglected section of the evaluation plan. 

What Are the Key Steps in Developing an Evaluation Plan Using the CDC’s Framework for Program Evaluation? 

CDC’s Framework for Program Evaluation in Public Health (1999) is a guide to effectively evaluate public health programs and use the findings for program improvement and decision-making. While the framework is described in terms of steps, the actions are not always linear and are often completed in a back-and-forth effort that is cyclical in nature. Similar to the framework, the development of an evaluation plan is an ongoing process. You may need to revisit a step during the process and complete other discrete steps concurrently. Within each step of the framework, there are important components that are useful to consider in the creation of an evaluation plan. 

Figure 1: CDC Framework for Program Evaluation in Public Health 

For a description of the image, access the appendix

Steps: 

1. Engage stakeholders 

2. Describe the program 

3. Focus the evaluation design 

4. Gather credible evidence 

5. Justify conclusions 

6. Ensure use and share lessons learned 

In addition to CDC’s Framework for Program Evaluation in Public Health, there are evaluation standards that will enhance the quality of evaluations by guarding against potential mistakes or errors in practice. As indicated by the inner circle in Figure 1, the evaluation standards are grouped around four important attributes: utility, feasibility, propriety, and accuracy.

• Utility: Serve information needs of intended users. 

• Feasibility: Be realistic, prudent, diplomatic, and frugal.

• Propriety: Behave legally, ethically, and with due regard for the welfare of those involved and those affected. 

• Accuracy: Evaluation is comprehensive and grounded in the data. 

(The Joint Committee on Standards for Educational Evaluation, 1994)

Step 1: Engage Stakeholders

(Centers for Disease Control and Prevention, 2011)

Defining the Purpose in the Plan

Identifying the purpose of the evaluation is equally as important as identifying the end users or stakeholders who will be part of a consultative group. These two aspects of the evaluation serve as a foundation for evaluation planning, focus, design, and interpretation and use of results. The purpose of an evaluation influences the identification of stakeholders for the evaluation, selection of specific evaluation questions, and the timing of evaluation activities. It is critical that the program is transparent about intended purposes of the evaluation. If evaluation results will be used to determine whether a program should be continued or eliminated, stakeholders should know this up front. The stated purpose of the evaluation drives the expectations and sets the boundaries for what the evaluation can and cannot deliver. In any single evaluation, and especially in a multi-year plan, more than one purpose may be identified; however, the primary purpose can influence resource allocation, use, stakeholders included, and more. Purpose priorities in the plan can help establish the link between purposes and intended use of evaluation information. While there are many ways of stating the identified purpose(s) of the evaluation, they generally fall into three primary categories:

1. Rendering judgments: Accountability

2. Facilitating improvements: Program development

3. Knowledge generation: Transferability

(Patton, 2008)

A primary feature of an evaluation plan is the identification of an ESW, which includes members who have a stake or vested interest in the evaluation findings, those who are the intended users who can most directly benefit from the evaluation (Patton, 2008; Knowlton, Philips, 2009), as well as others who have a direct or indirect interest in program implementation. Engaging stakeholders in the ESW enhances intended users’ understanding and acceptance of the utility of evaluation information. Stakeholders are much more likely to buy into and support the evaluation if they are involved in the evaluation process from the beginning. Moreover, to ensure that the information collected, analyzed, and reported successfully meets the needs of the program and stakeholders, it is best to work with the people who will be using this information throughout the entire process. Engaging stakeholders in an evaluation can have many benefits. In general, stakeholders include people who will use the evaluation results, support or maintain the program, or who are affected by the program activities or evaluation results. Stakeholders can help in the following ways:

• Determine and prioritize key evaluation questions.

• Pretest data collection instruments.

• Facilitate data collection.

• Implement evaluation activities.

• Increase credibility of analysis and interpretation of evaluation information.

• Ensure evaluation results are used.

Several questions pertaining to stakeholders may arise among program staff, including the following questions:

• Who are the program’s stakeholders?

• How can we work with all of our stakeholders?

• How are stakeholders’ role(s) described in the plan?

This section will help programs address these and other questions about stakeholders and their roles in the evaluation to guide them in writing an effective evaluation plan.

Who Are the Program's Stakeholders?

The first question to answer when the program writes its evaluation plan is to decide which stakeholders to include. Stakeholders are consumers of the evaluation results. As consumers, they will have a vested interest in the results of the evaluation. In general, stakeholders are those who are interested in the program and would use evaluation results, such as clients, community groups, and elected officials; those who are involved in running the program, such as program staff, partners, management, the funding source, and coalition members; and those who are served by the program, their families, or the general public. Others may also be included as these categories are not exclusive.

How Do You Use an ESW to Develop an Evaluation Plan?

It is often said of public health programs that everyone is your stakeholder. Stakeholders will often have diverse and, at times, competing interests. Given that a single evaluation cannot answer all possible evaluation questions raised by diverse groups, it will be critical that the prioritization process is outlined in the evaluation plan and that the stakeholder groups represented are identified.

It is suggested that the program enlists the aid of an ESW of 8–10 members that represents the stakeholders who have the greatest stake or vested interest in the evaluation (Centers for Disease Control, 2008). These stakeholders, or primary intended users, will serve in a consultative role on all phases of the evaluation. As members of the ESW, they will be an integral part of the entire evaluation process from the initial design phase to interpretation, dissemination, and ensuring use. Stakeholders will play a major role in the program’s evaluation, including consultation and possibly even data collection, interpretation, and decision-making based on the evaluation results. Sometimes stakeholders can have competing interests that may come to light in the evaluation planning process. It is important to explore agendas in the beginning and come to a shared understanding of roles and responsibilities, as well as the purposes of the evaluation. It is important that both the program and the ESW understand and agree to the importance and role of the workgroup in this process.

In order to meaningfully engage your stakeholders, you will need to allow time for resolving conflicts and coming to a shared understanding of the program and evaluation. However, the time is worth the effort and leads toward a truly participatory, empowering approach to evaluation.

How Are Stakeholders’ Roles Described in the Plan?

It is important to document information within your written evaluation plan based on the context of your program. For the ESW to be truly integrated into the process, ideally, they will be identified in the evaluation plan. The form this takes may vary based on program needs. If it is important politically, a program might want to specifically name each member of the workgroup, their affiliation, and specific role(s) on the workgroup. If a workgroup is designed with rotating membership by group, then the program might just list the groups represented. For example, a program might have a workgroup that is comprised of members that represent funded programs (three members), non-funded programs (one member), and national partners (four members) or a workgroup that is comprised of members that represent state programs (two members), community programs (five members), and external evaluation expertise (two members). Being transparent about the role and purpose of the ESW can facilitate buy-in for evaluation results from those who did not participate in the evaluation—especially in situations where the evaluation is implemented by internal staff members. Another by-product of workgroup membership is that stakeholders and partners increase their capacity for evaluation activities and increase their ability to be savvy consumers of evaluation information. This can have downstream impacts on stakeholders’ and partners’ programs such as program improvement and timely, informed decision-making. A stakeholder inclusion chart or table can be a useful tool to include in your evaluation plan.

The process for stakeholder engagement should also be described in other steps related to the development of the evaluation plan, which may include the following steps:

Step 2: Describe the program. A shared understanding of the program and what the evaluation can and cannot deliver is essential to the success of implementation of evaluation activities and use of evaluation results. The program and stakeholders must agree upon the logic model, stage of development description, and purpose(s) of the evaluation.

Step 3: Focus the evaluation. Understanding the purpose of the evaluation and the rationale for prioritization of evaluation questions is critical for transparency and acceptance of evaluation findings. It is essential that the evaluation address those questions of greatest need to the program and priority users of the evaluation.

Step 4: Planning for gathering credible evidence. Stakeholders have to accept that the methods selected are appropriate to the questions asked and that the data collected are credible or the evaluation results will not be accepted or used. The market for and acceptance of evaluation results begins in the planning phase. Stakeholders can inform the selection of appropriate methods.

Step 5: Planning for conclusions. Stakeholders should inform the analysis and interpretation of findings and facilitate the development of conclusions and recommendations. This in turn will facilitate the acceptance and use of the evaluation results by other stakeholder groups. Stakeholders can help determine if and when stakeholder interpretation meetings should be conducted.

Step 6: Planning for dissemination and sharing of lessons learned. Stakeholders should inform the translation of evaluation results into practical applications and actively participate in the meaningful dissemination of lessons learned. This will facilitate ensuring use of the evaluation. Stakeholders can facilitate the development of an intentional, strategic communication and dissemination plan within the evaluation plan.

Evaluation Plan Tips for Step 1

• Identify intended users who can directly benefit from and use the evaluation results.

• Identify an evaluation stakeholder workgroup of 8–10 members.

• Engage stakeholders throughout the plan development process as well as the implementation of the evaluation.

• Allow for adequate time to meaningfully engage the evaluation stakeholder workgroup.

At This Point, Your Plan Should Include the Following:

• Identified the primary users of the evaluation.

• Created the evaluation stakeholder workgroup.

• Defined the purposes of the evaluation.

References

Centers for Disease Control and Prevention. (2011). Developing an Effective Evaluation Plan. CDC. https://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf