Vocabulary

• Iterative: Doing something repeatedly, making small changes each time to improve it. In public health, this means continuously adjusting and refining strategies or plans based on feedback and new information.
• Conceptualizing: Creating a clear idea or plan for something. In public health, it's having a solid understanding of a health issue or program before taking action, like defining goals and outlining strategies.
• Surveillance: Keeping watch over a situation or condition. In public health, it means regularly collecting and analyzing data or diseases or health trends to monitor their spread and take action to control them.

(Centers for Disease Control and Prevention, 2011)

Overview

The amount of information you can gather concerning your program is potentially limitless. Evaluations, however, are always restricted by the number of questions that can be realistically asked and answered with quality, the methods that can be employed, the feasibility of data collection, and the available resources. These are the issues at the heart of Step 3 in the CDC framework: focusing the evaluation. The scope and depth of any program evaluation is dependent on program and stakeholder priorities; available resources, including financial resources; staff and contractor availability; and amount of time committed to the evaluation. The program staff should work together with the ESW to determine the priority and feasibility of these questions and identify the uses of results before designing the evaluation plan. In this part of the plan, you will apply the purposes of the evaluation, its uses, and the program description to narrow the evaluation questions and focus the evaluation for program improvement and decision making. In this step, you may begin to notice the repetitive process of developing the evaluation plan as you revisit aspects of Step 1 and Step 2 to inform decisions to be made in Step 3.

Useful evaluations are not about special research interests or what is easiest to implement but what information will be used by the program, stakeholders (including funders), and decision-makers to improve the program and make decisions. Establishing the focus of the evaluation began with the identification of the primary purposes and the primary intended users of the evaluation. This process was further solidified through the selection of the ESW. Developing the purposeful intention to use evaluation information and not just produce another evaluation report starts at the very beginning with program planning and your evaluation plan. You need to garner stakeholders’ interests and prepare them for evaluation use. This step facilitates conceptualizing what the evaluation can and cannot deliver.

It is important to collaboratively focus the evaluation design with your ESW based on the identified purposes, program context, logic model, and stage of development. Additionally, issues of priority, feasibility, and efficiency need to be discussed with the ESW and those responsible for the implementation of the evaluation. Transparency is particularly important in this step. Stakeholders and users of the evaluation will need to understand why some questions were identified as high priorities while others were rejected or delayed.

Developing Evaluation Questions

In this step, it is important to solicit evaluation questions from your various stakeholder groups based on the stated purposes of the evaluation. The questions should then be considered through the lens of the logic model, program description, and stage of development of the program. Evaluation questions should be checked against the logic model. Changes may be made to either the questions or the logic model, thus reinforcing the iterative nature of the evaluation planning process. The stage of development discussed in the previous chapter will facilitate further narrowing the evaluation questions. It is important to remember that a program may experience characteristics of several stages simultaneously once past the initial planning stage. You may want to ask yourself how long your program has been in existence? If your program is in the planning stage, it is unlikely that measuring distal outcomes will be useful for informing program decision-making. However, in a multi-year evaluation plan, you may begin to plan for and develop the appropriate surveillance and evaluation systems and baseline information needed to measure these distal outcomes (to be conducted in the final initiative year) as early as year one. In another scenario, you may have a coalition that has been established for 10 years and is in the maintenance stage. However, contextual changes may require you to rethink the programmatic approach being taken. In this situation, you may want to do an evaluation that looks at both planning stage questions (Are the right folks at the table? And are they really engaged?), as well as maintenance stage questions (Are we having the intended programmatic impact?). Questions can be further prioritized based on the ESW and program information needs as well as feasibility and efficiency issues.

Often if a funder requires an evaluation plan, you might notice text like the following: Submit with application a comprehensive written evaluation plan that includes activities for both process and outcome measures. Distinguishing between process and outcome evaluation can be similar to considering the stage of development of your program against your program logic model. In general, process evaluation focuses on the first three boxes of the logic model: inputs, activities, and outputs (CDC, 2008). This discussion with your ESW can further facilitate the focus of your evaluation.

EXAMPLE OF A PROCESS TO DEVELOP EVALUATION QUESTIONS

Program: SafeTeens in Lincoln County. The county health department is starting a project to form a teen safety coalition and conduct a seatbelt campaign.

The various stakeholders may have different evaluation questions they want answered, and the answers will come in different stages of the program.

Planners need to assess the feasibility of each question.

After assessing stakeholders’ expectations and the feasibility, final evaluation questions can be written.

For a description of the image, access the appendix

Outcome evaluation, as the term implies, focuses on the last three outcome boxes of the logic model: short-term, intermediate, and long-term outcomes.

For a description of the image, access the appendix

As a program can experience the characteristics of several stages of development at once, so, too, a single evaluation plan can and should include both process and outcome evaluation questions at the same time. Excluding process evaluation questions in favor of outcome evaluation questions often eliminates the understanding of the foundation that supports outcomes.

As you and the ESW take ownership of the evaluation, you will find that honing the evaluation focus will likely solidify interest in the evaluation. Selection of final evaluation questions should balance what is most useful to achieving your program’s information needs while also meeting your stakeholders’ information needs. Having stakeholders participate in the selection of questions increases the likelihood of their securing evaluation resources, providing access to data, and using the results. This process increases personal ownership of the evaluation by the ESW. However, given that resources are limited, the evaluation cannot answer all potential questions.

The ultimate goal is to focus the evaluation design such that it reflects the program stage of development, selected purpose of the evaluation, uses, and questions to be answered. Transparency related to the selection of evaluation questions is critical to stakeholder acceptance of evaluation results and possibly the continued support of the program.

Even with an established multi-year plan, Step 3 should be revisited with your ESW annually (or more often if needed) to determine if priorities and feasibility issues still hold for the planned evaluation activities. This highlights the dynamic nature of the evaluation plan. Ideally, your plan should be intentional and strategic by design and generally cover multiple years for planning purposes. But the plan is not set in stone. It should also be flexible and adaptive. It is flexible because resources and priorities change and adaptive because opportunities and programs change. You may have a new funding opportunity and a short-term program added to your overall program. This may require insertion of a smaller evaluation plan specific to the newly funded project, but with the overall program evaluation goals and objectives in mind. Or, resources could be cut for a particular program, requiring a reduction in the evaluation budget. The planned evaluation may have to be reduced or delayed. Your evaluation plan should be flexible and adaptive to accommodate these scenarios while still focusing on the evaluation goals and objectives of the program and the ESW.

Budget and Resources

Discussion of budget and resources (financial and human) that can be allocated to the evaluation will likely be included in your feasibility discussion. In the Best Practices for Comprehensive Tobacco Control Programs (2007), it is recommended that at least 10% of your total program resources be allocated to surveillance and program evaluation. The questions and subsequent methods selected will have a direct relationship to the financial resources available, evaluation team member skills, and environmental constraints (for example, you might like to do an in-person home interview of the target population, but the neighborhood is not one that interviewers can visit safely). Stakeholder involvement may facilitate advocating for the resources needed to implement the evaluation necessary to answer priority questions. However, sometimes, you might not have the resources necessary to fund the evaluation questions you would like to answer most. A thorough discussion of feasibility and recognition of real constraints will facilitate a shared understanding of what the evaluation can and cannot deliver. The process of selecting the appropriate methods to answer the priority questions and discussing feasibility and efficiency is iterative. Steps 3–5 in planning the evaluation will often be visited concurrently in a back-and-forth progression until the group comes to consensus.

Evaluation Plan Tips for Step 3

• It is not possible or appropriate to evaluate every aspect or specific initiative of a program every year.

• Evaluation focus is context dependent and related to the purposes of the evaluation, primary users, stage of development, logic model, program priorities, and feasibility.

• Evaluation questions should be checked against the logic model and stage of development of the program.

• The iterative nature of plan development is reinforced in this step.

At This Point, Your Plan Should Include the Following:

• Identified the primary users of the evaluation.

• Created the evaluation stakeholder workgroup. 

• Defined the purposes of the evaluation.

• Described the program, including context.

• Created a shared understanding of the program. 

• Identified the stage of development of the program.

• Prioritized evaluation questions and discussed feasibility, budget, and resource issues.

References

Centers for Disease Control and Prevention. (2011). Developing an Effective Evaluation Plan. CDC. https://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf