Vocabulary

• Synthesize: To put together information or ideas from different sources to create a clear and complete understanding. In public health, it means combining data and research findings to draw conclusions or make recommendations.
• Proximity: Refers to how close or near something is to another thing. In public health, it might describe the closeness between people and potential health risks, like how close someone lives to a pollution source or how close people are to each other during an outbreak.

(Centers for Disease Control and Prevention, 2011)

Shared Understanding of the Program

A program description clarifies the program’s purpose, stage of development, activities, capacity to improve health, and implementation context.

The next step in the CDC Framework and the evaluation plan is to describe the program. A program description clarifies the program’s purpose, stage of development, activities, capacity to improve health, and implementation context. A shared understanding of the program and what the evaluation can and cannot deliver is essential to the successful implementation of evaluation activities and use of evaluation results. The program and stakeholders must agree upon the logic model, stage of development description, and purpose(s) of the evaluation. This work will set the stage for identifying the program evaluation questions, focusing the evaluation design, and connecting program planning and evaluation.

Narrative Description

A narrative description helps ensure a full and complete shared understanding of the program. A logic model may be used to succinctly synthesize the main elements of a program. While a logic model is not always necessary, a program narrative is. The program description is essential for focusing the evaluation design and selecting the appropriate methods. Too often groups jump to evaluation methods before they even have a grasp of what the program is designed to achieve or what the evaluation should deliver. Even though much of this will have been included in your funding application, it is good practice to revisit this description with your ESW to ensure a shared understanding and that the program is still being implemented as intended. The description will be based on your program’s objectives and context, but most descriptions include the following at a minimum:

• A statement of need to identify the health issue addressed.

• Inputs or program resources available to implement program activities.

• Program activities linked to program outcomes through theory or best practice program logic.

• Stage of development of the program to reflect program maturity.

• Environmental context within which a program is implemented.

Logic Model

The description section often includes a logic model to visually show the link between activities and intended outcomes. Reviewing the model with the ESW ensures a shared understanding of the model. Reviewing the logic model also ensures an accurate and complete reflection of your program. The logic model should identify available resources (inputs), what the program is doing (activities), and what you hope to achieve (outcomes). You might also want to articulate any challenges you face (such as the program’s context or environment). Figure 2 illustrates the basic components of a program logic model. View the logic model from left to right: the further away from the intervention, the more time needed to observe outcomes. There are two major challenges in evaluating chronic disease prevention: attribution versus contribution and distal outcomes not occurring near the program interventions or policy changes. In addition, given the complexities of dynamic implementation environments, realized impacts may differ from intended impacts. However, the rewards of understanding the proximal and distal impacts of the program intervention often outweigh the challenges.

Logic model elements include the following criteria:

• Inputs: Resources necessary for program implementation.

• Activities: The actual interventions that the program implements in order to achieve health outcomes.

• Outputs: Direct products obtained as a result of program activities.

• Outcomes (short-term, intermediate, long-term, distal): The changes, impacts, or results of program implementation (activities and outputs).

Figure 2: Sample Logic Models

For a description of the image, access the appendix

Stage of Development

You will need an accurate assessment of the program’s stage of development to fully describe your program and prepare you to focus your evaluation. The developmental stages that programs typically move through are planning, implementation, and maintenance. In the example of a policy or environmental initiative, the stages might look somewhat like the following steps:

1. Assess the environment and assets.

2. Policy or environmental change is in development.

3. The policy or environmental change has not yet been approved.

4. The policy or environmental change has been approved but not implemented.

5. The policy or environmental change has been in effect for less than one year.

6. The policy or environmental change has been in effect for one year or longer.

Steps 1–3 would typically fall under the planning stages. Steps 4–5 would fall under implementation, and Step 6 would fall under maintenance. It is important to consider a developmental model because programs are dynamic and evolve over time. Programs are seldom fixed in stone and progress is affected by many aspects of the political and economic context. When it comes to evaluation, the stages are not always a once-and-done sequence of events. When a program has progressed past the initial planning stage, there may be occasions when additional environment and asset assessments are needed.  The evaluation for a multi-year plan should consider future plans to prepare datasets and baseline information for evaluation projects that consider more distal impacts and outcomes. Preparation is an advantage of completing a multi-year evaluation plan with your ESW.

The stage of development conceptual model is complementary to the logic model. Figure 3.1 shows how general program evaluation questions are distinguished by both logic model categories and the developmental stage of the program. This places evaluation within the appropriate stage of program development (planning, implementation, and maintenance). The model offers suggested starting points for asking evaluation questions within the logic model while respecting the developmental stage of the program. This will prepare the program and the workgroup to focus the evaluation appropriately based on program maturity and priorities.

Figure 3.1: Stage of Development and Logic Model Category

For a description of this image, access the appendix

Figure 3.2: Stage of Development by Logic Model Category Example

For a description of the image, access the appendix

Key evaluation questions and needs for information will differ based on the stage o development of the program. Additionally, the ability to answer key evaluation questions will differ by stage of development of the program and stakeholders need to be aware of what the evaluation can and cannot answer. For the policy program example above, below are different questions to ask yourself at different stages of the evaluation process.

Planning Stage Questions:

• Is there public support for the policy?

• What resources will be needed for implementation of the policy?

Implementation Stage Questions:

• Is there compliance with the policy?

• Is there continued or increased public support for the policy?

• Are there major exemptions or loopholes to the policy?

Maintenance Stage Questions:

• What is the economic impact of the policy?

• What is the health impact of the policy?

For more on stage of development and Smoke-Free Policies, please see the Evaluation Toolkit for Smoke-Free Policies at the Centers of Disease Control and Prevention: Smoking and Tobacco Use website.

Evaluation Plan Tips for Step 2

• A program description will facilitate a shared understanding of the program between the program staff and the evaluation workgroup.

• The description section often includes a logic model to visually show the link between activities and intended outcomes.

• The logic model should identify available resources (inputs), what the program is doing (activities), and what you hope to achieve (outcomes).

• A quality program evaluation is most effective when part of a larger conceptual model of a program and its development.

At This Point, Your Plan Should Include the Following:

• Identified the primary users of the evaluation.

• Created the evaluation stakeholder workgroup. 

• Defined the purposes of the evaluation.

• Described the program, including context.

• Created a shared understanding of the program.

• Identified the stage of development of the program.

Figure 4: Preventing Initiation of Tobacco Use Among Young People

For a description of the image, access the appendix

References

Centers for Disease Control and Prevention. (2011). Developing an Effective Evaluation Plan. CDC. https://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf