Vocabulary

• Timetable: A schedule or plan that shows when different activities or events will happen. In public health, it might outline when specific tasks or milestones need to be completed within a project or program.
• Triangulation: Using multiple sources of information or methods to verify or validate findings. In public health, it involves comparing data from different sources or using different research methods to ensure accuracy and reliability.
• Tabulate: Organizing data into a table or chart to make it easier to understand and analyze. In public health, it involves arranging information systematically to summarize findings or trends. 
• Stratify: Dividing a population or sample into distinct groups or categories based on specific characteristics. In public health, it might involve separating data by age, gender, or socioeconomic status to analyze health disparities or trends.
• Demographic: Relating to the characteristics of a population, such as age, gender, ethnicity, or income. In public health, demographic data is often used to understand patterns of health and disease within different groups. 
• Threshold: The level at which something begins to have an effect or trigger a response. In public health, it might refer to the point at which an intervention becomes necessary to address a health issue. 
• Benchmark: A standard or reference point used for comparison or evaluation. In public health, it might be a set of criteria or targets used to measure the performance or effectiveness of a program or intervention.
• Baseline: The initial or starting point against which future changes or progress are measured. In public health, it often refers to the initial assessment of a health condition or outcome before implementing an intervention or program.

Overview

Step 5 is needed in order to draw conclusions about the data you collect to evaluate your program. Justifying conclusions includes analyzing the information you collect, and drawing conclusions from your data. Engaging the ESW in this step is critical to ensuring the acceptance of evaluation findings, and ensuring they are meaningful and credible. Actively meeting with stakeholders to discuss preliminary findings helps the interpretation phase. In fact, stakeholders often have novel insights to guide interpretation that evaluation staff may not have.

Some programs focus their efforts on collecting data, but fail to work with the data for analysis, interpretation, feedback, and conclusions. These programs suffer from being Data Rich but Information Poor (DRIP).  This can lead to many missed opportunities the data may have provided.

Your plans should include time for interpretation and review by stakeholders. Interpreting results in the context of the evaluation goals will make justifying conclusions possible. Reliability and validity issues and possible sources of biases should be discussed in the planning stages with the ESW. The propriety standard helps guide the evaluator’s decisions on how to ethically analyze and interpret data to assure that all stakeholder values are respected. If possible, the ESW should consider triangulation of data and remedies to threats to the credibility of the data should be addressed as early as possible.

Evaluation Plan Tips for Step 5

• The planning for conclusions step is needed to turn the data collected into meaningful information for action.

• Including your stakeholder group in this step is directly tied to acceptance of data and conclusions.

• Errors in planning in this step can create serious delays in the final evaluation report.

• It is critical that your plans include time for interpretation and review from stakeholders (including your critics) to increase transparency and validity of your process and conclusions.

At This Point, Your Plan Should Include the Following:

• Identified the primary users of the evaluation.

• Created the evaluation stakeholder workgroup. 

• Defined the purposes of the evaluation.

• Described the program, including context.

• Created a shared understanding of the program. 

• Identified the stage of development of the program. 

• Prioritized evaluation questions and discussed feasibility issues.

• Discussed issues related to credibility of data sources.

• Identified indicators and/or performance measures linked to chosen evaluation questions.

• Determined implementation roles and responsibilities for program staff, evaluation staff, contractors, and stakeholders.

• Developed an evaluation plan methods grid.

• Addressed possible threats to the credibility of your data. 

• Developed a timetable that includes cleaning and analyzing data as well as stakeholder interpretation sessions. 

Step 5: Justify Conclusions 

You will need to analyze and interpret the evidence gathered in Step 4 whether your evaluation is conducted to show program effectiveness, to help improve the program, or to demonstrate accountability. Step 5 encompasses analyzing the evidence, making claims about the program based on the analysis, and justifying the claims by comparing the evidence against stakeholder values. 

Why Is It Important to Justify Conclusions? 

As the figure below notes, conclusions become justified when analyzed and synthesized findings (the evidence) are interpreted through the prism of values (standards that stakeholders bring, and then judged accordingly. Justification of conclusions is fundamental to utilization-focused evaluation. When agencies, communities, and other stakeholders agree that the conclusions are justified, they will be more inclined to use the evaluation results for program improvement.

Different stakeholders may bring different standards and values to the table. The work of Step 5 benefits from the efforts of the previous steps: differences in values and standards will have been identified during stakeholder engagement in Step 1. 

Analyzing and Synthesizing The Findings 

Data analysis is the process of organizing and classifying the information you have collected, tabulating it, summarizing it, comparing the results with other appropriate information, and presenting the results in an easily understandable manner. The five steps in data analysis and synthesis are straightforward:  

1. Enter the data into a database and check for errors. If you are using a surveillance system such as BRFSS or PRAMS, the data have already been checked, entered, and tabulated by those conducting the survey. If you are collecting data with your own instrument, you will need to select the computer program you will use to enter and analyze the data, and determine who will enter, check, tabulate, and analyze the data.  

2. Tabulate the data. The data need to be tabulated to provide information (such as a number or percentage) for each indicator. Some basic calculations determine the following:

• • The number of participants.  

  • The number of participants achieving the desired outcome.

  • The percentage of participants achieving the desired outcome.  

3. Analyze and stratify your data by various demographic variables of interest, such as participants’ race, sex, age, income level, or geographic location.

4. Make comparisons. When examination of your program includes research as well as evaluation studies, use statistical tests to show differences between comparison and intervention groups, between geographic areas, or between the pre-intervention and post intervention status of the target population.  

5. Present your data in a clear and understandable form. Data can be presented in tables, bar charts, pie charts, line graphs, and maps. 

In evaluations that use multiple methods, evidence patterns are detected by isolating important findings (analysis) and combining different sources of information to reach a larger understanding (synthesis).

Setting Program Standards for Performance 

Program standards are the benchmarks used to judge program performance. They reflect stakeholders’ values about the program and are fundamental to sound evaluation. The program and its stakeholders must articulate and negotiate the values that will be used to consider a program successful, adequate, or unsuccessful. Possible standards that might be used in determining these benchmarks include the following:  

• Needs of participants.  

• Community values, expectations, and norms.  

• Program mission and objectives.  

• Program protocols and procedures.  

• Performance by similar programs.  

• Performance by a control or comparison group.  

• Resource efficiency.  

• Mandates, policies, regulations, and laws.  

• Judgments of participants, experts, and funders.  

• Institutional goals.  

• Social equity.  

• Human rights. 

When stakeholders disagree about standards and values, it may reflect differences about which outcomes are deemed most important. Or, stakeholders may agree on outcomes but disagree on the amount of progress on an outcome necessary to judge the program a success. This threshold for each indicator, sometimes called a benchmark or performance indicator, is often based on an expected change from a known baseline. In Step 5, you will negotiate consensus on these standards and compare your results with performance indicators to justify your conclusions about the program. Performance indicators should be achievable but challenging and should consider the program’s stage of development, the logic model, and the stakeholders’ expectations. Identifying and addressing differences in stakeholder values and standards early in the evaluation is helpful. 

Interpreting the Findings and Making Judgments 

Judgments are statements about a program’s merit, worth, or significance formed when you compare findings against one or more selected program standards. When forming judgments about a program, multiple program standards can be applied and stakeholders may reach different or even conflicting judgments.

Conflicting claims about a program’s quality, value, or importance often indicate that stakeholders are using different program standards or values in making their judgments. This type of disagreement can prompt stakeholders to clarify their values and reach consensus on how the program should be judged. 

Illustrations from Cases 

We can use the affordable housing program to illustrate this chapter’s main points about the sources of stakeholder disagreements and how they may influence an evaluation. For example, the various stakeholders may disagree about the key outcomes for success. Maybe the staff consider the completion of the house as most important. By contrast, the groups that supply volunteers may demand that home ownership produce improvement in life outcomes and better jobs. Even when stakeholders agree on the outcomes, they may disagree about the amount of progress to be made on these outcomes. For example, while churches may want to see improved life outcomes just for the individual families they sponsor, some foundations may want to change communities as a whole by changing the mix of renters and homeowners. It is important to identify disagreements about values early in the evaluation. That way, consensus can be negotiated so that program description and evaluation design and focus reflect the needs of the various stakeholders.

Standards for Step 5: Justify Conclusions

References

Centers for Disease Control and Prevention. (2011). Developing an Effective Evaluation Plan. CDC. https://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf

Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. (2011). U.S. Department of Health and Human Services. https://www.cdc.gov/evaluation/guide/CDCEvalManual.pdf