Tools and Settings
Content
Questions and Tasks
Step 5 is needed in order to draw conclusions about the data you collect to evaluate your program. Justifying conclusions includes analyzing the information you collect, and drawing conclusions from your data. Engaging the ESW in this step is critical to ensuring the acceptance of evaluation findings, and ensuring they are meaningful and credible. Actively meeting with stakeholders to discuss preliminary findings helps the interpretation phase. In fact, stakeholders often have novel insights to guide interpretation that evaluation staff may not have.
Some programs focus their efforts on collecting data, but fail to work with the data for analysis, interpretation, feedback, and conclusions. These programs suffer from being Data Rich but Information Poor (DRIP). This can lead to many missed opportunities the data may have provided.
Your plans should include time for interpretation and review by stakeholders. Interpreting results in the context of the evaluation goals will make justifying conclusions possible. Reliability and validity issues and possible sources of biases should be discussed in the planning stages with the ESW. The propriety standard helps guide the evaluator’s decisions on how to ethically analyze and interpret data to assure that all stakeholder values are respected. If possible, the ESW should consider triangulation of data and remedies to threats to the credibility of the data should be addressed as early as possible.
The planning for conclusions step is needed to turn the data collected into meaningful information for action.
Including your stakeholder group in this step is directly tied to acceptance of data and conclusions.
Errors in planning in this step can create serious delays in the final evaluation report.
It is critical that your plans include time for interpretation and review from stakeholders (including your critics) to increase transparency and validity of your process and conclusions.
Identified the primary users of the evaluation.
Created the evaluation stakeholder workgroup.
Defined the purposes of the evaluation.
Described the program, including context.
Created a shared understanding of the program.
Identified the stage of development of the program.
Prioritized evaluation questions and discussed feasibility issues.
Discussed issues related to credibility of data sources.
Identified indicators and/or performance measures linked to chosen evaluation questions.
Determined implementation roles and responsibilities for program staff, evaluation staff, contractors, and stakeholders.
Developed an evaluation plan methods grid.
Addressed possible threats to the credibility of your data.
Developed a timetable that includes cleaning and analyzing data as well as stakeholder interpretation sessions.
You will need to analyze and interpret the evidence gathered in Step 4 whether your evaluation is conducted to show program effectiveness, to help improve the program, or to demonstrate accountability. Step 5 encompasses analyzing the evidence, making claims about the program based on the analysis, and justifying the claims by comparing the evidence against stakeholder values.
As the figure below notes, conclusions become justified when analyzed and synthesized findings (the evidence) are interpreted through the prism of values (standards that stakeholders bring, and then judged accordingly. Justification of conclusions is fundamental to utilization-focused evaluation. When agencies, communities, and other stakeholders agree that the conclusions are justified, they will be more inclined to use the evaluation results for program improvement.
Different stakeholders may bring different standards and values to the table. The work of Step 5 benefits from the efforts of the previous steps: differences in values and standards will have been identified during stakeholder engagement in Step 1.
Data analysis is the process of organizing and classifying the information you have collected, tabulating it, summarizing it, comparing the results with other appropriate information, and presenting the results in an easily understandable manner. The five steps in data analysis and synthesis are straightforward:
1. Enter the data into a database and check for errors. If you are using a surveillance system such as BRFSS or PRAMS, the data have already been checked, entered, and tabulated by those conducting the survey. If you are collecting data with your own instrument, you will need to select the computer program you will use to enter and analyze the data, and determine who will enter, check, tabulate, and analyze the data.
2. Tabulate the data. The data need to be tabulated to provide information (such as a number or percentage) for each indicator. Some basic calculations determine the following:
The number of participants.
The number of participants achieving the desired outcome.
The percentage of participants achieving the desired outcome.
3. Analyze and stratify your data by various demographic variables of interest, such as participants’ race, sex, age, income level, or geographic location.
4. Make comparisons. When examination of your program includes research as well as evaluation studies, use statistical tests to show differences between comparison and intervention groups, between geographic areas, or between the pre-intervention and post intervention status of the target population.
5. Present your data in a clear and understandable form. Data can be presented in tables, bar charts, pie charts, line graphs, and maps.
In evaluations that use multiple methods, evidence patterns are detected by isolating important findings (analysis) and combining different sources of information to reach a larger understanding (synthesis).
Program standards are the benchmarks used to judge program performance. They reflect stakeholders’ values about the program and are fundamental to sound evaluation. The program and its stakeholders must articulate and negotiate the values that will be used to consider a program successful, adequate, or unsuccessful. Possible standards that might be used in determining these benchmarks include the following:
Needs of participants.
Community values, expectations, and norms.
Program mission and objectives.
Program protocols and procedures.
Performance by similar programs.
Performance by a control or comparison group.
Resource efficiency.
Mandates, policies, regulations, and laws.
Judgments of participants, experts, and funders.
Institutional goals.
Social equity.
Human rights.
When stakeholders disagree about standards and values, it may reflect differences about which outcomes are deemed most important. Or, stakeholders may agree on outcomes but disagree on the amount of progress on an outcome necessary to judge the program a success. This threshold for each indicator, sometimes called a benchmark or performance indicator, is often based on an expected change from a known baseline. In Step 5, you will negotiate consensus on these standards and compare your results with performance indicators to justify your conclusions about the program. Performance indicators should be achievable but challenging and should consider the program’s stage of development, the logic model, and the stakeholders’ expectations. Identifying and addressing differences in stakeholder values and standards early in the evaluation is helpful.
Judgments are statements about a program’s merit, worth, or significance formed when you compare findings against one or more selected program standards. When forming judgments about a program, multiple program standards can be applied and stakeholders may reach different or even conflicting judgments.
Conflicting claims about a program’s quality, value, or importance often indicate that stakeholders are using different program standards or values in making their judgments. This type of disagreement can prompt stakeholders to clarify their values and reach consensus on how the program should be judged.
Tips to Remember When Interpreting Your Findings
First and foremost, interpret evaluation results with the goals of your program in mind.
Keep your audience in mind when preparing the report. What do they need and want to know?
Consider the limitations of the evaluation:
Possible biases
Validity of results
Reliability of results
Are there alternative explanations for your results?
How do your results compare with those of similar programs?
Have the different data collection methods used to measure your progress shown similar results?
Are your results consistent with theories supported by previous research?
Are your results similar to what you expected? If not, why do you think they may be different?
Source: US Department of Health and Human Services. Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: US Department of Health and Human Services, Centers for Disease Control and Prevention, Office on Smoking and Health, November 2001.
We can use the affordable housing program to illustrate this chapter’s main points about the sources of stakeholder disagreements and how they may influence an evaluation. For example, the various stakeholders may disagree about the key outcomes for success. Maybe the staff consider the completion of the house as most important. By contrast, the groups that supply volunteers may demand that home ownership produce improvement in life outcomes and better jobs. Even when stakeholders agree on the outcomes, they may disagree about the amount of progress to be made on these outcomes. For example, while churches may want to see improved life outcomes just for the individual families they sponsor, some foundations may want to change communities as a whole by changing the mix of renters and homeowners. It is important to identify disagreements about values early in the evaluation. That way, consensus can be negotiated so that program description and evaluation design and focus reflect the needs of the various stakeholders.
Standard
Questions
Utility
Have you carefully described the perspectives, procedures, and rationale used to interpret the findings?
Have stakeholders considered different approaches for interpreting the findings?
Feasibility
Is the approach to analysis and interpretation appropriate to the level of expertise and resources?
Propriety
Have the standards and values of those less powerful or those most affected by the program been taken into account in determining standards for success?
Accuracy
Can you explicitly justify your conclusions? Are the conclusions fully understandable to stakeholders?
Following is an example of how one project planned to meet the Evaluation Standards, after an intervention to improve seatbelt compliance by teens.
Table 5.1: Evaluation Standards and Data Analysis and Synthesis [EXAMPLE]
Develop a description of the perspectives, procedures, and rationaleused to interpret the findings.Include information on your plans for data collection, storage, andanalysis.
We will measure seatbelt usage by direct observation at the same intersection, before and after the campaign, to discover if rates of compliance have changed. Paper surveys will be stored at the health dept office and compiled by staff.
How will you present the findings to the ESW and other keystakeholders?Does each stakeholder require a different format?Be specific and consider the stakeholder data needs from Step 4.
We will present findings to the youth coalition and to the ESW in 2 different meetings, in the same oral presentation and written report formats for both.
Feasibility:
Is the proposed data analysis and synthesis approach realistic/feasiblegiven the current situation? Why or why not?
This approach is realistic because youth have expressed willingness to help with observational surveys.
What modifications need to be made to make the data analysisapproach meaningful for stakeholders and feasible for evaluators?
We can show seatbelt compliance both before and after, which will show stakeholders the difference their work has made.
How were the benchmarks selected?
Observed seatbelt use before campaign will be considered the baseline, and rate of use afterward will be considered the amount of improvement.
Are stakeholder standards taken into consideration when selecting
benchmarks for success?
We will obtain stakeholder comments and suggestions through surveys, before concluding if the campaign has had significant impact.
How will you encourage less vocal stakeholders to participate?
Survey responses will be anonymous so less vocal stakeholders will not feel inhibited in giving honest opinions.
What conclusions do you expect to be able to draw from the data?
We anticipate being able to show if the campaign has made a difference in the community’s seatbelt compliance.
Develop data distribution plan where information is accessible to key
stakeholders.
We will present findings in meetings, as well as distributing a final written report via email to all stakeholders.
Checklist for Step 5: Justifying Your Conclusions
Check data for errors.
Consider issues of context when interpreting data.
Assess results against available literature and results of similar programs.
If multiple methods have been employed, compare different methods for consistency in findings.
Consider alternative explanations.
Use existing standards (for example, Healthy People 2010 objectives) as a starting point for comparisons.
Compare program outcomes with those of previous years.
Compare actual with intended outcomes.
Document potential biases.
Examine the limitations of the evaluation.
In this example, the ESW planned their evaluation tasks:
Table 5.2: Task Planning [EXAMPLE]
Question
Response
Who will manage data analysis?
Health dept staff will analyze the survey results.
How will the data be analyzed (is there a particular method or program you intend to use)?
Excel will be used to calculate paired t-tests using values from baseline and after the intervention.
How will the data be presented to stakeholders (report, presentation, pie charts, etc.)
Data will be presented in both oral presentations and a written report, using bar charts to show how rates have changed.
Will you compare your data to another source? If so, describe the comparator study. If not, justify why no comparisons will be made.
Because we are specifically concerned with safety in our own area, we will not compare the results with other locations.
Who will be involved in interpreting data?Making judgments about conclusions?Making recommendations based on data?
Health dept staff will interpret the data after obtaining feedback from stakeholders. The ESW will reach a consensus for judgments and recommendations.
How will you address data that conflicts with stakeholder values?
We will decide before the surveys if any stakeholders have conflicting values.
What is your plan for dealing with results that directly contradict your evaluation assumptions?
If results contradict our assumptions we will review and revise future procedures.
What potential confounding variables may impact the results? List all the confounding variables you can identify at this stage.
Confounding variables may include: different age groups observed, different traffic patterns at the 2 observation times, and weather conditions making fewer observations possible.
What can be done to mitigate the effects of the confounding variables?
We will note conditions at baseline and try to match as similar conditions as possible at second observation point.
How do you expect the results of your evaluation to compare with other similar evaluations?
We expect that we will have as good or better results as similar projects, because of our emphasis on increased publicity.
What are some of the potential limits to your data collection, analysis, and interpretation methods? What can be done to limit the effects of these limitations?
Our methods may be limited by fewer samples to observe and a smaller data pool.
We can repeat the observations if necessary to increase the data available.
References
Centers for Disease Control and Prevention. (2011). Developing an Effective Evaluation Plan. CDC. https://www.cdc.gov/obesity/downloads/cdc-evaluation-workbook-508.pdf
Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide. (2011). U.S. Department of Health and Human Services. https://www.cdc.gov/evaluation/guide/CDCEvalManual.pdf