Vocabulary

Lifestyle: The way in which a person or group lives, including their habits, behaviors, and chosen activities.

Prestigious: Having a high reputation or esteemed status, often associated with success, influence, or excellence.

Policymakers: Individuals or groups responsible for creating and implementing policies, rules, or laws, especially in government or organizational settings.

Lifespan: The total duration of a person's life or the length of time that something exists or functions.

Funders: Individuals, organizations, or entities that provide financial support or funding for projects, activities, or initiatives.

Coalition: A temporary alliance or partnership between different groups, organizations, or individuals for a common purpose or goal.

Demographic: Characteristics that describe a population, such as age, gender, income, education, and other defining factors.

Illegible: Difficult or impossible to read; handwriting or text that is unclear or not easily understood.

Wholesaler: A business or person that sells goods in large quantities to retailers or other businesses rather than directly to consumers.

Retailer: A business or person that sells goods or services directly to consumers; the final step in the distribution chain before products reach the end user.

Methods and Evaluation

Methods: Communication Channels 

(O’ Sullivan et al., 2003)

You will spend the bulk of your communication budget on creating materials and placing them in the most suitable channels and on using the most appropriate tools for communicating to audiences. This chapter will help you select the communication channels and tools that are most likely to move the strategic approach forward in the most cost-efficient manner.

Step 1: Choosing the Channels That Are the Most Likely to Reach the Intended Audience

Health communicators have defined communication channels as modes of transmission that enable messages to be exchanged between “senders” and “receivers.”

The various types of communication channels are as follows:

• Interpersonal Channels, which include one-to-one communication, such as provider to client, spouse to spouse, or peer to peer.  

• Community-Based Channels, which reach a community (a group of people within a distinct geographic area, such as a village or neighborhood, or a group based on common interests or characteristics, such as ethnicity or occupational status). Here are the forms of community communication:

• Community-based media, such as local newspapers, local radio stations, bulletin boards, and posters.

• Community-based activities, such as health fairs, folk dramas, concerts, rallies, and parades.

• Community mobilization, a participatory process of communities identifying and taking action on shared concerns.

• Mass Media Channels, which reach a large audience in a short period of time and include:

• Television

• Radio

• Newspapers

• Magazines

• Outdoor or Transit Advertising

• Direct Mail

• The Internet

The following is a table that outlines the types of channel, audience reached, and its advantages and disadvantages:

(U.S. Department of Health and Human Services et al., 2018)

Other channels may include, but are not limited to the following:

Social networking 

Social networking services allow users to create online profiles and build social networks with others who share similar personal or professional interests and activities. 

• Facebook remains the most popular social media platform—two out of three adults report they use it. Users interact with friends, family, and others with similar interests. 

• X (formerly known as Twitter) is a news and social networking service that uses short messages (280 characters) called tweets. One out of four adults use it, and it’s particularly popular among younger adults. 

Image Sharing

Image sharing is an engaging way to present content. Remember to use copyright-free images when possible. 

• Instagram allows you to share pictures and videos publicly or privately. You can also share media from your Instagram account on Facebook, Twitter, and Tumblr. This platform is especially popular with younger audiences—71% of people ages 18–24 use it. 

• Pinterest is a visual discovery tool you can use to create and share collections of visual bookmarks called “boards.” Pinterest users pin visuals to boards and share them with followers. This platform is more than twice as popular with women than men. 

• Snapchat is a social messaging application that lets users personalize and share photos and video clips with friends. Like Instagram, Snapchat is most popular with younger adults—78% of people ages 18–24 use it. 

Online video sharing 

While not always classified as social media, online video platforms are a great way to share tailored health messages. 

• YouTube is the number-one site for sharing and watching videos—three out of 4 adults use it. It’s great for reaching a large audience quickly. 

• Vimeo works similarly to YouTube. It has a smaller archive of videos, but they tend to be more targeted for specific audiences. 

New Media Tools

You can share these tools on social media to link to relevant resources.

• Podcasts can be an effective way to deliver health information to target audiences, similar to radio programs. Visit the CDC podcasts page to browse hundreds of podcasts on a variety of health and safety topics. 

• Infographics are visual representations of data, which can make complex information much easier to understand. For examples, check out the CDC Infographic page.

Tools to Support Social Media

You can use these helpful tools to coordinate and enhance your social media use. 

• Hootsuite is a social media management tool that features a dashboard-style interface. You can manage multiple social media profiles with one account—including Twitter, Facebook, LinkedIn, Instagram, and other platforms. Hootsuite helps you keep your social media efforts up to date and organized, and it allows you to schedule posts in advance. It also offers basic analytics for the accounts you’ve synced. 

• TweetDeck is a dashboard-style interface that helps you manage your Twitter accounts. It features customizable columns that let you filter content by lists, mentions, trends, favorites, particular hashtags of interest, and more. Like Hootsuite, you can also use TweetDeck to schedule tweets in advance. 

• Bitly is a link-managing tool you can use to shorten URLs for Twitter, text, or other forms of social networking. You can also track basic analytics for the links you create. It’s particularly useful when developing Twitter content, but be aware that bitlinks no longer distinguish between government and nongovernment links.

Brainstorming Channel Opportunities 

To start developing a channel strategy, write down opportunities (or openings) for sending your message during a typical day in the life of your audience. 

Typical times might include early morning, midmorning, midday, early afternoon, late afternoon, early evening, dinner, late evening, special events (list day, week, or month), seasonal opportunities (harvest time, holiday season). 

(O’ Sullivan et al., 2003)

Evaluate the Best Strategic Approach for the Channel Mix

Your next decision is to decide the focus of the channel mix. What is the best way to reach the intended audience, based on the objectives you’ve created? Should you focus on building reach, building frequency, or maximizing both?

Build Reach Quickly

Do you want to reach as many different people in the audience segment as quickly as possible? If so, the channel mix will be based on reach. This approach means that the lead channels selected are ones that can reach a large number of people in a short period of time. In some countries, television is considered such a medium. In other countries, it is radio. Community events can reach a large number of people within a community, but the frequency of message exposure is limited to the timeframe of the event and to the number of events planned for a community.

Emphasize Frequency

Should the channel mix be one that steadily conveys a message to build recall over a long period of time? If so, emphasize frequency, and use a medium that may not reach as many people quickly but is affordable enough to repeat messages regularly over an extended period of time. Radio in many countries is a good example of a channel that helps to build frequency. Radio advertising is relatively inexpensive, and radio spots can be repeated over and over during a campaign. Interpersonal communication (IPC) at a health clinic is a way to build frequency by ensuring that different levels of health providers reinforce the messages and by repeating the messages at each provider visit.

Combine Reach and Frequency

To build reach, but not at the expense of minimizing frequency, consider using an equal combination of these approaches. You will reach a large number of people on an ongoing basis. In some counties, a combination of television, radio, community events, and IPC is a way to build both reach and frequency at the same time.

Evaluate Each Channel’s Capacity to Reach the Audience in the Most Cost-Efficient Manner

A good channel mix balances a variety of factors, such as the size of the audience reached and the cost of reaching this audience. To compare each channel on a cost-efficiency basis, divide the cost of placing the message by the audience reached.

The Multichannel Approach

Research has demonstrated that a multichannel approach has a better chance of changing behavior than a single channel approach (Piotrow, Kincaid, Rimon, & Rinehart, 1997). In addition, a multichannel approach—especially an approach that uses mass media—can achieve objectives more quickly. Using several channels enables you to reach more people and to reach people in different environments with more frequency. The combination of multiple channels also offers a synergy to the campaign and gives it more impact. It is important for the primary audience as well as for other secondary and influencing audiences, who will most likely be exposed to these same messages. This exposure will, in turn, help to reinforce in them the necessity of supporting the campaign.

Select a Lead Channel and Supporting Channels, With a Rationale for Each

You must determine which channel will be the lead channel and which ones will serve as supporting channels. Just as a locomotive pulls the other cars on a train, the lead channel will be the engine that pulls the other channels with it. Think about your worksheets as you answer the following questions:  

• Which channel will reach the largest proportion of the intended audience?  

• Which channel will fit the message brief most appropriately?  

• Which channel will achieve the greatest impact? 

Although a mass medium may reach more people, it may not always make sense to choose it as a lead channel. 

See the following example in the table used to determine the lead channel and supporting channels. 

Example: Ghana’s “Life Choices”

In Ghana, a demand generation strategy for family planning (FP) was designed to encourage the use of modern contraceptives among several audience segments: young sexually active unmarried adults, young married adults who wanted to space the number of children that they planned to have, and more mature married adults who wanted to limit the number of children that they had. Since the strategic approach was to associate FP with the ability to achieve life goals and since the messages were designed to focus on specific characters, television became the lead channel to help deliver the story of each character’s life goal and subsequent FP choice.

Evaluating Health Communication Campaigns

Evaluating health communications tells programs and partners if campaigns are going as planned and achieving goals. Evaluating campaign progress by tracking activities helps determine if messages are reaching the intended audience and how to improve the campaign. This is sometimes called process evaluation. It can salvage an ineffective campaign by allowing the communications team to quickly respond to failing strategies. Evaluating campaign outcomes helps programs determine if objectives were met (Centers for Disease Control and Prevention, 2018).

Your evaluation system should address simple questions that are important to your community, your staff, and (last but never least!) your funding partners. Try to think about financial and practical considerations when asking yourself what sort of questions you want answered. The best way to insure that you have the most productive evaluation possible is to come up with an evaluation plan. (Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box, n.d., p. 36)

(Chapter 7, Section 8. Identifying and Analyzing Stakeholders and Their Interests, n.d.; Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box, n.d., p. 36)

Brainstorm Program Stakeholders

Stakeholders are those who may be affected by or have an effect on an effort. They may also include people who have a strong interest in the effort for academic, philosophical, or political reasons, even though they and their families, friends, and associates are not directly affected by it.

One way to characterize stakeholders is by their relationship to the effort in question.

• Primary stakeholders are the people or groups that stand to be directly affected, either positively or negatively, by an effort or the actions of an agency, institution, or organization.

• Example: In some cases, there are primary stakeholders on both sides of the equation: a regulation that benefits one group may have a negative effect on another.  A rent control policy, for example, benefits tenants but may hurt landlords.

• Secondary stakeholders are people or groups that are indirectly affected, either positively or negatively, by an effort or the actions of an agency, institution, or organization.  

• Example: A program to reduce domestic violence, for instance, could have a positive effect on emergency room personnel by reducing the number of cases they see. It might require more training for police to help them handle domestic violence calls in a different way. Both of these groups would be secondary stakeholders.

• Key stakeholders, who might belong to either or neither of the first two groups, are those who can have a positive or negative effect on an effort, or who are important within or to an organization, agency, or institution engaged in an effort.

• Example: Funders, elected or appointed government officials, heads of businesses, or clergy and other community figures who wield a significant amount of influence.

Take some time to make a list of your project or initiative's stakeholders, as well as which category they fall into.

• Community groups: Perhaps this is the most obvious category of stakeholders, because it includes the staff and/or volunteers involved in your initiative or project. It also includes the people directly affected by it: your target audience and agents of change.

• Grantmakers and funders: Grantmakers exist on an international, national, state, and local level and may be private companies and foundations, or local, county, state, or federal government organizations (Chapter 9. Developing an Organizational Structure for the Initiative | Section 1. Organizational Structure: An Overview | Main Section | Community Tool Box, n.d.).

• University-based researchers: This includes researchers and evaluators that a coalition or initiative may choose to bring in as consultants or full partners. Such researchers might be specialists in public health promotion, epidemiologists, behavioral scientists, specialists in evaluation, or some other academic field (Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box, n.d., p. 36). 

Each type of stakeholder will have a different perspective on your organization as well as what they want to learn from the evaluation. Every group is unique, and you may find that there are other sorts of stakeholders to consider with your own organization. Take some time to brainstorm about who your stakeholders are before you being making your evaluation plan (Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box, n.d.).

(Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box, n.d., p. 36)

How to Develop An Evaluation Plan

Four Main Steps to Developing an Evaluation Plan:

• Clarifying program objectives and goals

• Developing evaluation questions

• Developing evaluation methods

• Setting up a timeline for evaluation activities

Clarify Program Objectives and Goals

The first step is to clarify the objectives and goals of your initiative. What are the main things you want to accomplish, and how have you set out to accomplish them? Clarifying these will help you identify which major program components should be evaluated. 

Develop Evaluation Questions

For our purposes, there are four main categories of evaluation questions. Let's look at some examples of possible questions and suggested methods to answer those questions. Later on, we'll tell you a bit more about what these methods are and how they work

• Planning and implementation issues: How well was the program or initiative planned out, and how well was that plan put into practice?

• Possible questions: Who participates? Is there diversity among participants? Why do participants enter and leave your programs? Are there a variety of services and alternative activities generated? Do those most in need of help receive services? Are community members satisfied that the program meets local needs?

• Possible methods to answer those questions: Monitoring system that tracks actions and accomplishments related to bringing about the mission of the initiative, member survey of satisfaction with goals, or a member survey of satisfaction with outcomes.

• Assessing attainment of objectives: How well has the program or initiative met its stated objectives?

• Possible questions: How many people participate? How many hours are participants involved?

• Possible methods to answer those questions: Monitoring system (see above), member survey of satisfaction with outcomes, or goal attainment scaling.

• Impact on participants: How much and what kind of a difference has the program or initiative made for its targets of change?

• Possible questions: How has behavior changed as a result of participation in the program? Are participants satisfied with the experience? Were there any negative results from participation in the program?

• Possible methods to answer those questions: Member survey of satisfaction with goals, member survey of satisfaction with outcomes, behavioral surveys, or interviews with key participants.

• Impact on the community: How much and what kind of a difference has the program or initiative made on the community as a whole?

• Possible questions: What resulted from the program? Were there any negative results from the program? Do the benefits of the program outweigh the costs?

• Possible methods to answer those questions: Behavioral surveys, interviews with key informants, or community-level indicators.

Developing Evaluation Methods

Once you've come up with the questions you want to answer in your evaluation, the next step is to decide which methods will best address those questions. Here is a brief overview of some common evaluation methods and what they work best for.

(Chapter 38. Some Methods for Evaluating Comprehensive Community Initiatives | Section 2. Gathering Information: Monitoring Your Progress | Main Section | Community Tool Box, n.d.)

Monitoring and feedback system

This method of evaluation has three main elements:

• Process measures: Process measures are the activities that take place during the initiative which help you determine how well things are going. 

• Examples:

• Members who participate: The number and type of participants, frequency of attendance, and turnover rate of the members.

• Example: At the last general meeting of the initiative, 17 people attended. Of these folks, 10 were regular members, and 7 were sitting in on a meeting for the first time.

• Planning products: Written objectives, by-laws, or committees that contribute to the initiative.

• Example: The action plan for the coalition was approved by the coalition and will be implemented immediately.

• Media coverage: By radio, television, and print media.

• Example: Several five minute radio spots describing one of your group's projects aired on a local FM radio station.

• Financial resources: Grants or donations. Financial resources also can include in-kind services, such as free advertising or products that an individual or business might offer instead of money.

• Example: $8000 was received at a $50 per plate lunch that was held to raise money for local drug and alcohol abuse efforts.

• Services that are ultimately provided: classes, programs, workshops, publications or other services or communications provided for the community by the initiative.

• Example: Nutrition education workshops were conducted by child care providers.

• Community actions: Actions taken to encourage change in the community.

• Example: Merchants were asked to display signs describing the penalty for selling alcohol to minors and the need for proper identification.

• Outcome measures: Explain the overall impact that occurs as a result of these individual actions. Outcome measures highlight the changes that happen in the community as a result of the work done by your initiative.

• Examples:

• Changes in programs, such as a new or modified service program.

• Example: A parenting class was implemented by the initiative.

• Changes in policies, such as a new or modified policy.

• Example: A city ordinance was passed requiring owners of cigarette vending machines to place on every machine a sign that states "No cigarette sales to minors." The legislation was introduced at the urging of the Law Enforcement and Government Committee.

• Changes in practices, such as a new or modified practice.

• Example: Merchants displayed signs describing the penalty for selling alcohol to minors and the need for proper identification.

• Observational system: This is whatever you do to keep track of the initiative while it's happening.

• Examples:

• Event logs are written accounts of the major activities of the initiative. They might also be used to record any changes in the community brought about by the initiative, such as new programs, policies, or practices related to the initiative's goals and mission.

(Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box, n.d., p. 36)

Goal attainment report

If you want to know whether your proposed community changes were truly accomplished—and we assume you do—your best bet may be to do a goal attainment report. Have your staff keep track of the date each time a community change mentioned in your action plan takes place. Later on, someone compiles this information (for example, "Of our five goals, three were accomplished by the end of 1997.")

Behavioral surveys

Behavioral surveys help you find out what sort of risk behaviors people are taking part in and the level to which they're doing so. For example, if your coalition is working on an initiative to reduce car accidents in your area, one risk behavior to do a survey on will be drunk driving. The surveys should be distributed regularly so you can keep you finger on the pulse of the community.

Interviews with key participants

Key participants—leaders in your community, people on your staff, and so on—have insights that you can really make use of. Interviewing them to get their viewpoints on critical points in the history of your initiative can help you learn more about the quality of your initiative, identify factors that affected the success or failure of certain events, provide you with a history of your initiative, and give you insight which you can use in planning and renewal efforts.

Community-level indicators of impact

These are tested-and-true markers that help you assess the ultimate outcome of your initiative. For substance use coalitions, for example, the US Centers for Substance Abuse Prevention (CSAP) and the Regional Drug Initiative in Oregon recommend several proven indicators (such as single-nighttime car crashes, emergency transports related to alcohol) which help coalitions figure out the extent of substance use in their communities. Studying community-level indicators helps you provide solid evidence of the effectiveness of your initiative and determine how successful key components have been.

Set Up a Timeline for Evaluation Activities

Evaluation isn't something you should wait to think about until after everything else has been done. To get an accurate, clear picture of what your group has been doing and how well you've been doing it, it's important to start paying attention to evaluation from the beginning.

We suggest completing a table listing:

• Key evaluation questions (the five categories listed above, with more specific questions within each category)

• Type of evaluation measures to be used to answer them (what kind of data you will need to answer the question?)

• Type of data collection (what evaluation methods you will use to collect this data)

With this table, you can get a good overview of what sort of things you'll have to do in order to get the information you need.

(Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Examples | Community Tool Box, n.d.)

Example Table: Evaluation Questions and Methods

Here is a sample table of some of the questions from the evaluation plan of Youth Enrichment Services (YES), an organization that provides urban young people with services that encourage them to explore, challenge themselves physically and mentally, and interact with positive role models:

* Please note that the categories of types of data collection use broader terms than we've used to describe evaluation methods; here are some examples of methods that fall under these categories:

• Survey and Scale: For example, member survey of satisfaction with goals, member survey of satisfaction with process, member survey of satisfaction with outcomes, and behavioral surveys.

• Structured Interview: An interview with key participants.

• Staff Report and Log: This could be a monitoring system or goal attainment report.

• Direct Observation: This refers to the direct observation of behavior by researchers

• Archival Records: These can be community-level indicators of impact.

**Here are explanations of these terms:

• Case Study: An in-depth, start-to-finish analysis of a person, group, or community that focuses on developmental factors in relation to environment—good for answering "how" or "why" questions

• Pre- or Post-test Control Group: Monitoring a group of people who fit into your group's target population who you don't expose to your initiative, so that you can compare them with the people who were part of the initiative, helping you get a better idea of how much real impact you've made

• Time Series: effects are measured from a long series of repeated measurements taken both before and after an initiative
  

(O’ Sullivan et al., 2003, p. 195)

Identifying the Scope and Type of Evaluation

Determining the appropriate scope and type of evaluation that is both needed and possible is a key element in strategic design. At the basic level, evaluation serves these purposes:  

• Finding out whether the implementation activities spelled out in the work plan were actually carried out (process evaluation or monitoring)  

• Determining whether the objectives set forth in the strategy were achieved (impact assessment).

The objectives of your campaign guides every stage of evaluation. An objective of changing individual behavior requires an evaluation that will measure individual behavior over time; a policy objective of passing specific legislation will require a means to determine whether or what part of that legislation became law; and an objective of stimulating community activism will require from the start measures or indicators of community activism.

(O’ Sullivan et al., 2003)

Planning for Monitoring and Impact Assessment

Chronologically, once objectives have been established, evaluation must address the following:  

• First, monitoring of program activities and outputs  

• Second, impact assessment 

Each of these types of evaluation requires different actions and skills.

Process Evaluation or Monitoring

Monitoring is essential to be sure that the program is being carried out as planned and that no unintended, unforeseen, or unexpected events or shifts are taking place.

Monitoring requires attention to process, performance, and—to a lesser extent, outcomes: 

• Process monitoring—Here evaluators must measure whether activities occurred with the planned frequency, with the planned intensity, with the appropriate timing, and as directed to reach the intended audience. Ideally, monitoring begins at the start of the program activities and continues throughout the length of a program or campaign. Retrospective monitoring is less reliable than ongoing monitoring.  

• Performance monitoring—The quality, quantity, and distribution of communication outputs must be closely followed. For example, were the expected number of posters printed and distributed to the designated locations? Were the expected number of health care providers or others trained in the proper use of communication materials? Did all members of the management and communication team carry out their functions as planned? Were the quality and volume of the outputs—posters, serial dramas, and/or community events—at the expected and desired levels? In what ways did the performance of the management team meet expectations and work plan requirements? These measures of both process and performance monitoring should be as specific and as quantitative as possible, since it would be impossible to determine the success of the strategy if, in fact, it was not carried out as planned.  

• Outcome monitoring—Here the evaluation focus shifts from activities and actions back toward original objectives. If the objectives were increased attendance at certain specific clinics, increased purchase of certain products, or increases or decreases in a specified behavior, such as partner reduction or condom use, to what extent did these changes take place? During the monitoring process, extensive surveys may not be possible, but onsite observation and interviews are important to ensure that expected outcomes are beginning to take place. Unintended outcomes, different from those identified as original program objectives, would immediately call for close attention, feedback to program directors, and, if necessary, changes in either implementation or strategy.

Impact Assessment

Impact assessment seeks to answer the question “Did the communication strategy achieve the specified objectives?” Impact assessment then goes on to look at the difference that the strategy made in the overall program environment. Impact evaluation measures the impact on audiences and determines how to improve future projects.

Indicators

The first step in impact evaluation is to determine the indicators you will use to determine whether your objectives have been achieved. Examples of individual-level indicators for the behavior change communication strategies include the following (Bertrand & Escudero, 2002):  

• Percent of audience with a specific attitude (toward a product, practice, or service)  

• Percent of audience who believe that their spouses, friends, relatives, and community approve (or disapprove) of a product, practice, or service  

• Percent of non-users who intend to adopt a certain practice in the future  

• Percent of audience who are confident that they can adopt a particular behavior 

At a broader social level, the indicators listed below can be used to measure social change. Some of these indicators are measured qualitatively and others are more appropriately measured through quantitative techniques:

Indicators of Social Change

• Leadership  

• Degree and equity of participation  

• Information equity  

• Collective self-efficacy  

• Sense of ownership  

• Social cohesion  

• Social norms

A key issue in impact assessment is the research design or plan for the evaluation, which must be determined early in the project. Traditionally and particularly in biomedical research, the so-called Gold Standard for impact assessments is an experimental design in which individuals or communities are randomly assigned to be involved or not to be involved in a specific intervention. After the intervention is complete, the difference between those involved in the intervention and those not involved determines the impact of the project.

Identifying the Evaluation Design and Sources of Data

When considering how the evaluation of a communication effort should be designed and which sources of data will be used, it is helpful to keep the conceptual framework to the left in mind.

Levels of Measurement 

The evaluation of strategic communication depends upon the collection of data at different levels relevant to the objectives of the program. Here are the two major levels of measurement for communication evaluation data:  

• Population-based  

• Program-based 

Population-based measurement is useful in tracking initial, intermediate, and long-term outcomes. For example, surveys among the intended audience measure self-reported exposure, knowledge, attitudes, emotions, and other factors that are often precursors to behavior change (known as initial outcomes). 

Surveys can also track changes in behavior or practice over the life of a project (intermediate outcomes). These intermediate outcomes in turn influence the long-term outcomes related to health status, such as fertility or mortality rates. The following example on Zimbabwe measured both initial and intermediate outcomes. The one from Bolivia also included the long-term outcome of infant mortality.

Program-based measurement depends upon the collection of service statistics, sales data, client exit interviews, interviews or observations within clinic or service settings, and possibly a review of organizational and management factors relevant to program performance.

Types of Data Needed

In assessing communication programs, it is important to collect different types of data. Since communication affects individuals, groups, and communities, it is important to gather quantitative and qualitative information as well as information relevant to the appropriate unit of analysis.

• Quantitative data 

These data can be derived from surveys, service statistics, or sales data and involve active measures to gather information from individuals, communities, sites, or facilities in sufficient quantity, quality, and relevance for further analysis. None of these are easy to collect or without problems. 

• Surveys—The most common form of quantitative data with respect to strategic communication and behavior change is derived from surveys among randomly selected individual respondents. Surveys are a complex, highly specialized form of operational research that require implementation by experts.

• Service Statistics—Collection of service statistics may appear as a relatively easy task to be undertaken by visiting various facilities. In practice, however, service statistics have usually proved less satisfactory than surveys conducted by experienced survey researchers. Problems in the use of service statistics include the following:

• Different degrees of accuracy and completeness in maintaining service statistics 

• Different definitions of terms, such as initiation and continuation, as well as change in practices by different facilities 

• Illegible or incomprehensible records 

• Inaccessible records 

• Gaps in key data

Improvement of service statistics through management information systems is a continuing goal which might simplify the evaluation of some strategic communication programs, but it remains an ideal rather than an actuality in most countries.

• Sales Data—Collection of sales data can be an important element, particularly in the evaluation of social marketing programs. Here are some questions to be answered:

• At what point (wholesaler, distributor, retailer) will data be collected? 

• How will price and packaging differences be recorded? 

• How will free promotional materials be distinguished from sales materials? 

• How can substitution effects be taken into account when a lower-priced product displaces a higher-priced one?

• Qualitative Evaluation

Essential at the start of any project in order to understand the problem, the audience, and the overall situation, qualitative research can also play an important role throughout the project both in monitoring and in evaluating impact. The major roles for qualitative research in program evaluations include the following:

• Helping to evaluate activities and products as they are disseminated 

• Helping to explain how and why impact was achieved

Qualitative evaluation can be subtle, intuitive, and highly revealing when sensitively carried out, using ethnographic and unobtrusive measures. 

Key qualitative methods that can be used for evaluation are as follows:

• 1. Focus group discussions—Group discussions among homogeneous individuals led by a trained moderator can reveal community as well as individual values and prejudices, emotional intensity, points of controversy, and customary language used or audience verbatims. 

• 2. Interviews—Interviewers can tease out both information and emotional reactions by interviewing influentials, key informants, or typical audience members. Open-ended questions, followup to responses, and in-depth pursuit of significant issues as gathered through interviews can provide a wealth of valuable qualitative information.

• 3. Observation—Whether in person or through videotapes or even audiotapes, observation can provide an immediate insight into the reaction of an audience or client to specific types of communication or to recommended products and behaviors. Reproductive health programs offer less opportunity for direct observation than childcare and family health programs, but the observation of clinical practices or direct observation of those attending events or performances can provide valuable feedback. 

• 4. Diaries—These can be useful in literate societies or among literate professionals to record immediate day-by-day actions and reactions, to monitor ongoing activities, to capture a full history of events, and to understand better the process of behavior change (PBC) as it actually takes place over time.

• Combination—Quantitative and Qualitative Evaluation

• Evaluations that are both convincing as to causal effects and useful for future programming combine quantitative and qualitative measures. Quantitative evaluations can determine how much change took place and even how much change can be specifically attributed to different communication interventions.

• Qualitative evaluation is essential to frame the appropriate questions from which to derive quantitative data, to ensure the correct language so that the audience understands what is being asked, and to measure the intensity of emotions and certainty surrounding particular responses. Qualitative evaluation, above all, seeks to explore why and how change has taken place and to provide insights that can be useful in refining and improving future interventions. On the other hand, quantitative evaluation focuses on how much change has occurred.

Planning To Document and Disseminate Evaluation Results

The final stage for any evaluation should be a full documentation and report on the results. Evaluators who leave a few tables behind and do not write up or distribute results have not fulfilled their responsibilities. Results include insights and lessons learned in addition to data and tables. Since a program that is not evaluated and documented ceases to exist in the public mind after a very short time, this documentation is essential. A good evaluation should be clearly reported to at least three different audiences, each in appropriate ways: 

• To participants and the public—Basic data can be shared orally with community leaders, all others involved in the program itself, and the general public. Data can be explained in local media, and brief summaries can be provided to all who worked on the program and, to the extent possible, to those exposed to the intervention. 

• To donors—Whether government leaders, international agencies, or private foundations, donors are entitled to an honest and comprehensive report on the impact of projects that they have funded. Even where donors may appear busy and preoccupied, strategic communication programs have an obligation to present results. Presenting results can be done through meetings that the national and local press are invited to; through discussion groups, in which donors can participate; through reports that are released to the press; and through special media events particularly designed to call attention to evaluation results. Reports to donors should be accurate and clear and should give considerable attention to discussing not only the data, but also the implications of the data for future programming and other related activities.

• To the professional field—For professionals in the communication field and in whatever substantive field may be involved, peer-reviewed articles, presentations at professional meetings, book chapters, and even textbooks are essential to document important findings. Results for the professional and/or academic field need to describe in detail both the nature of the strategic communication interventions carried out and the methodologies used to collect and analyze the evaluation data. Where communication strategies suggest new directions or alter previous concepts or understandings, such innovations should be clearly highlighted and well defended. Communication to peers in the field should provide sufficient information, so that others are encouraged and are able to replicate the program wherever circumstances warrant. 

Strategic communication calls for strategic evaluation to be considered from the very beginning of the strategy design process. A strategic evaluation not only must include a full and adequate documentation of the process used, the objectives achieved, the impact, and where possible the cost-effectiveness of the program, but also guidelines and recommendations for improvement in future programs.

References

Centers for Disease Control and Prevention. (2018). Health Communications in Tobacco Prevention and Control. U.S. Department of Health & Human Services. https://www.cdc.gov/tobacco/stateandcommunity/guides/pdfs/health-communications-508.pdf

Chapter 7, Section 8. Identifying and Analyzing Stakeholders and Their Interests. (n.d.). Community Tool Box. https://ctb.ku.edu/en/table-of-contents/participation/encouraging-involvement/identify-stakeholders/main

Chapter 9. Developing an Organizational Structure for the Initiative | Section 1. Organizational Structure: An Overview | Main Section | Community Tool Box. (n.d.). Retrieved October 9, 2023, from https://ctb.ku.edu/en/table-of-contents/structure/organizational-structure/overview/main

Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Examples | Community Tool Box. (n.d.). Retrieved October 9, 2023, from https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/evaluation-plan/examples

Chapter 36. Introduction to Evaluation | Section 5. Developing an Evaluation Plan | Main Section | Community Tool Box. (n.d.). Retrieved October 9, 2023, from https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/evaluation-plan/main

Chapter 38. Some Methods for Evaluating Comprehensive Community Initiatives | Section 2. Gathering Information: Monitoring Your Progress | Main Section | Community Tool Box. (n.d.). Retrieved October 9, 2023, from https://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community-initiatives/monitor-progress/main

O’ Sullivan, G. A., Yonkler, J. A., Morgan, W., & Merritt, A. P. (2003). A field guide to designing a health communication strategy (pp. 17–51). Johns Hopkins Bloomberg School of Public Health/Center for Communication Programs. https://content.byui.edu/file/aaeaff60-24eb-4626-a1b2-8191d6faa261/1/Documents/Field%20Guide/02%20Chapter%2001%20Excerpts%20from%20A%20Field%20Guide--pages%2017-51.pdf

U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, & Agency for Toxic Substances and Disease Registry. (2018). Health Communication Playbook. https://www.cdc.gov/nceh/clearwriting/docs/health-comm-playbook-508.pdf