Community-based Evaluation Methods

Community-Based Evaluation Methods & Data Analysis Amy Carroll April 22, 2005

Different Types of Data 

Opinions, priorities





Aspirations, motivations

Services or resources provided



Level of awareness, knowledge, attitudes or beliefs

Resident utilization of services or resources provided



Numbers or rates of disease, illness, disability, injuries



Sales transactions, purchases



Policies



Pictures, other visuals





Behaviors, practices



Assets, skills



Networks, associations



Needs, fears, problems, concerns



Demographic



Quantitative vs. Qualitative Data Quantitative: 

 







Collected in the form of numbers or percentages Closed-ended questions Answers who?, what?, when? and where? Can demonstrate cause and effect Can “represent” a population Cannot collect new ideas or responses, only those considered ahead of time

Qualitative: 

  





Collected in the form of words, concepts, themes, or categories Open-ended questions Answers how?, why? Can provide richer, more in-depth data Can provide data in a respondent’s own words Can explore new ideas in a dynamic and unstructured way

Common Community-Based Evaluation Methods • • • • • • •

Secondary data Activity tracking Client tracking Mapping/ Photovoice Surveys Focus groups Key informant interviews

Which of these methods collect quantitative data? Which collect qualitative data?

Secondary Data • • • • • • •

• •

County health department Vital statistics  Birth & death certificates City or county government Schools (including school nurse records) Hospital or clinic records Local health promotion, health ed or advocacy agencies Private foundations which fund community programs Universities or local community colleges Local surveys or research efforts going on in your community?? Data you already collect!

Activity Tracking • • • •

Tracking and documenting program activity implementation Tracking and documenting number of people reached Process evaluation method, but important for outcome evaluation May include: weekly/monthly program implementation log, sign-in sheets at classes or health fairs, counts of materials/ brochures distributed, etc.

Client Tracking •

Tracking and documenting encounters with clients; describing the encounter and client characteristics

•

More than just counting

•

Can be used for process and outcome evaluation

•

May include: outreach encounter form, home visit form, clinic intake form, etc.

Mapping/ Photovoice •

Data can also include maps and photos → don’t just have to be numbers or words

•

Maps and photos can provide visual imagery to give meaning to quantitative data

•

They can show graphically what cannot be described with numbers or words, such as the conditions of a neighborhood or program facilities, or a community’s proximity to park space or health services.

•

Make sure your maps and photos are clear and illustrate the points you would like to make

Surveys •

•

Surveys are composed of carefully crafted questions on a topic or issue for the purpose of revealing information about community residents or services they utilize. The most commonly used survey methods are: 1. Mailed Surveys 2. Hand-out Surveys 3. Face to Face Surveys 4. Telephone Interview Surveys

Planning A Survey • • • • • • •

Determine your purpose Determine if the survey data collection is appropriate Select your target audience Select a survey method Design survey questionnaire Pilot test survey Publicize and distribute surveys

Determine the Data You Need •

Draft your research questions – what you want to learn from these interviews.

•

Identify the data you will need to collect in order to answer these questions.

•

The type of data you are looking for will help you identify the best persons in the community to survey, and the best survey questions to ask them.

Target Population The following are two steps in determining your survey target population. 1) Define your target population -identify the group of individuals from your community from who you want to collect information (single mothers, high school students, the uninsured, etc.) 2) Identify the geographic area of your target population.--Where is your group of interest located? What are the

Select A Survey Method Self Administered: These surveys are filled out by respondents themselves without the assistance of trained interviewers. A. Mail Surveys B. Drop-off Surveys

Administered by Interviewer: These surveys are filled out with the assistance of trained interviewers. A. Face to Face Surveys B. Telephone Surveys

Design Survey Questionnaire •

Your survey questions should be in direct relationship with what you want to learn from your target population.

•

Asking unrelated questions will interrupt the flow of the survey and tire out your respondents.

•

Be brief and strategic when designing questions!

•

Open-ended questions are great for answering “why” questions, but only include one or two.

•

Motivating people to fill out your survey will help you collect the data you need – this can include a benefits statement or incentives such as entry into a lottery or drawing.

Pilot Test Survey •

It is always a good idea to pilot test the survey with a small group of people from your target population.

•

The purpose for piloting the survey is to get feedback on the survey question and structure before implementing it. This will ensure you identify potentially confusing, offensive, leading, or loaded questions from your survey.

•

Publicize and Distribute Surveys 

Advertise your survey a couple of weeks before distribution in the community.



Publicizing what you are trying to do and how the information collected will benefit the community may help increase your response rate.



Create a survey distribution timeline and stick to it - postponing survey distribution date may limit the time you have set aside for data analysis.

Advantages and Disadvantages of Surveys Advantages 





 

Data can be collected from a lot of respondents easier than any other method Can get a large enough sample that can be representative of the larger population Findings can be generalized to the larger population Can cover a lot of topics Can easily compare

Disadvantages 





Survey instrument must be carefully constructed to avoid leading questions, and to make sure the appropriate responses are available Response rates can be low for self-administered surveys, especially mailed ones Response will be low if survey is too long

Focus Groups Focus group are a qualitative research method designed to learn more about how people think, feel, or make decisions (attitudes, perceptions, opinions, experiences) through focused discussions.

Focus Groups (cont.) 

A series of discussions involving 8-12 people, selected to share their perceptions of a defined topic



Stimulates participants to share their opinions openly in a group discussion



Captures rich data in participants’ own words



Flexible to capture new ideas and issues



Can be difficult to summarize and interpret results across groups



Need to be creative when recruiting busy people

Planning a Focus Group •

Determine the data you need

•

Determine and select your target audience

•

Develop a focus group discussion guide

•

Schedule focus groups

•

Recruit focus group participants

•

Recruit moderator and note-taker

•

Plan and conduct focus groups

Determine the Data You Need •

Draft your research questions – what you want to learn from these focus groups.

•

Identify the data you will need to collect in order to answer these questions.

•

The type of data you are looking for will help you identify the best persons to recruit for your focus groups, as well as the best questions to develop for

Target Audience • •

•

•

•

To whom do you want to talk? What types of experiences and perspectives you are hoping to capture in your discussion? Design and use a screening tool to recruit participants: ask questions that qualify people to participate. It is important that you include participants who will feel comfortable talking to each other. Participants should not know each

How Many Focus Groups? •

Conduct 2-3 focus groups for each subpopulation of interest (i.e. gender, age, race/ethnicity, language, etc.)

•

2-3 ensures that the perspectives and experience you capture are more representative of the larger population and not just by “chance”.

•

Length: 1.5 hrs.-2 hrs.

•

Focus Group Size: 6-12 participants

Focus Group Discussion Guide •

Discussion guide: An outlined script to help guide the moderator and focus the discussion

•

Organization: Introduction Purpose Ground rules Focus group questions Closing statements

Discussion Guide Questions • •

•

• •

Introductory questions: getting people to think about the general topic. Transition questions: questions that get participants to logically transition between the introduction topics and the ideas contained in the key questions. Key questions: 2-5 questions important to getting the information you have set out to collect. Ending questions: questions that provide closure and reflection. Summary question: note-taker summarizes the major themes heard throughout the discussion in a short oral summary (2-3 minutes), then will ask the participants if the summary covered all the major points.

Scheduling Focus Groups •

Location: The focus groups should take place in a convenient and accessible location. It is also important to select a neutral environment.

•

Time of day: most convenient for your target audience. Set time earlier than intend to start.

•

Incentives: Food or refreshments, money or gifts, drawings, child care, transportation, anything that is meaningful to those community members.

Conducting & Moderating •

The discussion guide will help guide the moderator with the questions you want answered.

•

Moderator Responsibilities: setting up ground rules, keeping the discussion focused and on track, drawing out comments from quiet participants, and asking for clarification or additional information from participants when needed.

•

Note-taker Responsibilities: takes comprehensive notes, operates audio equipment (tape-recorder), keeps track of time, handles environmental conditions and

Skills of the Moderator Extremely important to select moderators based on skills and experience: • Comfortable/familiar with leading meetings or groups • Able to exercise mild control over the group • Able to guide conversation back on target • Maintain group enthusiasm and interest • Curious about the topic • Respects participants and their comments • Creates and maintains a comfortable environment • Adequate background knowledge • Good oral and written skills

Advantages and Disadvantages of Focus Groups Advantages  

 



Flexible Captures rich, indepth data Immediate results Encourages and stimulates individuals to share more openly Data can be combined with quantitative data to provide a complete picture about an issue

Disadvantages 







May be challenging to recruit participants Need to schedule at least 2-3 focus groups to capture diversity Difficult to generalize results to the larger population because of small numbers of participants Difficult to compare results across groups

Key Informant Interviews •

Key informant interviews are qualitative, indepth interviews with key informants

•

Key informants are individuals who know what is going on in the community (community leaders, professionals, residents, advocates, elected officials, doctors, etc.)

•

The following are two common techniques used to conduct key informant interviews: 1. 2.

Telephone interviews Face-to face interviews

Planning Key Informant Interviews • • • • • • • • •

Determine the data you need Determine target population and brainstorm possible key informants Select key informants Select interview technique Develop an interview guide Determine documentation method Select designated interviewer(s) Conduct key informant interviews Compile and organize key informant interview data

Determine the Data You Need •

Draft your research questions – what you want to learn from these interviews.

•

Identify the information you will need to collect to answer these questions.

•

The type of data you are looking for will help you identify the best persons in the community to interview, and the best questions to ask them.

Target Audience and Key Informants •

Determine your target population (program clientele, racial/ethnic minority groups, adolescents, women, the elderly, HIV+ persons, etc.)

•

Brainstorm and create a list of possible key informants who are knowledgeable and closely involved with your target population.

•

Select key informants: • Who have first-hand knowledge about the target population, the program, and the issues you are trying to investigate. • Who represent different experiences, backgrounds, and viewpoints.

Select Interview Technique Telephone interviews (15-25 min): • •

May be the most convenient and least time intensive way to interview busy key informants. Disadvantage: not having the personalized interaction otherwise possible through a face-toface interview.

Face-to-face interviews (20-30 min): • •

Most frequently used format. But time intensive. Advantages: provides a free-exchange of ideas and lends itself to ask more complex questions.

Develop Interview Guide Introduction- Introduce yourself and your purpose and/or benefits statement. Key questions- Draft 5-10 questions important to getting the information you have set out to collect. Probing questions – Probing questions encourage participants to reflect more deeply on the meaning of their comments. Closing question—Provide an opportunity for the key informant to give any additional information or comments. Summary—Quickly summarize the major

Determine Documentation Method 



Note-taking: manually write the key informant’s comments while conducting the interview, on a copy of the interview guide. Tape-recording: use a tape recorder to document what key informants say. This approach may allow the interviewer to freely engage in the conversation without worrying about note-taking.

Conduct Interviews •

Thank respondents for their time and stress the importance of the interview.

•

Interviewers should listen carefully for recurring and new opinions or beliefs.

•

In order to compare with other interviews and identify emerging themes, it is important to get answers from every person interviewed on key questions.

Advantages and Disadvantages of Key Informant Interviews

Advantages 







Disadvantages

 Rich data can be gathered relatively cheaply & easily Allows interviewer to establish rapport with the  respondent, clarify questions, and draw out responses Allows for discussion of  topic without group dynamic of focus groups Provides an opportunity to build relationships with important

Selecting the “right” key informants may be difficult so they represent diverse backgrounds and viewpoints May be challenging to reach and schedule interviews with busy and/or hard-to-reach respondents Difficult to generalize results to the larger population unless interviewing many key informants

Things to Consider Before You Select Your Methods •

What data collection method(s) would best collect the type of data you need from your target population

•

What data collection method(s) would help you to best answer your research question(s)?

•

Think of your available resources: time, staffing, skills, funding, computer technology

•

Prior experience

•

Potential benefits

•

Potential challenges

How to Understand the Data You Collect (Data Analysis)

Why It’s a Good Idea to Plan Ahead for Data Analyses 

To determine if the data you collect are practical for analyses



To review the appropriateness of your chosen data collection method(s)



To inform the data collection instrument you develop



To decide if you will need help with analyses

1: Check the Data 

Make sure it’s all there



Make sure it makes sense



Catch any mistakes that happened while filling out the survey or entering the data



Perform data quality checks throughout your data collection process



Common mistakes: – A missing or incomplete response – An impossible response – A response unrelated to the question – A response that contradicts an earlier response

2: Go Back to the Research Questions 

Be sure to analyze only the data that will help you answer your research question(s)



Don’t get side-tracked into analyzing other pieces of data with your limited time and resources



It is normal to collect more data than you will need for your evaluation



You can return to any other interesting data after completing your evaluation

3: Reduce the Amount of Data 

 



Save your data in its original form so that you can return to it if you need to Look at it in small chunks at a time Eliminate any irrelevant data from your analysis Create summary documents to help you “eyeball” the data more easily to look for patterns or themes: – Numeric spreadsheet – Qualitative summary document

4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data 

Calculate averages – Used with “continuous” data: infinite number of values – Add numbered responses – Divide by number of responses – 2 + 4 + 6 + 10 = 22 22/4 = 5.5



Count frequencies – Used with “discrete” data: set number of response categories – Count number of responses – Number “yes”, number “no” – Number of choice “A”, Number of choice “B”,

4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data 

Calculate proportions – Is standardized – Equation: % = # of responses/ total # of respondents X 100 – Example: 45 females/ 100 respondents X 100 = 45% of the respondents were female



Calculate rates – Similar to proportions; also standardized – Easier to compare – Equation: Rate = freq of event in pop/ total pop X 100,000 – Example: 15 cases of lung cancer/ population of 3,500 X 100,000 = 428 persons per 100,000 have lung cancer

4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data 



Compare averages, frequencies, proportions, rates Compare data from different populations: – County vs. County – County vs. State – State vs. US



Compare data from different segments of a population: – Male vs. female – African American, white, Latino, Asian, American Indian, etc. – Children, teens, adults, seniors – Clientele of your program vs. those who are not

4: Analyze the Data Quantitative Data Analysis: Looking for Patterns in the Data 

Present the data in different ways to see additional patterns and relationships:

– Chart/ table – shows averages, counts, proportions, or rates side-by-side – Pie graph – demonstrates percentages of the whole – Bar graph – compares quantities – Line graph – shows trends over time



Determine your findings – Interesting results? – Interesting patterns or relationships? – These are your findings

Exercise 5.4A (Using Step 5 of Performing a Community Assessment) 

Look at sample survey and sample spreadsheet in Appendix C



Answer questions on page 51 and 52

Step 4: Analyze the Data Qualitative Data Analysis: Looking for Themes in the Data 

Qualitative data analysis can be deceptively trickier – Qualitative data is by nature “bigger” – Analysis requires more analytical thinking and interpretation



Qualitative data is more open to “bias”

– Numbers don’t lie – Analysts bring their values, assumptions and opinions – Analysts may think they “know” how people feel



Include at least 2, preferably 3 people in each stage of data analysis to avoid bias



Create as structured of a process as possible

Step 4: Analyze the Data Qualitative Data Analysis: Looking for Themes in the Data 1.

Read through all of the data at least twice –

2.

Create categories – –

3.

–

5. 6.

Focus group or interview questions Or specific assessment health issues

List themes that emerge from discussion –

4.

Stay “close” or “grounded” in the data

Sub-categories Ideas

Rank order according to frequency of appearance Summarize discussion around each theme Determine your findings

Exercise 5.4B (Using Step 5 of Performing a Community Assessment) 

Look at sample focus group discussion transcript in Appendix C



Answer questions on page 56

5: Verify Findings 

You now have: – Some patterns and relationships you found in your quantitative data – Some themes and categories you found in your qualitative data



Based on the results you tallied and summarized, pull out the main findings of each method you used



Verify these findings by re-tallying and resummarizing the data to make sure you get the same results



You can also verify your findings by comparing the results reached by 2 analysts independently

6: Interpret Findings and Draw Conclusions 

Determine what interpretations can be drawn from each finding



Invite different perspectives from different Partners and community members, as this will make your conclusions stronger



Are the results similar to what you expected? If not, discuss why you think they are different



Brainstorm alternative explanations for your results to make sure you have considered all possibilities



Make sure the conclusions answer the original assessment questions



Draw conclusions and recommendations that can be shared with external audiences

Exercise 5.6 (Using Step 5 of Performing a Community Assessment) 

Look at the secondary data tables in Appendix C



Answer questions on page 59