top of page

16. Evaluation

Writer's picture: Charity WattersCharity Watters

INTRODUCTION


WHAT IS MONITORING AND EVALUATION?


Monitoring is the regular collection of information about a project’s activities. It keeps track of project inputs and outputs such as activities, reports and documentation, finances and budgets, and supplies.


An evaluation asks whether a project is achieving what it sets out to do and whether it is making a difference. If this happens, evaluation seeks to understand how and why an intervention works. Evaluation keeps track of key outcomes related to a project’s objectives.


MONITORING AND EVALUATION PROVIDE:

  • Information on how well an intervention is doing and whether it is achieving its aims and objectives

  • Guidance on future intervention activities

  • An important part of accountability to donors


M&E HELPS TO:

  • Be accountable and responsible

  • Better manage risk and opportunities

  • Make informed decisions

  • Learn from experience


WHY MEASURING OUTCOMES IS IMPORTANT

  • Make programs more effective and sustainable: Continually shape our own strategies and projects so they become more effective.

  • Donor reporting: Showing donors that our projects are effective and represent the best possible outcomes.

  • Promoting an evidence-based approach: Demonstrating that Global Family’s projects are based on proven strategies for prevention, intervention, and aftercare.

  • Knowledge creation: Contributing to a broader body of knowledge around family-based childcare and anti- human trafficking interventions that influence other organizations and policies.


ADOPTING AN EVIDENCE-BASED APPROACH


In very rare circumstances will you choose to guess rather than know. The rise of evidence-based policy (EBP) and practice is driven by largely this idea, that it is better to have as much evidence possible and to be optimally informed in order to produce, theoretically, the most effective outcomes. EBP is often understood, in most cases on the part of the academic community, as involving a systematic and comprehensive understanding of high-quality and robust evidence. While this may be an ideal type of evidence use in policy and practice, evidence which is systematically collected and analyzed is not necessarily the most relevant, nor the most useful given either the problem or the stakeholders involved.


In most cases, evidence-informed practice starts with understanding what is effective and what is not effective. Measuring outcomes is important because it can be used to make programs more effective and sustainable, show donors and other stakeholders that a project represents the best possible outcomes, and contribute to a broader body of knowledge around a particular subject. In order to accomplish this, monitoring and evaluation (M&E) consists of the regular collection of information about a project’s activities, and then assessing whether a project is achieving its original goals and why and how and intervention operates. M&E can provide guidance on future activities and constitutes an integral part of accountability to intervention partners.


The general steps for evaluation consist of developing objectives, creating indicators, data collection, analysis of data, application of findings to implementation, and communicating the relevant information. Indicators should be specific and measurable, and they should communicate what has changed and for whom. You might choose to use a logic model, or the analysis of annual work plans to evaluate the efficiency and effectiveness of a program.


KEY ISSUES AND INDICATORS FOR CLUBS


CLUB ORGANIZATION

  • Number of clubs

  • Leaders trained

  • Volunteers recruited

COMMUNITY CAPACITY

  • Clubs achieve goals for awareness, community, and mobilization

  • Community resources (human, technical, and financial)

  • Networks, partnerships, and collaborations

SOCIAL CAPITAL

  • Interaction between individuals, groups, and sectors in a community


POST-MEETING EVALUATION

Meet with the individuals from your organization, volunteers, and partnering groups/organizations that were involved in the Awareness Campaign to debrief and assess. Talk about what went well, what could have been better, the content and activities that were effective, those that could be improved, and the information you want for next time. Then congratulate yourselves on a job well done!


Discussion questions:

  • What went well?

  • What could have been better?


LOGIC MODELING


Creating a logic model can be a great way of evaluating how your inputs and activities create outputs, outcomes, and impact. We’ve added some examples below to get you started!



PRE-POST EVALUATION


Overall, reviews and assessments of evidence on preventative programs are far from achieving data saturation. Robust evaluations of girl’s empowerment and other youth club programs are most often found in high-income countries, while evaluations from (or focusing on) interventions in low- and middle-income countries tend to be sub-sets of evaluations on general anti-human trafficking programs implemented by NGOs. Studies and evaluations often do not define or report outcomes, which is largely a result of the lack of a comparison group and lack of data on incidence of trafficking and child abuse. Additionally, many clubs are located in rural areas, which presents challenges for researchers, especially independent evaluators, to conduct in-depth research.


Conducting an evaluation of your club can help you assess the extent club activities have mitigated the risks of child trafficking and abuse. Evaluation may include identifying key risk factors, measuring risk factor prevalence amongst club participants, and understanding the perception of mitigation amongst club leaders. Findings can be used to improve program effectiveness; identify the components of the curriculum, training, and reporting structure that are effective and those that are ineffective; improve contextualization; and contribute to knowledge around community-based youth club implementation and preventative interventions targeting human trafficking and child abuse.


DATA COLLECTION


Baseline and end-line surveys were designed to collect data on changes in risk factor prevalence amongst club participants and to measure the club’s ability to mitigate those risks. Survey questions for club participants were written in a topic-sensitive manner so that participants who themselves or whose friends may have been affected by trafficking or abuse would not be adversely affected or triggered. Survey questions were informed by EMpower’s research and guidelines for monitoring and evaluation programs for youth[1]. Questions that measure self-esteem were adapted from the Rosenberg self-esteem scale[2] and questions that measure gender attitudes were adapted from the GEM scale developed by Promundo[3].


MATERIAL TRANSLATION


Translate the survey questions. If any questions need to be re-worded in order to contextualize them or for other reasons, such as they are not appropriate or may not be well understood by respondents, report all changes that are made.


TRAINING FOR CLUB LEADERS


If you are the club leader, please fill out one survey per club that you oversee. If you oversee or manage club leaders that lead their own clubs, arrange a meeting for them to assemble. It is recommended that this meeting is scheduled for a full day and that the following outline is used.


1. Surveys as a Tool to Improve Effectiveness


Explain that a survey is a tool that is used to improve the effectiveness of programs, such as clubs, in achieving their goals. A survey is a list of questions for each individual participate to fill out. It will ask them about their experiences in the club as well as in outside activities and will also ask them about their experiences in observing or identifying abuse and exploitation. It is important that club leaders fill out surveys themselves, as well as assist club participants in completing surveys. The goal of this meeting is to discuss the goals of the surveys and provide instructions for helping participants complete them.


2. Explain the Purpose of the Evaluation and the Baseline Survey


Explain that the information from the surveys will most importantly be used to improve Global Family’s processes in club management and sustainability, and to improve the writing, development, and contextualization of the curriculum. In explaining the purpose of the evaluation, it is important that leaders do not feel that they have been unsatisfactory in their work. Rather, we aim to improve Global Family’s own processes and curriculum.


Explain that responses will be anonymous. No names are collected on the survey forms, and any outcomes that are reported in Global Family’s internal reports or published elsewhere will be attached to them.


3. Answer Questions


If leaders have questions, answer them here.


4. Fill out the Survey for Club Leaders


Distribute the survey for club leaders and provide one copy for each individual and allow 1-2 hours for this activity. Explain that while filling out the survey, they should not discuss amongst themselves their answers. Instead, each person should fill out the survey with their own answers.


If they have questions, you may assist them in understanding the questions and the type of answer to be provided but refrain from providing answers for the surveys yourself. For example, if someone asks you how to answer the question, “Why did you join the club?”, you may explain, “This question is asking the reason why you joined the club. What caused you to join? Were there any desires or people who influenced you to join?”. You may not provide a potential answer to the question, for instance by saying, “You may have joined because someone asked you to join, because you were interested in helping girls in the community, or because you wanted to learn from the experience.”


When everyone is finished, collect the completed surveys. Thank them for their involvement and reiterate that their feedback will be very useful in improving clubs, both in this country and around the world. If anyone refuses to complete a survey, ask them why and record their answers. If some questions are not completed, ask why they were unable to answer these questions and record their answers.


It is important that you do not inform the club leaders of each other’s outcomes. This is so that leaders would not analyze each other’s answers, and so that the end-line survey will provide an accurate account of the clubs’ effectiveness without the information that baseline survey responses would have provided.


5. Discuss the Process for Distributing the Survey Amongst Club Participants


You may want to have a short break before discussing how leaders can distribute the survey amongst the youth who are involved in their club.


REPORTING


Report the number of club participants and club leaders who complete the baseline and end-line surveys, summarizing response rates in the following table.


Complete the following table to show the reason for incomplete responses.


Complete the following table to report results for each topic.


When analyzing results, look at significant findings, common themes, sub-group results, and implications for policy, practice, and research.

  • Significant findings: Findings that represent significant results and what these mean for our programs, and an analysis of effective components, ineffective components, or components with unknown effectiveness

  • Common themes: Findings that represent significant results and what these mean for our programs

  • Sub-group analysis: Difference between sub-groups amongst participants (e.g. age, location, duration of participation, time the club has been active

  • Implications for policy, practice, and research: How findings contribute to the current evidence base and their implications for policy and practice


APPLYING FINDINGS TO PROJECT IMPLEMENTATION


Key questions:

  • What are the current strengths of the project and how can we build upon these?

  • What are the gaps or weaknesses in the project that need to be addressed?

  • Based on our findings, what are the key steps that can be taken?

  • How much time and resources will it take to improve on strengths and address weaknesses?

  • Would taking these steps bring value to this community or to the beneficiaries of the project?


SURVEYS



REFERENCES


[1] EMpower. (2018). Evaluating programs for youth. New York, London, Hong Kong: EMpower. [2] Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press. [3] UN Women. (2013). Making women count: An annual publication on gender and evaluation by UN Women multi country office for India, Bhutan, Sri Lanka and Maldives. New Delhi: UN Women.

1 view0 comments

Recent Posts

See All

Comments


bottom of page