Friday 20 January 2012

Which evaluation model to use?


Program to be evaluated Description and evaluation of a prenatal exercise program for urban Aboriginal women. 
I believe that an outcomes- based evaluation would be a good choice to evaluate the prenatal exercise program for Aboriginal women in Saskatoon.  Outcomes- based evaluation looks at the benefits of a program to clients.  It would be beneficial to assess how program participants’ knowledge of GDM (gestational diabetes) and the importance of prenatal care were improved through participation in the program.
An outcomes-based evaluation focuses on inputs (materials and resources), activities (the processes of the program),outputs (units of service), and outcomes, including outcome targets and indicators.  An outcomes-based evaluation requires a reflection on what an organization wants to achieve with a program, not just a look at how a program worked (activities) and how many were served (outputs).  In this case, an evaluation of the prenatal exercise program for Aboriginal women in Saskatoon, should look at how the knowledge and attitudes of the participants were affected, not just how many women participated in the exercise program. 
Outcomes based evaluation encourages the creation of a logic model to clarify and define inputs, activities, outputs and outcomes.  From the logic model, outcomes can become apparent.  Asking the questions “why are we doing these activities and imputing staff and resources?” can point to the real outcomes we are looking to achieve with the program.  In the case of the prenatal exercise program for Aboriginal women in Saskatoon, having the women participate in physical activity for a period of time is beneficial in itself, but what are the real desired outcomes of the program?  Does the program aim to help Aboriginal women gain knowledge on how to reduce their chances of gestational diabetes?  Does it aim to change the women’s attitudes to continue to participate in physical activity on their own?  The use of an outcomes-based evaluation would encourage the stakeholders to identify their real goals for this health promotion program.  It would also assist in the definition of outcome targets (where they want to be) and indicators (how do we know that we are there).   

Klomp, J., Dyck, R., and Sheppard, S. (2003). Description and evaluation of a prenatal exercise program for urban Aboriginal women.  Canadian Journal of Diabetes, 27: 231-238

Just a note—the idea of Outcomes based evaluation was interesting to me.  I had always thought that programs were simply evaluated based on outputs, like the number of people served. I had never thought about the outcomes piece of it.  I am involved in some homelessness initiatives that promote sustained housing for clients, and I realize that the result of an evaluation based on outputs—clients housed, and an outcomes-based evaluation, looking at how the clients sustained housing as a result of the program, would be very different.   It changes how I look at the evaluations I have seen.  I learned something!

Saturday 14 January 2012

Review of Get B.U.S.Y 2008/2009 Evaluation Report

The Get B.U.S.Y Evaluation Report was prepared by the Centre For Community Based Research for the Boys and Girls Clubs of Canada in August 2009.  The Boys and Girls Club of Canada(BGCC) provides programs to children and youth that support physical, educational and social development.  The Centre for Community Based Research(CCBR) is a non-profit organization focused on social research to strengthen communities(Centre for Community Based Research, 2009, p.3).  The evaluation looks specifically at the BGCC program Get B.U.S.Y, which focuses on healthy eating, physical activity, as well as youth leadership.  Ten Boys and Girls Clubs, in various communities across Canada received funding to run the Get B.U.S.Y program, and 9 of those clubs participated in the evaluation. 

The CCBR began this evaluation process with a kind of needs assessment to set evaluation priorities.  They met with the clubs involved in the Get B.U.S.Y program and developed a Logic Model to establish activities, outputs and outcomes for the program.  The Get B.U.S.Y Logic Model ( CCBR, 2009,p.7) clearly outlines the aims of the program and illustrates the focus for evaluation, including increased knowledge of benefits of physical activity, eating and youth pursuing their own goals within the program.  
The evaluation conducted was formative as the researchers sought to use information gathered to inform the continued implementation of the Get B.U.S.Y program in other Boys and Girls clubs, and also to guide future evaluations to be conducted by the CCBR for the Boys and Girls Clubs of Canada.  The evaluation is largely goal-based.  The Boys and Girls clubs, with the CCBR, set out goals in the Logic Model they created, and the evaluation was used to assess the effectiveness of the program to attain the goals.  
The initial plan for the evaluation included 15 tools for collecting data such as surveys and questionnaires for clients and staff, attendance forms, and program materials like journals and calendars.  Based on feedback from the participating Boys and Girls Clubs that they could not realistically administer all these tools, the CCBR simply used pre and post surveys for clients and a staff activity log(CCBR, 2009, p8-9)
Strengths
The Get B.U.S.Y Program Evaluation Report has many strengths.  The 23 page report followed proper formating and was easy to navigate.  The report was very thorough in providing background information on the Boys and Girls Clubs of Canada, the Centre for Community Based Research and the Get B.U.S.Y program itself.  I could easily understand and envision what the program would look like in action.  
The evaluation plan was clearly described.  In the report, it states that the researchers had intended to provide 15 different tools to collect data.  I appreciate the inclusion of this fact because otherwise I may have questioned why they only used pre and post surveys and staff logs.  As a teacher, I understand the feeling of despair when additional paperwork is pilled on, so I respect the researchers’ decision to scale back to 3 tools. 
I believe that this program evaluation was conducted to assess the Get B.U.S.Y program, but intentionally or not, to inform the practices of the CCBR going forward in conducting research and evaluation for the Boys and Girls Clubs of Canada. In the analysis portion of the report, the researchers seem to indicate that the Get B.U.S.Y program effectively meets the program outcomes.  The real “learning” from this evaluation seems to be more centered on how the CCBR collects data from the Boys and Girls Clubs.  The report comments on the rewriting of questions to make them simpler and more kid (client) friendly, and on the realization that the completion of evaluation materials took up time that should have been spent on the program itself.  I don’t know if this kind of metacognitive evaluation is common in program evaluation, but as a learner I enjoyed reading their thought process.  
I also like the inclusion of statistical data in tables and quotes from staff and clients to provide a more personal view.  
Weaknesses
There are some weaknesses in this program evaluation report as well.  First, the report was written in an informal voice.  It made it a quick and simple read for me as an outsider, but this report is being presented to a high profile, not for profit, Canada wide organization, so I would have expected it to be more formal.  
While I appreciated of the description of original plan with 15 evaluation tools, and the reflections in the report about issues with data collection (not all materials were returned), it made me question the professionalism of the CCBR.  Perhaps because I work with children, I understand what is possible, but I thought it was strange that only after the data collection was complete did the researchers think it was important to use simple words and questions to survey clients from 8-17 years old.  
Lastly, I found that the evaluation report included many more recommendations for future evaluation practices and considerations and very few recommendations for the program itself.



Centre for Community Based Research. (2009).  Get B.U.S.Y Evaluation Report. Retrieved from http://www.bgccan.com/fr/ClubsPrograms/Programs-National/Documents/Get%20Busy%20Final%20Report%20Pilot%201.pdf