constructing a good life through the exploration of value and valuing
Working through an evaluation project (the planning, data collection & analysis, reporting) is a bit like a design process. These images are from architect Jody Brown, who has a pretty good handle on the generic challenges of designing and executing pretty much any project. He is a frequent contributor on Houzz.
A good stop for resources on YPE is Act for Youth. YPE is described as:
an approach that engages young people in evaluating the programs, organizations, and systems designed to serve them. Through YPE, young people conduct research on issues and experiences that affect their lives, developing knowledge about their community that can be shared and put to use. There are different models of YPE: some are completely driven by youth, while others are conducted in partnership with adults.
A list of resources points the reader to other literature on YPE.
If nothing else, we have learned a great deal about what doesn’t work in terms of evaluating schools. The global penchant for using a few outcomes measures just doesn’t do the trick… this is perhaps most obvious in the USA where judging the quality of schools continues to spiral downward from NCLB to Race to the Top, but around the world we see a similar story. And, we see a few counterpoints, such as the success of the Finnish school system where the focus is decidedly not on standardized outcomes on a few measures.
In British Columbia, Canada where education is a decided provincial matter and where provincial politics can actually lead to quite radical shifts in policies and programmatic initiatives, this is a moment of potential change. BC schools have been for many years now held hostage by the scores on the Foundation Skills Assessment (FSA), a test given to all 4th and 7th grade students in the province. Support for the FSA has been eroding over the past several years with a chorus of skeptical teacher, school administrator and school trustee voices.
One initiative, The Great Schools Project, has been developing alternative ideas about school evaluation. The website gives s sense of the GSP platform and a bit more information about the issues can be heard in a segment of a local radio talk show.
Validity, reliability, comparability, and fairness are not just measurement issues, but social values that have meaning and force outside of measurement wherever evaluative judgments and decisions are made.
Messick, S. (1994). The Interplay of Evidence and Consequences in the Validation of Performance Assessments. Educational Researcher, 23(2), 13–23.
The Video Girls for Change project uses the Most Significant Change approach to evaluate programming for girls in various developing countries, and uses video summaries to communicate the findings from the evaluations.
The Most Significant Change technique is a form of participatory monitoring and evaluation that directly involves the voices and perspectives of beneficiaries. Essentially the Most Significant Change process involves the collection of stories of significant change from the field, followed by the systematic selection of the most significant of these stories by panels of designated community members and other stakeholders.
Participatory video is an accessible, flexible medium for recording community stories of change. With InsightShare’s games and excercises and experiential learning approach participants can rapidly learn video skills, allowing people to tell their Most Significant Change stories in a familiar context and to someone they trust. The process itself is fun, direct and the results can be played and reviewed immediately. It also helps to avoid situations where project staff or external evaluators speak on behalf of communities, allowing intended beneficieries to speak for themselves.
When participatory video and the Most Significant Change technique are skilfully brought together, the human stories behind development projects can be captured in an accessible form, even for those with low levels of literacy. These combined methodologies promote peer-to-peer learning, collective reflection, triangulation and wide distribution of these important stories. Participatory video allows for everyone to get involved, contribute, feel, and respond to, other people’s stories and can strengthen community ties and identification with developmental objectives.
Meta-evaluation is obviously a good idea that too often is foregone because of limited resources (especially when the evaluation may often be underfunded). Large, high profile, and expensive evaluations of equally large, high profile and expensive initiatives are more likely to incorporate meta-evaluation to assure that the evaluation meets accepted quality standards, and that evaluative judgements and conclusions are trustworthy. Such an example is the meta-evaluation of the evaluation of the Paris Declaration on Development Aid. The evaluation received the 2012 Outstanding Evaluation Award from the American Evaluation Association. The evaluation looks at how the principles of aid effectiveness have been put into practice by international development partners and the results this is having in developing countries. The international joint evaluation include a synthesis report, 21 country evaluations, 7 donor studies and several thematic reviews.
The meta-evaluation, conducted by M. Q. Patton identifies strengths, weaknesses and lessons learned from this large scale evaluation.
While the strategy of students evaluating professors is common in higher education, this approach is rare in K-12 education. One component of the Measures of Effective Teaching Project at Harvard is just such data. Based on a a decade old survey developed by Ronald Ferguson, an economist at Harvard, a shorter survey had been developed that asks students to describe their classroom instructional climate. Importantly, students (all the way from Kindergarten through high school) are not asked to judge their teachers, but to provide a description of what the classroom environment looks and feels like to them. The survey includes the following kinds of questions:
Caring about students (Encouragement and Support)
> Example: The teacher in this class encourages me to do my best.”
Captivating students (Learning Seems Interesting and Relevant)
> Example: “This class keeps my attention – I don’t get bored.”
Conferring with students (Students Sense their Ideas are Respected)
> Example: “My teacher gives us time to explain our ideas.”
Controlling behavior (Culture of Cooperation and Peer Support)
> Example: “When I am confused, my teacher knows how to help me understand.”
Challenging students (Press for Effort, Perseverance and Rigor)
> Example: “My teacher wants us to use our thinking skills, not just memorize things.”
Consolidating knowledge (Ideas get Connected and Integrated)
> Example: “My teacher takes the time to summarize what we learn each day.”
A recent story in The Atlantic the results are summarized.
The most refreshing aspect of Ferguson’s survey might be that the results don’t change dramatically depending on students’ race or income… But overall, even in very diverse classes, kids tend to agree about what they see happening day after day.
Whether these data should be used in teacher evaluation requires careful consideration, but from a larger evaluative perspective what this demonstrates is the very valuable data that those who are meant to benefit most from programs and interventions can provide. If you ask the right questions, and if you respect their experiences and perspectives.
The Adventures in Evaluation podcasts provide chatty info about a number of issues facing evaluators. The most recent podcast explores the issue of credentialing evaluators from the USA perspective by talking with Jean King.
The podcasts are often accompanied by links to resources and materials relevant to the topic being discussed.
Follow
