Image based research and evaluation

Images are all around us; we are all image-makers and image readers. Images are a rich source of data for understanding the social world and for representing our knowledge of that social world. Image-based research has a long history in cultural anthropology and sociology as well as the natural sciences, but is nonetheless still relatively uncommon.

This chapter, Seeing is Believing, describes imaged based research and evaluation and focuses especially on issues of credibility of images and image based inquiry strategies.

There are a few examples in this chapter from my research on the impact of high stakes testing. Data collection focusing on kids’ experiences of testing involved drawing and writing. You can see more of these data on my website, as well as view a presentation I did on this topic for the Claremont Graduate School 2006 summer institute on the credibility of evidence.

Tracy%2C%20Savannah%2C%20and%20Faith%20urban4_1.jpg

The Complexity of Causation

In these times of ever greater technological sophistication there is a presumption that complexity and erudition will lead to true knowledge about the way things work, will identify unequivocally what causes what.

Michael Scriven has written a very nice piece on the logic of causation–a more complex and sophisticated notion for sure, but not because of the use of complex methods like randomized clinical trials. Indeed, Scriven describes the rather ordinary notion of observation as key to discerning causation. He reminds us that even preschoolers, in some contexts, know perfectly well what can cause what.

While Scriven does not speculate about why there is such romanticism about experimental design, this seems worthy of analysis. One side of globalism is the invocation of elite authorities to determine what is right and good. The economic (and therefore political and cultural) imperative is used to justify the few making decisions for the many. (If you can stand it, read Thomas Friedman’s The World is Flat to get a sense of this thinking.) Suggesting that believable causal claims ensue from only RCTs suggests only a special class of people with the knowledge, ways and means to do this sort of research have knowledge worthy of being shared. While Scriven calls for cooperation among the camps of causation warriors, logic alone is unlikely to win the day.

On the ranking of schools in BC

It is nice to see the Georgia Straight reporting on schooling issues with this week’s story on the debate between Paul Shaker (Dean of Education at SFU) and Peter Crowley of the Fraser Institute.

Shaker challenges the overuse and over interpretation of FSA scores as indicators of the quality of schools. Although lots of numbers may look scientific, Shaker asserts the Fraser Institute approach would never stand up under serious peer review.

Read the Georgia Straight article here.

Accountabilism–eaten alive by attempted precision

Read on for an interesting characterization of accountability gone mad–the over-reaching of precision which ultimately destroys…From the February 2007 issue of Harvard Business Review:

The Folly of Accountabalism

Accountability has gone horribly wrong. It has become “accountabalism,” the practice of eating sacrificial victims in an attempt to magically ward off evil. The emphasis on accountability was an understandable response to some god-awful bookkeeping-based scandals. But the notion would never have evolved from a buzzword into the focus of voluminous legislation if we hadn’t also been lured by the myth of precision: Because accountability suggests that there is a right and a wrong answer to every question, it flourishes where we can measure results exactly. It spread to schools—where it is eating our young—as a result of our recent irrational exuberance about testing, which forces education to become something that can be measured precisely. When such disincentives as the threat of having to wear an orange jumpsuit for eight to ten years didn’t stop the Enron nightmare and other bad things from happening, accountabalism whispered two seductive lies to us: Systems go wrong because of individuals; and the right set of controls will enable us to prevent individuals from creating disasters. Accountabalism is a type of superstitious thinking that allows us to live in a state of denial about just how little control we individuals have over our environment.

Accountabalism manifests itself in a set of related beliefs and practices: It looks at complex systems that have gone wrong for complex reasons and decides the problem can be solved at the next level of detail. Another set of work procedures is written, and yet more forms are printed up. But businesses are not mechanical, so we can’t fine-tune them by making every process a well-regulated routine. Accountabalism turns these complex systems into merely complicated systems, sacrificing innovation and adaptability. How can a company be agile if every change or deviation requires a new set of forms?

Accountabalism assumes perfection—if anything goes wrong, it’s a sign that the system is broken. That’s not true even of mechanical systems: Entropy, friction, and manufacturing tolerances ensure that no machine works perfectly. Social systems are incapable of anything close to perfection, so if something goes wrong in one, that need not mean the system is broken. If an employee cheats on expenses by filling in taxi receipts for himself, the organization doesn’t have to “fix” the expense-reporting system by requiring that everyone travel with a notary public.

Accountabalism is blind to human nature. For example, it assumes that if we know we’re being watched, we won’t do wrong—which seriously underestimates the twistiness of human minds and motivations. We are capable of astounding degrees of self-delusion regarding the likelihood of our being caught. Further, by overly formalizing processes, accountabalism refuses to acknowledge that people work and think differently. It eliminates the human variations that move institutions forward and provide a check on the monoculture that accounts for most disastrous decisions. It also makes work no fun.

Accountabalism bureaucratizes and atomizes responsibility. While claiming to increase individual responsibility, it drives out human judgment. When a sign-off is required for every step in the work flow, those closest to a process lack the leeway to optimize or rectify it. Similarly, by assuming that an individual’s laxness caused a given problem—if so-and-so hadn’t been asleep at the switch or hadn’t gotten greedy or hadn’t assumed that somebody else would clean up the mess, none of this would have happened—accountabalism can miss systemic causes of failure, even, ironically, as it responds to the problem by increasing the system’s reach.

Accountabalism tries to squeeze centuries of thought about how to entice people toward good behavior and dissuade them from bad into simple rules by which individuals can be measured and disciplined. It would react to a car crash by putting stop signs at every corner. Bureaucratizing morality or mechanizing a complex organization gives us the sense that we can exert close control. But grown-ups prefer clarity and realism to happy superstition.

David Weinberger (self@evident.com), a marketing consultant and a coauthor of The Cluetrain Manifesto: The End of Business as Usual (Perseus, 2000), is also a research fellow at Harvard Law School’s Berkman Center for Internet & Society in Cambridge, Massachusetts. His book Everything Is Miscellaneous will be published in May by Times Books.

Short definitions for realism

At the heart of the quantitative-qualitative debates in evaluation (which have been muted but not resolved in the field) lie fundamentally different notions about the world–different ontological perspectives. This debate often dissolves into caricatures of ontological positions, a particularly common one being using a paint brush that colours every neo-positivist a “realist,” by which is meant naive realism. Of course, most contemporary realists are not ‘naive’ and subsribed to a more nuanced sense in which there is a real knowable world out there. Click here for a little more of that nuance.

If only school was this interesting…

9daydreaming.jpg
These are not kids in a classroom preparing to take whatever state mandated tests in whatever state. The world Maira Kalman creates in her illustrations and poetry is funny, sad, stunningly prophetic, and whimsical. This classroom reflects a decidedly pre-NCLB utopia, that never existed, but perhaps dwells in our minds sufficiently to remember there is an alternative.

Note: Maira Kalman’s books have been read at my house for years, by adults and children alike. She has lightened the instructions for good English in an illustrated version of Strunk and White

Call for Proposals for New Directions for Evaluation

New Directions for Evaluation

Sandra Mathison, Editor-in-Chief

0787983942.jpgNew Directions for Evaluation (NDE), is a peer-reviewed journal that publishes empirical, methodological, and theoretical works on all aspects of evaluation. An NDE issue contains between 37,500 and 42,500 words, and usually consists of a brief editorial introduction and 6 to 8 chapters that address and develop the topic, method, or theme. Typically, an issue includes several authors, but single or co- authored issues will be considered. All proposals will be considered, and we especially encourage proposals that focus on the following topics:

History of evaluation: What is the genesis of evaluation as a discipline, as a profession, as a practice? How has evaluation evolved over time?
Worth of evaluation: Evaluation is difficult and costly. Do the benefits of evaluation warrant the costs, real and opportunity? What is the value of evaluation?
Human rights and evaluation: Much evaluation is focused on human service programs providing services to individuals and communities. How does evaluation effect, protect, acknowledge human rights? Are strategies like ethics review boards useful and appropriate in evaluation?
Causation and evaluation: What are the frameworks for understanding and establishing causation in evaluation? How can and should attribution of effects be made?
Qualitative-quantitative debate in evaluation: How has this debate effected evaluation methodology? Does this debate mask poor quality in either or both? What does the advocacy of mixed methods contribute to this debate?
Evaluation of complex systems: What can evaluation contribute, and how, in increasingly complex systems of program delivery (interconnections across levels of government, complex social systems, partnerships)?

Other topics of interest:

o evaluation in the small non-profit sector
o organizational self evaluation
o quantitative methods in evaluation (e.g. non-parametric statistics, HLM, latent trait theory)
o representation and reporting in evaluation (e.g. visual imagery, media, effective formats)

o conflict and dispute resolution and evaluation
o internal evaluation
o uses of technologies in evaluation (e.g. software, iPods, hand held devices, monitors, surveillance)
o evaluation of technologies (e.g. online learning, computers, internet)

More details about submitting proposals for NDE can be found at http://www.eval.org/Publications/NDE.asp

Contact Sandra Mathison at nde@eval.org