Category Archives: Uncategorized

Creating Educative Personal Experiences ~ learning evaluation from the Olympics and other things that happen in your life

In the early 1990s, Wayne Ross and I wrote an article with this title. (The full article is available here.) While we were talking about the role of personal experiences in learning to teach, rereading this article suggests a broader scope to the value of personal experiences in learning just about anything, including evaluation or research. Because evaluation is absolutely everywhere the opportunities to hone or knowledge and skills is limitless. I’ve had fun with the Olympics and evaluation project, revisiting some basic ideas in evaluation and sharing them with you.

Athletes (and evaluators) learn from mistakes.

And they learn from successes.

Fidelity is over-rated… or understanding “hurry, hurry hard”

I couldn’t get through this project of learning about evaluation from the Olympics without a mention of curling. Born on the Canadian prairies, I curl! We curled during phys ed class and as a young adult it was an important context for socializing. Curling is a polite game, winning is important but good sportsmanship is more important ~ players are on their honour and there are no judges or referees. And what other sport has a tradition of all the competitors getting together after the match for rounds of drinks, what is called “broomstacking.” Maybe it’s an easy game to make fun of, but try it and you’ll discover there’s more to it than it seems.

Curling is a sport that has many skills that can be isolated, practice and mastered. Like drawing to the button, or peeling off a guard, or a take out with a roll behind a guard, or throwing hack weight. And there’s learning to know when to sweep and yell at the top of your lungs, “hurry, hurry hard!” Countries relatively new to the sport focus on these skills and demonstrate extraordinary abilities of execution, which is important to winning. But winning the game also requires something more elusive. These teams often confuse fidelity with quality, an all too common mistake in program evaluation. Being able to execute shots with precision is necessary, but not sufficient to win, in either curling or programs.

Strategy is also key in curling and is not so easily mastered through repetitious practice of isolated skills. Curling has been called “chess on ice.” There are aggressive and conservative strategies. Strategy depends in large part on the context ~ factors such as the ice, skill levels, whether you have the hammer (the last rock thrown), and so on. Strategy in program delivery, especially on the ground interpretations and practice, also depends on the context and practitioners use their strategic knowledge to adjust interventions to achieve maximum success. This strategic adjustment must often trade away fidelity to the intervention plan or map, and too frequently this is seen as a failure. Program evaluations sensitive to both programmatic intentions and local variation are more comprehensive and meaningful for understanding how and why programs work, or don’t.

Olympics controversy: slopestyle boarding ~ it’s all about the holistic scoring!

I introduced this series of posts by highlighting the evaluation of snowboarding… but there are multiple events within snowboarding and they do not all use the same evaluation strategy. While many of the specific events use a component evaluation strategy (separate judges looking at different parts of the athletes’ performance), the slope style event uses a holistic evaluation strategy, that is, each of six judges gives a grade from 1 – 100 considering a range of features of the run (including things like creativity, difficulty, execution of tricks, landings) but it is the overall impression that is the primary focus.

Yesterday’s first round of slopestyle boarding introduces us to a number of evaluation issues, but let’s focus on one: holistic scoring isn’t transparent and justifying the evaluative claim can be dodgy.

When top ranked Canadian slopestyler Mark McMorris received a score of 89.25 (which put him 7th) his response was: “It’s a judged sport; what can you do?” Canadian team coach Leo Addington repeated this perspective: “It’s a judged sport, and they saw what they saw and they put down what they thought.” He went on: “It’s always hard to tell without putting each run side by side, and the judging has many criteria – execution, amplitude, use of force, variety, progression. All those things are included in their thoughts … and they’re judging the entire run, completely, all those little things that sometimes we miss or don’t miss. It’s really hard to tell unless you study each one in slow motion all the way through.”

Holistic scoring is common in many evaluation contexts (assessments of student writing, lots of program evaluation) and expert judges (like teachers) assert they know a good performance when they see one without necessarily dissecting the performance. But it is more difficult to justify a holistic score and more difficult to challenge its veracity.

Coach Addington’s suggestion that judging each run in slow motion rather than as it is actually occurring is an interesting, although misguided, suggestion. Of course we see things differently in slow mo (that’s what the sports replay is all about) but that isn’t what is being judged in the case of most sports… what is being judged is the actual, authentic performance and even when replays show an error in judgement (let say about a penalty in hockey or a ball/strike in baseball) that judgement is mostly not overturned. So, the justification for a holistic score can’t be that you change the evaluand in order to make it clearer how you arrive at the judgement.

So how can holistic scoring be improved and justified? Slopestyle is a very recent “sport” and so the accumulation of collective expertise about what counts as a quality performance isn’t well formulated and one imagines that over time the quality of the judging will improve… there will be higher levels of agreement among judges and among judges, coaches and athletes. In fact, in the slopestyle instance, the coaches and judges do have relationships that provide for learning from each other. Again, quoting Coach Addington: “We [coaches and judges] can sit down and discuss and say, ‘What did you see, what did we see?’ Maybe we missed something, and we learn from it and that’s how it’s evolving. They learn our perspective and we learn their perspective.” While the media has mistakenly conjured up an image of fraternization between judges and coaches, they misunderstand that evaluations that are fair, transparent and justifiable are necessarily dependent on just such conversations. Holistic scoring approaches can only be better with the development of greater expertise through experience in knowing what that ideal type, the strong overall impression looks like.

NOTE: For the rest of the snowboarding world and media the real story in snowboarding has been US snowboarder Shaun White’s withdrawal from the slopestyle competition.

a blog post about whether I should be blogging…

The International Studies Association (political science folks) is discussing a proposal to ban Association journal editors, editorial board members and anyone associated with its journals from blogging. Here is the language:

“No editor of any ISA journal or member of any editorial team of an ISA journal can create or actively manage a blog unless it is an official blog of the editor’s journal or the editorial team’s journal,” the proposal reads. “This policy requires that all editors and members of editorial teams to apply this aspect of the Code of Conduct to their ISA journal commitments. All editorial members, both the Editor in Chief(s) and the board of editors/editorial teams, should maintain a complete separation of their journal responsibilities and their blog associations.”

Singling out blogs, but no other social media or letters to the editor or op eds, the ISA asserts that blogging is some how unseemly, that it is a kind of discourse that is not proper professional behavior, and that if one blogs one is likely to sink into some abyss losing a grasp on one’s dignity and respectability.

At best this proposal is quaint, a desire for a past when professors stayed in their offices and wrote for and engaged with their peers through narrow publication channels (like the ISA journals). At worst, this is a draconian effort to challenge academic freedom, to squelch professors’ engagement in public life, and to control access to knowledge. The silliness of this proposal does little to obviate its threat to civic engagement of scholars, both the activist minded and those who understand the world is bigger than the university campus.

Elliot Eisner ~ in memoriam

Elliot Eisner brought the concepts of connoisseurship and criticism from the world of art to enable new ways of thinking about educational evaluation. He died at home on January 10, 2014 and the field of evaluation has lost an important founding thinker.

In 1976, Eisner made an observation that is as true today as it was then,

First, the forms of evaluation that are now employed to assess the effectiveness of school programs have profound consequences upon the character of teaching, the content of curriculum, and the kinds of goals that schools seek to attain. Evaluation procedures, more than a reasoned philosophy of education, influence the educational priorities at work within the schools. Second, these evaluation procedures rest upon largely unexamined assumptions that are basically scientific in their epistemology, technological in their application, and have consequences that are often limited and at times inhospitable to the kinds of goals the arts can achieve.

He went on to describe how connoisseurship and criticism, concepts from the arts through which he conceptualized the artistry of teaching and schooling as a cultural artifact both of which required appreciation (through connoisseurship) and critique (through articulation of the ineffable qualities of teaching, learning and schools).

Eisner’s The Educational Imagination: On the Design and Evaluation of School Programs is a classic.

American Evaluation Association meeting Day 1

Unfortunately I missed the open plenary for the meeting… flight late, baggage AWOL, but did catch the poster session and meet and greet with the evaluation authors.

So great to see so many interesting posters. Such a rich offering of evaluation practice around the world and across all imaginable domains. A special shout out to my students Kristy Jang and Arwa Alkhalaf whose poster focused on the advantages and disadvantages of various technologies in an evaluation they are doing.

Also nice to catch up with evaluation authors–thanks especially to Marv Alkin for his new book, Evaluation Essentials: From A to Z, which I am using in my class and my students are liking it. Marv offers a straightforward and readable introduction, just like the title suggests, and although there is a key education-focused case included in the book, you can easily substitute any case if a different domain of evaluation is more appropriate. And congrats to Michael Patton on Developmental Evaluation, which several of my students are currently reading as well.

Potpourri of streaming videos from Western Michigan’s Evaluation Center

The Evaluation Center’s Evaluation Cafe presentations from last year are online and many can be viewed as streamed video. There is a wide range of topics and many excellent speakers ~ just a few highlights are:

Michael Scriven on causal attribution
Patricia Rogers also on causal attribution
Lois ellin Datta on the appropriateness of RCTs for evaluation