Category Archives: Policy, politics

Holding accountability to account

One of the hallmarks of any quality evaluation is that it ought to be subject itself to evaluation. Many evaluation schemes in education, such as the test driven accountability scheme, are not evaluated. The Action Canada Task Force on Standardized Testing has released a report analyzing the place of standardized testing as an accountability measure in Canadian K-12 education systems, using Ontario as a case study focus. “A review of standardized testing in this province and others is not only timely – it’s urgently needed,” says Sébastien Després, a 2012-2013 Action Canada Fellow and co-author of the report.

The Task Force offers four recommendations that could be the heart of an evaluation of accountability schemes in K-12 education across Canada.

Recommendations
We recommend that the Ontario government establish a suitable panel with a balanced and diverse set of experts to conduct a follow-up review of its standardized testing program. In particular:

A. Structure of the tests relative to objectives
i. The panel should review whether the scope of the current testing system continues to facilitate achievement of education system objectives.
ii. The panel should review whether the scale and frequency of testing remains consistent with the Ministry of Education’s objectives for EQAO testing.

B. Impact of testing within the classroom
i. The panel should review the impact on learning that results from classroom time devoted to test preparation and administration.
ii. The panel should review the impact of testing methods and instruments on broader skills and knowledge acquisition.
iii. The panel should review the appropriateness and impact of the pressure exerted by standardized testing on teachers and students.

C. Validity of test results
i. The panel should review whether or not standardized testing provides an assurance that students are performing according to the standards set for them.
ii. The panel should review the impact of measuring progress by taking a limited number of samples throughout a student’s career.

D. Public reporting and use of test results
i. The panel should review the impact of the potential misinterpretation and misuse of testing results data, and methods for ensuring they are used as intended.
ii. The panel should review supplemental or alternative methods of achieving public accountability of the educational system.

Mathison keynotes BC Teacher Federation Annual General meeting

The full story is in the Teacher, the BCTF monthly news magazine.

formative evaluation and the Wisconsin protests

Sam Culbert in this NYT op ed identifies the typical performance review as ‘subjective,’ by which he means unfair and capricious. Alternatively, he suggests that personnel evaluation should focus on goal setting and continuous improvement. In other words, personnel evaluation should be formative evaluation. Culbert correctly identifies the often poor quality of personnel evaluation and doing formative evaluation does make sense for almost all employees. The Wisconsin union protests speak to the necessary engagement of unions in creating public (and private) workplaces where performance review might serve both individual and collective needs.

the complexity of teacher evaluation

Good report on the flaw in some current thinking about teacher evaluation from NEPC. The value added approach is a simplistic strategy for determining teaching effectiveness.

Due Diligence and the Evaluation of Teachers
by Derek C. Briggs, Ben Domingue
February 8, 2011

The research on which the Los Angeles Times relied for its August 2010 teacher effectiveness reporting was demonstrably inadequate to support the published rankings. Using the same L.A. Unified School District data and the same methods as the Times, this study probes deeper and finds the earlier research to have serious weaknesses.

Chomsky on Dewey… what might a “good” school look like

The folly of ‘value-added’ teacher evaluation

Gotta love Alfie Kohn, and in this Huffington Post article he clearly illustrates the problems with the value added (meaning increased test scores) approach to evaluating teacher performance. But Kohn’s article also points out common misconceptions in many evaluation contexts, like uniformity is the same as quality and value is easily perverted or narrowed by the indicators selected.

new AFT blog: What Should Count

A new blog from the AFT, What Should Count, that is self described as follows:

The American Federation of Teachers believes that accountability should be about making sure students have resources to learn and succeed: rich curricula, excellent facilities, talented—and well-supported—faculty, and robust academic standards that are devised and improved by the people who deliver them. This website is designed to serve not only as a clearinghouse of accountability initiatives at the international, national, state and local levels, but also as a starting point for discussing accountability systems that best help our students succeed.

Time will tell whether the AFT contributes positively to the discourse on assessment K-16, but they do have some atoning to do, so this may be a positive start. With Albert Shanker as president, the AFT embraced the standards and assessment reform that began with A Nation at Risk and supported testing new teachers. Initially the AFT neither endorsed or opposed NCLB, but Sandra Feldman’s 2003 comments suggest an endorsement:

The federal NCLB Act poses yet another test of our ability to be con- structive, responsive, and creative while simultaneously fighting and protecting against the indefensible. The law is built around goals we’ve long supported: high academic standards and achievement, eradicating achievement gaps between the haves and the have-nots, making sure that every teacher in every school is qualified, and, yes, accountability. The law also mandates reporting outcomes by student subgroup which is the right thing to do because it puts inequities out there for all to see. (Feldman, S. (2003, July). Keynote address to the AFT QuEST (Quality Education Standards in Teaching) conference, Washington, DC.)

AFT’s opposition to test driven reform has, however, been sharper in recent years in response to pressure from its rank and file members. The AFT’s conservatism and strategy of working behind the scenes doesn’t obviously position the teacher union that represents most urban school teachers as a force for change. Current president Randi Weingarten’s testimony before the House Committee on Education and Labor suggests a continued support for national standards, standardization, and using test scores (at least in part) in determining teacher pay.
So, good for the AFT for creating this blog, but here’s hoping they do much more to contribute to a quality work life for their members, and quality education for children living in US cities.

Hope and Change in Educational Evaluation?

I have to confess I was disappointed when Shepard Fairey developed the posters for Obama’s campaign. Shepard Fairey, before the Obama campaign, a skateboarder and graffiti artist stood for a challenge to authority, war and capitalism. Perhaps best known for the Obey campaign (Andre the Giant has Posse) but also his images directly challenging capitalism. Fairey is a contradictory character–using others copyrighted images but threatening to sue those who use his images. And so maybe creating the icon images for Obama, who is little distanced from the corporate, capitalist interests Fairey’s work critiques, are just part of that contradiction. Fairey himself says he is too corporate for the street artists and too street for the establishment.

But hope and change in education are not what the Obama posse is delivering. In fact, it is intensification of what came before–even higher (and probably national) standards, ‘better’ (and probably national) tests, support for charter schools, and now teacher pay for test results are what is being offered. The initial enthusiasm of new resources for education from the stimulus package is fading in the face of draconian demands from Education Secretary Arne Duncan and it becomes clearer with each speech that Obama is misinformed about the facts and this misinformation is critical to perpetuating the sense of crisis in education. Obama claimed that school drop out rates have increased three-fold when drop out rates have dropped by a third; he claimed that 8th grade achievement has dropped, when in fact it has risen; and set up fake goals–like having the highest proportion of college graduates in the world, when the US is almost there already.

It would have given me hope for change had Obama called for an evaluation of NCLB, even in small ways such as an analysis of NAEP scores pre NCLB and now. It would have given me hope for change if Obama demonstrated some understanding of the inevitable corruptability of high stakes assessment, whether in the form of mc tests or performance tasks. It would have given me hope for change had Obama focused on ways to cultivate options within public schools and promoted magnet schools rather than charter schools.

I don’t recall who said, authority has no wisdom, but methinks we are seeing this is action.

Serving the Public Interest through Educational Evaluation

This is a pre-publication version of a chapter that analyzes the nature of educational evaluation in a global, neo-liberalist world. The chapter includes some advice to evaluators on how to take back evaluation to serve democratic values, that is, the inclusion of all stakeholders (especially those most often shut out) and open deliberation about what it means for education and schooling to be good or bad.

This chapter will appear in Ryan & Cousins’ edited International Handbook of Educational Evaluation to be published by Sage.

Evaluation’s contribution to the economic disaster

Investment ratings by the big three rating companies (Moody’s, Standard & Poors, Fitch’s) are supposed to be the independent evaluative data that provide potential investors with a sound judgement of where to best put their money. Turns out, though, that Moody’s gave high ratings to AIG just before they were bailed out by the US government, and to Lehmann Bros just before they declared bankruptcy. Also turns out that Moody’s (and the other raters) receive big bucks from the companies they are rating, giving at least the appearance of less than independent evaluation and more cynically giving the appearance that companies buy ratings. Moody’s, on the other hand, blames rogue employees diverting attention from these pecuniary relationships. While there may be some incompetent or unethical evaluators at Moody the suggestion this is a one-off diverts attention from the very fundamental connection that exists within our society between capital and goodness. In his article Blowback (American Journal of Evaluation, Vol. 29, No. 4, 416-426 (2008)), Ernie House illustrates similar entanglements in the drug industry, where the appearance of good science masks profit making.

When good becomes synonymous with profit (which is what happens in neo-liberalism) then evaluation CAN serve the profit making (or the ideology that prizes profit making) which becomes the indicator of good. Evaluation no longer gets left to evaluators ~ the drug industry funds its own clinical trials and the capitalists fund their own rating systems to support their end goals. This scenario creeps into educational and social programming evaluation and should set off alarm bells.

It may be a little comforting to see the value of Moody stock dropping.