AEA’s Values – Walking the Talk with Sandra Mathison

AEA’s Values – Walking the Talk with Sandra Mathison, NDE Editor

AEA’s Values Statement
The American Evaluation Association values excellence in evaluation practice, utilization of evaluation findings, and inclusion and diversity in the evaluation community.

i. We value high quality, ethically defensible, culturally responsive evaluation practices that lead to effective and humane organizations and ultimately to the enhancement of the public good.
ii. We value high quality, ethically defensible, culturally responsive evaluation practices that contribute to decision-making processes, program improvement, and policy formulation.
iii. We value a global and international evaluation community and understanding of evaluation practices.
iv. We value the continual development of evaluation professionals and the development of evaluators from under-represented groups.
v. We value inclusiveness and diversity, welcoming members at any point in their career, from any context, and representing a range of thought and approaches.
vi. We value efficient, effective, responsive, transparent, and socially responsible association operations.

******************************************************************************************************************************************************************

I am a Professor of Education at the University of British Columbia where I continue my lifelong engagement in learning and teaching about evaluation. I am also currently Editor-in-Chief of New Directions for Evaluation, have served on various AEA committees, and have been a member of the AEA Board of Directors.

As evaluation theory and practice have taken root and spread around the globe it is exciting to see AEA adopting a Values Statement that reflects the importance of our intellectual and moral obligations as an organization and a profession. As Editor-in-Chief of one of AEA’s journals, I see these values as critical signposts for fostering the involvement, at many levels, of diverse perspectives. These values are important in how I do my work, which includes assembling a team of associate editors and editorial board members that has a global reach and reflects the cultural diversity that AEA’s values encourage. These values are also important in shaping the content of the journal as I have worked to include different points of view about evaluation; the perspectives of experienced and novice evaluators; and perspectives of evaluators from all parts of the world.

AEA’s Values Statement is also an important anchor for me individually, in doing evaluation as well as teaching about evaluation. From the beginning of my career until the present I have been guided by AEA’s values. I strive to foster inclusion through participatory approaches, practice in ethically defensible ways through transparency and thoughtfulness, and to do evaluation in the service of both clients and the greater good.

I contribute to AEA’s value of continual development of evaluators through my teaching and mentoring of graduate students as they learn the craft of evaluation. I am aware at all times of not simply transmitting knowledge and skills to the next generations of evaluators, but also the importance of imbuing novice evaluators with the foundational values on which AEA stands. As a professional organization, AEA provides useful guidance and reminders that evaluation is much more than a technical practice; it is also a moral and values-laden one. In my teaching I am eager for students to see themselves as ethically engaged, open to and engaged with many forms of diversity, and to see their future work as evaluators as meaningful and useful in building a better world.

teacher evaluation: a meta-evaluation of 10 approaches

The emphasis on assessing student learning through large scale standardized and high stakes testing programs has not been an effective strategy for school reform and improvement. No real surprise there. Reform has turned on teacher performance, often guaged by the very tests that are so fraught with socio-political and technical problems. Too often moving onto another strategy is an ideological move with precious little evidence one way or the other. While there are problems with this study that looks at how teacher evaluation is conceptualized and conducted in these 10 places, the effort to engage in evaluation of the evaluations is laudable.

technology and evaluation

There can be no doubt that technology can make the work of evaluation easier and the array of software and applications is evergrowing. Check out the AEA365 blog and search for technology related posts… there are plenty. The challenge for evaluators will be thoughtful use of technology and avoiding technology driven evaluation practices. One of the best examples of technology driving and structuring thinking, knowledge construction and presentation is Powerpoint… Microsoft has created simple software that too often control what counts as information. Edward Tufte’s critique of ppt, PowerPoint Does Rocket Science–and Better Techniques for Technical Reports, is required reading for anyone who has ever or will ever use powerpoint.

Today Apple revealed the availability of, Look For, an iphone app for recording classroom observations of teaching and it is marketed as a tool for teacher evaluation. With a quick click (and some added notes if you like) principals can record whether teachers are “making subject matter meaningful” or “facilitating the learning process.” The promo for Look For says the app has the following features:
-Create unlimited observations
-Sort observations by school, teacher, subject and date
-Select from hundreds of qualification points within 6 basic categories
-Easily email and share reports and progress instantly
-Track teacher progress through each of the 6 instructional categories
-Supports state and national standards

Everyone wants technology to make their lives and jobs easier, and principals are no exception. But is this like ppt? An app that pre-defines and standardizes what counts as good teaching and limits sensitivity to context may be time saving, but does it promote good evaluation? Establishing criteria is key to good evaluation, but this is and ought to be a slippery part of the process… we cannot and should not know all of the relevant criteria a priori and we ought to be open to recognizing good and bad making attributes of teaching in situ. Principals and teachers need to be able to recognize and acknowledge what is not easily or necessarily captured by the 6 instructional categories.

So maybe Look For is a good app, but only if used in a critical way… true for all technology.

data visualization and theinfo.org

theinfo.org is a website that collects info about large data sets… tips and tricks aplenty. There is a section on data visualization that includes great examples, as well as tools for data visualization. The Baby Name Voyager is a fun one but there are lots of serious examples too.

As an aside, one guy behind theinfo.org is Aaron Swartz, currently under indictment for hacking into MIT’s servers and downloading more than 4.8 million articles from JSTOR. To give them credit, JSTOR did not bring charges, MIT did. It is an interesting piece in the open access conversation particularly and the commodification of knowledge in the academy more generally. Read more here. And Swartz’s manifesto follows…

capitalist Bill Gates may do what other capitalists could not

If you doubt that neo-liberalism dominates the educational reform landscape take a look at this NYT story, which does a nice job of following the Gates Foundation money and how the spending has influenced the adoption of national curriculum standards, worked against teacher unions, and infiltrated school districts, think tanks and even the unions. This strategy is not new and is exactly what right wing Christian groups have done to influence schools by getting elected/appointed to school board positions. In both cases, there is something unsavory about the subterfuge, the lack of transparency, the buying of influence… as opposed to public deliberations about the schools we want and how to get them.

some ideas for stakeholder engagement

While this document from the IBM Center for the Business of Government Using Online Tools to Engage The Public, focuses on strategies for governments to engage the public in issues, there are a number of strategies and ideas that make sense for evaluators thinking about how to engage stakeholders in the evaluation process. The discussion focuses on the use of online strategies, a particularly useful focus when evaluators are working across sites and geographic areas.

There are 10 strategies in all, 4 for collaboration, 2 for surveying perceptions and opinions, and 4 for prioritizing ideas offered, complete with specific suggestions for software and an illustrative example of each.

rights-based evaluation

Rights-based evaluation is not a new idea and has been a prominent way of thinking in the international development world for at least a decade. A broad framework for human rights is established by a number of declarations including: the UN’s 1948 Universal Declaration of Human Rights and six core human rights treaties: the International Covenant on Civil and Political Rights; the International Covenant on Economic, Social and Cultural Rights; the Convention on the Rights of the Child; the Convention against Torture and other Cruel, Inhuman or Degrading Treatment or Punishment; the International Convention on the Elimination of All Forms of Racial Discrimination; and the Convention on the Elimination of All Forms of Discrimination against Women. (Each of these declarations can be found at the UNICEF website.)

In evaluation we consider needs assessment to be a key tool, and this can be relatively easily understood as a rights assessment, especially when we understand that rights are what is essential to live as human beings, the basic standards without which people cannot survive and develop with dignity. Human rights are thus inherent to the person, inalienable and universal. People may still need more than they have a right to, but this would seem a fundamental baseline.

A useful, short guide to the idea of a rights-based approach for program development, as well as monitoring and evaluation is Applying a Rights-based Approach: An Inspirational Guide for Civil Society.

Rights-based evaluation requires evaluators to be committed to general principles as key sources of criteria in evaluating; to taking a socio-cultural perspective and not just an inside a program box view; and requires evaluators to give as much time to the question of whether or not the right job is being done and not focusing only on whether the job is being done right.

formative evaluation and the Wisconsin protests

Sam Culbert in this NYT op ed identifies the typical performance review as ‘subjective,’ by which he means unfair and capricious. Alternatively, he suggests that personnel evaluation should focus on goal setting and continuous improvement. In other words, personnel evaluation should be formative evaluation. Culbert correctly identifies the often poor quality of personnel evaluation and doing formative evaluation does make sense for almost all employees. The Wisconsin union protests speak to the necessary engagement of unions in creating public (and private) workplaces where performance review might serve both individual and collective needs.