e-valuation

technology and evaluation

Sandra Mathison — Fri, 26 Aug 2011 16:26:36 +0000

There can be no doubt that technology can make the work of evaluation easier and the array of software and applications is evergrowing. Check out the AEA365 blog and search for technology related posts… there are plenty. The challenge for evaluators will be thoughtful use of technology and avoiding technology driven evaluation practices. One of the best examples of technology driving and structuring thinking, knowledge construction and presentation is Powerpoint… Microsoft has created simple software that too often control what counts as information. Edward Tufte’s critique of ppt, PowerPoint Does Rocket Science–and Better Techniques for Technical Reports, is required reading for anyone who has ever or will ever use powerpoint.

Today Apple revealed the availability of, Look For, an iphone app for recording classroom observations of teaching and it is marketed as a tool for teacher evaluation. With a quick click (and some added notes if you like) principals can record whether teachers are “making subject matter meaningful” or “facilitating the learning process.” The promo for Look For says the app has the following features:
-Create unlimited observations
-Sort observations by school, teacher, subject and date
-Select from hundreds of qualification points within 6 basic categories
-Easily email and share reports and progress instantly
-Track teacher progress through each of the 6 instructional categories
-Supports state and national standards

Everyone wants technology to make their lives and jobs easier, and principals are no exception. But is this like ppt? An app that pre-defines and standardizes what counts as good teaching and limits sensitivity to context may be time saving, but does it promote good evaluation? Establishing criteria is key to good evaluation, but this is and ought to be a slippery part of the process… we cannot and should not know all of the relevant criteria a priori and we ought to be open to recognizing good and bad making attributes of teaching in situ. Principals and teachers need to be able to recognize and acknowledge what is not easily or necessarily captured by the 6 instructional categories.

So maybe Look For is a good app, but only if used in a critical way… true for all technology.

data visualization and theinfo.org

Sandra Mathison — Tue, 02 Aug 2011 16:16:17 +0000

theinfo.org is a website that collects info about large data sets… tips and tricks aplenty. There is a section on data visualization that includes great examples, as well as tools for data visualization. The Baby Name Voyager is a fun one but there are lots of serious examples too.

As an aside, one guy behind theinfo.org is Aaron Swartz, currently under indictment for hacking into MIT’s servers and downloading more than 4.8 million articles from JSTOR. To give them credit, JSTOR did not bring charges, MIT did. It is an interesting piece in the open access conversation particularly and the commodification of knowledge in the academy more generally. Read more here. And Swartz’s manifesto follows…

capitalist Bill Gates may do what other capitalists could not

Sandra Mathison — Mon, 23 May 2011 00:50:07 +0000

If you doubt that neo-liberalism dominates the educational reform landscape take a look at this NYT story, which does a nice job of following the Gates Foundation money and how the spending has influenced the adoption of national curriculum standards, worked against teacher unions, and infiltrated school districts, think tanks and even the unions. This strategy is not new and is exactly what right wing Christian groups have done to influence schools by getting elected/appointed to school board positions. In both cases, there is something unsavory about the subterfuge, the lack of transparency, the buying of influence… as opposed to public deliberations about the schools we want and how to get them.

some ideas for stakeholder engagement

Sandra Mathison — Mon, 16 May 2011 20:02:53 +0000

While this document from the IBM Center for the Business of Government Using Online Tools to Engage The Public, focuses on strategies for governments to engage the public in issues, there are a number of strategies and ideas that make sense for evaluators thinking about how to engage stakeholders in the evaluation process. The discussion focuses on the use of online strategies, a particularly useful focus when evaluators are working across sites and geographic areas.

There are 10 strategies in all, 4 for collaboration, 2 for surveying perceptions and opinions, and 4 for prioritizing ideas offered, complete with specific suggestions for software and an illustrative example of each.

rights-based evaluation

Sandra Mathison — Thu, 21 Apr 2011 19:20:37 +0000

Rights-based evaluation is not a new idea and has been a prominent way of thinking in the international development world for at least a decade. A broad framework for human rights is established by a number of declarations including: the UN’s 1948 Universal Declaration of Human Rights and six core human rights treaties: the International Covenant on Civil and Political Rights; the International Covenant on Economic, Social and Cultural Rights; the Convention on the Rights of the Child; the Convention against Torture and other Cruel, Inhuman or Degrading Treatment or Punishment; the International Convention on the Elimination of All Forms of Racial Discrimination; and the Convention on the Elimination of All Forms of Discrimination against Women. (Each of these declarations can be found at the UNICEF website.)

In evaluation we consider needs assessment to be a key tool, and this can be relatively easily understood as a rights assessment, especially when we understand that rights are what is essential to live as human beings, the basic standards without which people cannot survive and develop with dignity. Human rights are thus inherent to the person, inalienable and universal. People may still need more than they have a right to, but this would seem a fundamental baseline.

A useful, short guide to the idea of a rights-based approach for program development, as well as monitoring and evaluation is Applying a Rights-based Approach: An Inspirational Guide for Civil Society.

Rights-based evaluation requires evaluators to be committed to general principles as key sources of criteria in evaluating; to taking a socio-cultural perspective and not just an inside a program box view; and requires evaluators to give as much time to the question of whether or not the right job is being done and not focusing only on whether the job is being done right.

need a logic model, look no further

Sandra Mathison — Thu, 21 Apr 2011 17:28:48 +0000

I am certain logic models can be and sometimes are useful, although I’m not a big fan. The Nonprofit Quarterly provides a humorous generic model, generically titled: Logical Model for Creating Achievable and Sustainable Change Modalities Over Time.

formative evaluation and the Wisconsin protests

Sandra Mathison — Wed, 02 Mar 2011 16:49:19 +0000

Sam Culbert in this NYT op ed identifies the typical performance review as ‘subjective,’ by which he means unfair and capricious. Alternatively, he suggests that personnel evaluation should focus on goal setting and continuous improvement. In other words, personnel evaluation should be formative evaluation. Culbert correctly identifies the often poor quality of personnel evaluation and doing formative evaluation does make sense for almost all employees. The Wisconsin union protests speak to the necessary engagement of unions in creating public (and private) workplaces where performance review might serve both individual and collective needs.

getting to formative

Sandra Mathison — Mon, 28 Feb 2011 17:44:07 +0000

Just as it makes little sense to talk about the validity of a test, it makes little sense to talk about a formative test. Although there is a good emphasis on formative assessment of student learning, there is an unfortunate confusion about what the formative means. Too often the instrument is identified as formative, when in fact it is how the information from the instrument is used that makes the evaluation formative. The same test and the results of that test can be used either formatively or summatively. Just as the test is not valid (it is the inferences that are made that have or lack validity), neither is the test itself formative or summative. Popham has a nice little discussion of this in his Ed Week piece Formative Assessment–A Process, Not a Test.

the complexity of teacher evaluation

Sandra Mathison — Tue, 08 Feb 2011 16:49:00 +0000

Good report on the flaw in some current thinking about teacher evaluation from NEPC. The value added approach is a simplistic strategy for determining teaching effectiveness.

Due Diligence and the Evaluation of Teachers
by Derek C. Briggs, Ben Domingue
February 8, 2011

The research on which the Los Angeles Times relied for its August 2010 teacher effectiveness reporting was demonstrably inadequate to support the published rankings. Using the same L.A. Unified School District data and the same methods as the Times, this study probes deeper and finds the earlier research to have serious weaknesses.

American Evaluation Association meeting Day 1

Sandra Mathison — Thu, 11 Nov 2010 15:33:42 +0000

Unfortunately I missed the open plenary for the meeting… flight late, baggage AWOL, but did catch the poster session and meet and greet with the evaluation authors.

So great to see so many interesting posters. Such a rich offering of evaluation practice around the world and across all imaginable domains. A special shout out to my students Kristy Jang and Arwa Alkhalaf whose poster focused on the advantages and disadvantages of various technologies in an evaluation they are doing.

Also nice to catch up with evaluation authors–thanks especially to Marv Alkin for his new book, Evaluation Essentials: From A to Z, which I am using in my class and my students are liking it. Marv offers a straightforward and readable introduction, just like the title suggests, and although there is a key education-focused case included in the book, you can easily substitute any case if a different domain of evaluation is more appropriate. And congrats to Michael Patton on Developmental Evaluation, which several of my students are currently reading as well.