Just as it makes little sense to talk about the validity of a test, it makes little sense to talk about a formative test. Although there is a good emphasis on formative assessment of student learning, there is an unfortunate confusion about what the formative means. Too often the instrument is identified as formative, when in fact it is how the information from the instrument is used that makes the evaluation formative. The same test and the results of that test can be used either formatively or summatively. Just as the test is not valid (it is the inferences that are made that have or lack validity), neither is the test itself formative or summative. Popham has a nice little discussion of this in his Ed Week piece Formative Assessment–A Process, Not a Test.
Follow
the complexity of teacher evaluation
Good report on the flaw in some current thinking about teacher evaluation from NEPC. The value added approach is a simplistic strategy for determining teaching effectiveness.
Due Diligence and the Evaluation of Teachers
by Derek C. Briggs, Ben Domingue
February 8, 2011
The research on which the Los Angeles Times relied for its August 2010 teacher effectiveness reporting was demonstrably inadequate to support the published rankings. Using the same L.A. Unified School District data and the same methods as the Times, this study probes deeper and finds the earlier research to have serious weaknesses.
American Evaluation Association meeting Day 1
Unfortunately I missed the open plenary for the meeting… flight late, baggage AWOL, but did catch the poster session and meet and greet with the evaluation authors.
So great to see so many interesting posters. Such a rich offering of evaluation practice around the world and across all imaginable domains. A special shout out to my students Kristy Jang and Arwa Alkhalaf whose poster focused on the advantages and disadvantages of various technologies in an evaluation they are doing.
Also nice to catch up with evaluation authors–thanks especially to Marv Alkin for his new book, Evaluation Essentials: From A to Z, which I am using in my class and my students are liking it. Marv offers a straightforward and readable introduction, just like the title suggests, and although there is a key education-focused case included in the book, you can easily substitute any case if a different domain of evaluation is more appropriate. And congrats to Michael Patton on Developmental Evaluation, which several of my students are currently reading as well.
spicy nodes ~ a tool for mapping ideas
Spicy Nodes is a web development tool, but some of my students have used it to create concept maps, in developing research questions, and keeping track of layered information about research constructs. This might be an alternative to the linear logic model formats.
Schools We Want ~ panel discussion
Chomsky on Dewey… what might a “good” school look like
free app for managing references
what is impact evaluation?
Thanks to google and a question raised in my program evaluation class, here are some short definitions and links to more elaboration. As I said in class, impacts sound like outcomes to me (perhaps longer term ones), but I see that the term is appropriated by the causation through experimentation evaluation crowd.
****************************************************
Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones.
en.wikipedia.org/wiki/Impact_evaluation
A type of outcome evaluation that focuses on the broad, long-term impacts or results of program activities. For example, an impact evaluation could show that a decrease in a community’s crime rate is the direct result of a program designed to provide community policing.
www.ojp.usdoj.gov/BJA/evaluation/glossary/glossary_i.htm
A form of outcome evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program.
www.epa.gov/evaluate/glossary/i-esd.htm
Sometimes associated with the establishment of causal attribution between a development Activity and immediate, intermediate or longer-term outcomes or results.
nzaidtools.nzaid.govt.nz/evaluation-and-activity-cycle/definitions
measures the programs causal effect on the outcomes it seeks to achieve. The specific technique for estimating impacts varies according to the setting but all methodologies center on ways of constructing a plausible comparison or control group to establish the counterfactual.
www.absoluteastronomy.com/topics/Social_safety_net
a scientifically rigorous methodology to establish a causal association between programs and what they aimed to achieve beyond the outcomes on individuals targeted by the program(s).
www.globalhivmeinfo.org/DigitalLibrary/Digital%20Library/Glossary%20of%20Monitoring%20and%20Evaluation%20Terms.doc
It is usually undertaken at the end of an intervention to determine overall achievements in relation to the stated objectives. It attempts to establish a cause and effect relationship to show the extent to which the intervention has resulted in changing the circumstances of the target beneficiaries.
www.cop-mfdr-africa.org/forum/attachment/download
A study of the attribution of changes in the outcome to the intervention. Impact evaluations have either an experimental or quasi-experimental design.
www.3ieimpact.org/userfiles/file/Impact%20Evaluation%20Glossary.doc
The folly of ‘value-added’ teacher evaluation
Gotta love Alfie Kohn, and in this Huffington Post article he clearly illustrates the problems with the value added (meaning increased test scores) approach to evaluating teacher performance. But Kohn’s article also points out common misconceptions in many evaluation contexts, like uniformity is the same as quality and value is easily perverted or narrowed by the indicators selected.
new AFT blog: What Should Count
A new blog from the AFT, What Should Count, that is self described as follows:
The American Federation of Teachers believes that accountability should be about making sure students have resources to learn and succeed: rich curricula, excellent facilities, talented—and well-supported—faculty, and robust academic standards that are devised and improved by the people who deliver them. This website is designed to serve not only as a clearinghouse of accountability initiatives at the international, national, state and local levels, but also as a starting point for discussing accountability systems that best help our students succeed.
Time will tell whether the AFT contributes positively to the discourse on assessment K-16, but they do have some atoning to do, so this may be a positive start. With Albert Shanker as president, the AFT embraced the standards and assessment reform that began with A Nation at Risk and supported testing new teachers. Initially the AFT neither endorsed or opposed NCLB, but Sandra Feldman’s 2003 comments suggest an endorsement:
The federal NCLB Act poses yet another test of our ability to be con- structive, responsive, and creative while simultaneously fighting and protecting against the indefensible. The law is built around goals we’ve long supported: high academic standards and achievement, eradicating achievement gaps between the haves and the have-nots, making sure that every teacher in every school is qualified, and, yes, accountability. The law also mandates reporting outcomes by student subgroup which is the right thing to do because it puts inequities out there for all to see. (Feldman, S. (2003, July). Keynote address to the AFT QuEST (Quality Education Standards in Teaching) conference, Washington, DC.)
AFT’s opposition to test driven reform has, however, been sharper in recent years in response to pressure from its rank and file members. The AFT’s conservatism and strategy of working behind the scenes doesn’t obviously position the teacher union that represents most urban school teachers as a force for change. Current president Randi Weingarten’s testimony before the House Committee on Education and Labor suggests a continued support for national standards, standardization, and using test scores (at least in part) in determining teacher pay.
So, good for the AFT for creating this blog, but here’s hoping they do much more to contribute to a quality work life for their members, and quality education for children living in US cities.