Federal Panel Considers Giving Colleges a ‘Gentle Shove’ Toward Accountability Testing

The Chronicle: Federal Panel Considers Giving Colleges a ‘Gentle Shove’ Toward Accountability Testing

The chairman of the federal Commission on the Future of Higher Education says it will not propose mandatory testing of college students but will urge colleges and accrediting agencies to voluntarily adopt assessments of student learning, and will recommend two tests.”There is no movement, no intent, no expectation of a mandated test at the federal level,” the chairman, Charles Miller, wrote in an e-mail sent to his fellow commission members this month.

In the e-mail message, titled “Standardized Anxiety,” Mr. Miller predicted that, with college costs on the rise, institutions would come under increasing pressure to prove their worth to taxpayers and students’ parents, and that many colleges would turn to testing. With time, he said, a test will emerge as the gold standard, “not by mandate, but by a normal process of early movers and best practices.”

“I believe the tipping point has already been reached. We need to give it a gentle shove,” he wrote.

The message refutes rumors, which began circulating soon after the commission’s creation in September by Secretary of Education Margaret Spellings, that the federal advisory board was weighing a No Child Left Behind-style testing regime for colleges. The commission, which is made up of 19 business and higher-education leaders, is charged with devising a comprehensive plan for the future of higher education by August 1.

But the commission’s testing recommendations will still be controversial. Critics of college-level testing say that it is best left to professors, that tests should be tied to institutional curricula, and that no test can measure the effects colleges have on students’ social and emotional development.

“You can’t quantify life transformation,” says Mark F. Smith, director of government relations at the American Association of University Professors.

Unknown Investment Returns

The call for increased accountability in higher education is hardly new. For years state and federal policy makers have been pressing colleges and universities to demonstrate their worth. As tuitions have risen, outpacing inflation, many state legislatures and accreditation agencies have tried to create ways to gauge the improvements college educations make in students’ knowledge and skills. Some of these efforts at accountability already use testing to gauge students’ growth.

Yet there remains a nagging sense among many policy makers and some members of the commission that there is no “bottom line” way of knowing what colleges contribute to students’ intellectual development. As one commissioner, Richard K. Vedder, a professor of economics at Ohio University, puts it, “Harvard students are brilliant going in, and they’re brilliant coming out. The question is: How much value did Harvard add?”

The commissioners’ concern stems, in part, from recent studies suggesting that college graduates perform poorly on basic-skills tests such as the National Assessment of Adult Literacy. That survey, last administered by the Department of Education in 2003, found that less than a third of college graduates could read complex texts and make complicated inferences.

Adding to the alarm are reports from business leaders on the commission that college graduates are unprepared for 21st-century jobs. The executives note that American companies spend billions of dollars to give their workers remedial education each year.

Richard Stephens, a commission member who is senior vice president for human resources and administration for the Boeing Company, said testing would allow companies to evaluate job candidates based on how well their alma maters prepare students for the work force.

“We receive two million job applications each year,” he said. “I want to translate where they went to school into value in the marketplace.”

The push for increased accountability is also coming from the Education Department, which provides one-third of higher-education funds. In announcing the commission’s formation, Ms. Spellings said it was “time to examine how we can get the most out of our national investment” (The Chronicle, September 30, 2005).

Commissioners say the message from the department, policy makers, and the public is that higher education can no longer rest on its laurels.

“For years higher education has said that we do something very special that only we can understand,” said another commission member, Robert M. Zemsky, who is chief executive of the Learning Alliance for Higher Education at the University of Pennsylvania. “We can’t do that anymore. An increasing number of people are becoming concerned that it’s all smoke and mirrors.”

Advocating Accountability

The commission’s biggest advocate of accountability is Mr. Miller, a former chairman of the University of Texas System’s Board of Regents and the chief architect of that university’s accountability system, which includes testing to measure student learning. Mr. Miller, a nationally known investor who once managed millions in public funds, says he believes colleges owe it to the people who invest in them to be sure they are using the money effectively.

Mr. Miller worried many college officials when he told reporters at a December meeting in Nashville that the commission might propose tying an institution’s eligibility for federal student aid to standardized testing (The Chronicle, December 9, 2005). “We’re looking for leverage points,” he said. “Funding is one, accreditation is another.”

While most higher-education associations remained silent, lobbyists for independent colleges and universities went on the defensive, warning that standardized testing would homogenize higher education.

“We need to play to the diversity of higher education and not come up with a Prussian, centralized model of how to go forward,” David L. Warren, president of the National Association of Independent Colleges and Universities, said at a commission meeting held in San Diego in February. At least two commission members wrote to Mr. Miller to object to standardized testing, with one, Charlene R. Nunley, president of Montgomery College, saying she was worried about conflicts between the national commission’s recommendations and state efforts.

Recognizing the potential firestorm, Mr. Miller has taken care to clarify his position recently, stressing that he sees markets, not mandates, as the path to testing.

His theory, he explained in a recent interview, is that some colleges will resort to tests to justify their rising tuitions. Once some colleges start publishing test results, their competitors will be pressured to do so as well, driving the cost of testing down. Since policy makers, employers, and the public will want to be able to compare colleges, institutions will gravitate naturally toward a single test, which will then dominate the market.

To give colleges the “gentle shove” the chairman envisions, the commission may urge accreditation agencies to place more emphasis on measuring the effects colleges have on students, or ask the Education Department to require accreditors to do so. Accrediting agencies have been involved in student assessment since the 1980s, when the Education Department directed them to include measures of student learning. But the agencies now often let colleges set their own standards for student assessment.

That approach is much more palatable than mandatory testing to commission members with higher-education backgrounds. If a test proved valid, says David Ward, president of the American Council on Education, “some legitimacy would develop through the marketplace.”

“And that’s much better legitimacy than insisting everyone take it,” he adds.

Mr. Miller stresses, however, that it will be increasingly difficult for higher education to justify requests for increased state and federal aid without providing more data about student learning.

“To make the case for more money,” he said, “you have to show that you’ve done the best you can with what you have.”

Asked if he was still considering tying testing to federal support, Mr. Miller said that he would recommend it only if colleges didn’t develop testing voluntarily, “if there was no progress, or resistance.”

While the commission will not require the use of a single test, it is likely to endorse two: the Collegiate Learning Assessment, or CLA, and the Measure of Academic Proficiency and Progress (see “Sample Questions From 2 Tests”), both of which Mr. Miller highlighted in a memorandum on accountability sent to fellow commission members in January (The Chronicle, February 17).

Mr. Miller is particularly impressed with the CLA, which he described at the commission’s Nashville meeting as “a middle-of-the-fairway, perfect drive.” The test, which was developed by the Council for Aid to Education and the RAND Corporation, has been used in the Texas accountability system and has been administered to 30,000 students since 2002. This year it is being used by 121 institutions — including 35 members of the Council of Independent Colleges, which have signed up voluntarily — and is being adapted for use by community colleges.

The test’s creators say it provides a truer gauge of an institution’s quality than its SAT scores, endowment levels, student-satisfaction scores, or self-reports. However, they stress that the CLA should not be used as a diagnostic tool by itself, but as part of a portfolio of assessments.

The goal of the test, says Stephen P. Klein, a senior research scientist with the RAND Corporation, is to measure the skills “that colleges say are important to them in their mission statements”: critical thinking, analytic reasoning, and writing.

He adds that the test eliminates fears about “teaching to the test” because it does not measure content knowledge, but broad-based skills that all students should possess.

Richard H. Ekman, president of the Council of Independent Colleges, says the CLA allows colleges to show what they contributed to student learning because it evaluates students’ achievement based on where they began and where they ended up. He suggests that if the commission wants to encourage the use of the test, it should ask the Education Department to subsidize the $6,000 cost of the one-year measurement for smaller colleges.

“If you could eliminate the financing problem, that would increase use,” he said.

While the CLA was designed to measure institutional performance, not individual student performance, its creators say it could be modified for use as an exit exam. Students already receive their test results, but colleges receive aggregate results unless given permission by students to receive individual scores.

Will Students Care?

The second test, the MAPP, is the new version of the Educational Testing Service’s Academic Profile, which was retired in January. The test, which is multiple-choice, is also designed for institutional-improvement purposes, and allows colleges to compare themselves with peer institutions of their choosing.

Assessment experts say the emergence of those two tests makes it hard to argue that testing can’t work on the college level.

“It’s almost impossible to kill this thing by saying that what we teach in college is so unmeasurable that there is no way you can assess it,” says Patrick M. Callan, president of the National Center for Public Policy and Higher Education.

Mr. Callan’s group conducted a five-state pilot project that sought to measure the performance of colleges and universities within a state by comparing data on professional-licensing and graduate-admissions tests, the results of the National Assessment of Adult Literacy, and students’ scores on the Collegiate Learning Assessment and another test designed for two-year institutions. The commission is likely to recommend an expansion of the national center’s project in the commission’s final report.

Not everyone is taken with the new tests, however. In an e-mail message to Mr. Miller, Mr. Zemsky, of the University of Pennsylvania, said tests like the CLA are flawed because students do not have a stake in how well they do.

Mr. Zemsky says colleges do not need a new test — they simply need to make better use of the results from the many standardized exams already taken by their students. He proposes that benchmark scores, such as the median for all students who take the Graduate Record Examination, or GRE, be published, so consumers could compare competing institutions based on the relationship between their SAT scores and the scores on exams for graduate schools.

“We’re not short on data, we’re short on data that has been made public in a consistent way,” he said in an interview.

Mr. Miller acknowledges that concerns about motivation are valid, but he says the evidence from the CLA suggests that students are indeed motivated to do well.

“It’s not a fatal flaw,” he said.

‘A Wake-Up Call’

Critics of such tests, like Leon Botstein, president of Bard College, say that the answer to assessment lies not in standardization, but in “rigorous self-policing and peer review” by college faculty members. He calls the controversy over testing “a wake-up call for institutions, to make sure they’re doing their job.”

Indeed, some commission members are skeptical that colleges will ever embrace testing without outside pressure. “If you leave it up to the faculty, things will never get done,” said Arthur J. Rothkopf, a president emeritus of Lafayette College and senior vice president at the U.S. Chamber of Commerce. “Faculty are great on the status quo.”

Meanwhile state-college lobbyists are trying to get out in front of the commission by offering their own ideas about how testing fits within a broader sense of accountability. They say the debate over accountability offers a prime opportunity for colleges to define their priorities and design measurements that reflect them.

“Public universities realize this is coming,” said Travis J. Reindl, director of state policy analysis at the American Association of State Colleges and Universities, which is working with the National Association of State Universities and Land-Grant Colleges to develop position papers on testing. “It is better to get out in front, manage it, and shape it than to be a victim of it.”

Sample Questions From 2 Tests

Following are sample questions from the Collegiate Learning Assessment, or the CLA, and the Measure of Academic Proficiency and Progress, known as the MAPP.

CLA Performance Task

Introductory material: You advise Pat Williams, the president of DynaTech, a company that makes precision electronic instruments and navigational equipment. Sally Evans, a member of DynaTech’s sales force, recommended that DynaTech buy a small private plane (a SwiftAir 235) that she and other members of the sales force could use to visit customers. Pat was about to approve the purchase when there was an accident involving a SwiftAir 235. Your document library contains the following materials:

Newspaper article about the accident
Federal accident report on in-flight breakups in single-engine planes
Internal correspondence (Pat’s e-mail to you & Sally’s e-mail to Pat)
Charts relating to SwiftAir’s performance characteristics
Excerpt from magazine article comparing SwiftAir 235 to similar planes
Pictures and descriptions of SwiftAir Models 180 and 235

Questions:

Do the available data tend to support or refute the claim that the type of wing on the SwiftAir 235 leads to more in-flight breakups? What is the basis for your conclusion?
What other factors might have contributed to the accident and should be taken into account?
What is your preliminary recommendation about whether or not DynaTech should buy the plane, and what is the basis for this recommendation?

CLA Make-an-Argument Prompt:

There is no such thing as “truth” in the media. The one true thing about the information media is that it exists only to entertain.

CLA Critique-an-Argument Prompt:

A well-respected professional journal with a readership that includes elementary-school principals recently published the results of a two-year study on childhood obesity. (Obese individuals are usually considered to be those who are 20 percent above their recommended weight for height and age.) This study sampled 50 schoolchildren, ages 5-11, from Smith Elementary School. A fast-food restaurant opened near the school just before the study began. After two years, students who remained in the sample group were more likely to be overweight — relative to the national average. Based on this study, the principal of Jones Elementary School decided to confront her school’s obesity problem by opposing any fast-food restaurant openings near her school.

MAPP Social-Sciences Question:

Economic growth and territorial expansion in medieval and early-modern times depended on urbanization, trade, colonization, conquest, and the clearing of land. But the increase in human contact established by these endeavors carried enormous risks of infection by disease. By adopting the concept of the unification of the globe by disease, we can discern the relationships between, and therefore better understand, phenomena seemingly very diverse. For example, we can see a connection between the outbreaks of plague in Western Europe, which recurred from the 14th through the 16th centuries, and beginning in the 16th century, the drastic depopulation of the New World by the spread of disease among its population.

This “community of disease” came about as the large populations of the world — the Chinese, the European, the Native American — expanded, and travel and trade among them greatly increased. The trade routes (traveled by armies of rats and fleas as well as humans) stretched across many disease-ridden areas. The danger became acute the moment traders from Genoa crossed the Black Sea and arrived in Central Asia, newly unified by the Mongols; and then once again, when Christopher Columbus, also Genoese, set sail westwards. Ultimately a large part of the world’s population perished from disease.

So great is the role of demography as something that shapes the development of human civilization that these devastating losses of life and the painfully slow recoveries from them influenced much of the world history.

Reading question: In using the phrase “community of disease,” the author of the passage most likely intends to indicate that:

Members of a particular society generally develop resistance to diseases that occur frequently in their society.
Only members of the same society are likely to be susceptible to certain diseases.
The exposure of diverse peoples of the world to the same diseases constitutes a link between these peoples.
The devastating effect of disease is a unifying factor among the people who suffer from it.

Critical-thinking question: The author’s discussion in the passage above presupposes that, before expanded trade routes linked Europe, China, and the Americas, which of the following was the case?

The large populations of the world did not suffer from disease.
Infectious diseases were found only in Western Europe.
Infectious diseases were unlikely to spread between the large populations of the world.
The traders from Genoa contributed to the spread of infectious diseases more than any other group.

Writing question: Which of the following is the best revision of “So great is the role of demography as something that shapes the development of human civilization that”?

So great is the role of demography in shaping the development of human civilization that
Of such greatness is the role of demography to shape the development of human civilization
The role of demography is of such greatness in shaping the development of human civilization
The role of demography is great enough to shape the development of human civilization such that

SOURCE: Council for Aid to Education; Educational Testing Service