Category Archives: Conferences

American Educational Research Association

Apr 14, 12:25-13:55h

Katherine Drake and Cheryl Carlstrom, Saint Paul Public Schools

“Evaluation of Supplemental Educational Services in a Midwestern Urban School District”

Supplemental Educational Services (SES) is defined as tutoring and other supplemental academic enrichment provided outside of the school day that is specifically designed to help students achieve proficiency on state academic standards as measured by the state’s assessment system. In compliance with the No Child Left Behind Act of 2001, any Title I school or district identified as not making Adequate Yearly Progress (AYP) for three or more consecutive years must offer SES to all low-income students attending the school. While the U.S. Department of Education requires that SES providers demonstrate effectiveness in improving student achievement, states and districts have limited capacity to monitor providers and to evaluate their performance.

To date, research on SES has not consistently shown a demonstrable impact on student achievement (Authors, 2007a; Authors, 2007b; Authors, 2010). The research, evaluation, and assessment office of a Midwestern urban school district conducted an evaluation in order to measure the effectiveness of SES providers at improving the academic achievement of students who received service in that district during the 2009-10 school year. The study was designed to answer three questions: 1. Who participated in SES? 2. What was the impact of SES on student achievement? 3. How did SES providers compare in terms of student achievement outcomes?

Data from a supplemental service database that included provider, session type, and attendance information for 1,692 registered students were linked to student demographic data as well as to fall 2009 and spring 2010 Measures of Academic Progress (MAP) reading and math data and to spring 2009 and spring 2010 data from the statewide tests in reading and math data. After reviewing participant demographics overall, by school, and by provider, we compared the achievement test scores of SES students to SES-eligible students (i.e., all students who received free or reduced price lunch and enrolled in an SES school) who did not register for SES.

The participant file was split by subject area in which each student received service (math or reading), with some students receiving assistance in both subject areas. Propensity Score Matching was used to identify comparison groups (by subject area) from the pool of all 2009-2010 non-participating SES-eligible students. Analysis of MAP and statewide test data supports existing research findings of little to no difference in test performance of students who receive SES compared to those who do not. Where statistically significant differences were found, matched students who did not receive SES outperformed those who did. When hours of service were considered, students who received 20 or more hours of SES met achievement test targets at the same rate as those who participated in fewer than 20 hours. While the percent of students achieving proficiency on the statewide assessment, in both math and reading, did vary by provider, no single provider showed success across all measures and subject areas.

In conclusion, this evaluation confirms the lack of support for SES found by researches in other urban school districts.

American Educational Research Association

Apr 16, 14:15-15:45h

Patricia Burch (Univ of Southern California) and Carolyn Heinrich (Univ of Texas – Austin)

“Preliminary Findings of a Multisite Study of the Implementation and Effects of Supplemental Educational Services”

Under No Child Left Behind (NCLB), schools that have not made adequate yearly progress in increasing student academic achievement for two years or more are required to offer parents of children in low-income families the opportunity to receive extra academic assistance, or supplemental educational services (SES). This paper works with ideas from the field of policy sociology to develop an understanding of SES as shaped by a complex interplay of forces; this field is particularly useful in analyzing a complex policy such as SES because it seeks to understand the intersection of macro and micro systems. This multilayered policy requires these various systems and actors to interact in new ways and in relatively uncharted policy territory. Research Setting and Design Findings are based on data from an ongoing multi-site study of the implementation and impact of supplemental educational services. The central purpose of this study is to understand whether and how providing students with academically focused out-of-school tutoring in reading and mathematics contributes to improvements in their academic performance, specifically in reading and mathematics. It involves three linked phases of research. Phase 1 is an in-depth qualitative study designed to define key elements of SES program models and the policy and practice variables that mediate implementation of these models and to also inform the construction of the measures of SES treatment for quantitative analysis. Phase 2 is a quantitative study investigating selection into SES (i.e., who registers and participates) and SES program impacts, using propensity score matching and fixed-effect methods with nonequivalent (internal) comparison groups. Phase 3 is a follow-up qualitative study to examine whether program features identified in Phase 1 continue over time and to further inform our interpretation of the quantitative findings of program impact from Phase 2. We are conducting this research in five urban school districts located in four states and representing different student demographics. Findings Our analysis reveals three primary themes. First, in order to understand and address the policy problems in SES, we need to see the policy as driven by a particular theory of action – the idea that public policy needs to be modeled around and accommodate the market place. Second, while NCLB employs the common rhetoric to equalize educational opportunities for the poor, the design of SES contributes to power asymmetries that sometimes privilege the financial interests of individual private firms and frustrate collective access to information. Thus, expanding the ability of SES to serve the goal of equal opportunity requires addressing the ways in which design and discourse of the policy denies equal access to knowledge for eligible participants and stakeholders. The other key to improving SES lies in the quality of instruction inside the SES classroom and the relationships between teachers, parents, students and tutors. In short, any effort to improve SES must address the perverse or inadequate incentives written into the design of current policy while solutions for improving policy should be anchored in deep understanding of actual classroom/tutoring conditions and the quality of relationships built there.