Measuring Juku Efficacy

Currently, the coin of the accelerated or enriched supplementary education realm are claims as to the number of graduates of a particular juku who have been admitted to specific and prestigious educational institutions at the next level of instruction (i.e. middle school, high school, or university). For remedial education, an improvement in class standing or grades appears to be the generally accepted standard by which juku efficacy is measured.

While these indicators do point to juku performance, they really don’t say very much about a particular juku, nor a particular student.

First of all, neither the advancement rate nor a grade improvement can be compared to students who did not attend juku, i.e. there is no control group and no proxy of any kind that would at least mimic a control group, for example through value-added testing. This lack of a control group is particularly glaring when juku themselves require entrance examinations. If you only accept students who do well on standardized examinations (SAPIX, Nichinoken would be among the nationally known high-flyer juku that would be examples of this category), and you devote some additional resources to them (whether it is time, attention, teaching methods, or whatever really), lo and behold they do well on entrance examinations.

Secondly, advancement rates are only relevant information to prospective parents, they really do nothing for current parents as any conclusions about the efficacy of a juku do not come until after an entrance examinations, i.e. when it is too late given the rigid sequencing of educational stages in Japan.

Thirdly, advancement rates and grade improvements give no indication of what about a juku’s offering may have helped this particular student. Is it a motivational effect, simply additional in-puts (hours, attention, teaching materials), the classroom environment, or is it some teaching methodology as many juku would claim? We and parents/students have no idea whatsoever which element of the juku instruction may have led to an improvement in a grade of a standardized test score.

Fourthly, the notion of marginal utility seems to be entirely absent from most discussions about supplementary education. Especially in Japan where the dominant attitude is one that equates amount/intensity of effort with educational success, there’s little sense of whether that extra hour of practice/homework really leads to a greater/deeper learning, even when this is directed entirely at a standardized examination. Intuitively, most parents’ sense seems to be that there really is no such thing as too much learning/review/practice.

Bottom line? We really don’t have a solid empirical indication of whether supplementary education contributes significantly to individual and collective learning outcomes (as higher PISA scores in countries with well-established supplementary education sectors might suggest), nor which elements of supplementary education are contributing to learning outcomes for what (kind of) students.