Challenging Courses, Student Achievement, and Barriers to Implementation
The classification of each grade (5-8) within each partnership school as a high implementing, moderate implementing, or low implementing grade was based on classroom observations and the percent of teachers in the school at the given grade level who had participated in summer courses. The Reformed Teaching Observation Protocol [RTOP], specifically designed for mathematics and science observations, was used as the classroom observation tool. The instrument focuses on five areas: (1) lesson design and implementation, (2) propositional knowledge, (3) procedural knowledge, (4) communicative interactions, and (5) student/teacher relationships. 150 observations across all participating districts have been conducted over the course of the project. Of those 150 observations, 40 were repeated observations. Two trained observers visited each classroom and rated the instructor separately, then came together to discuss their ratings and reach consensus on discrepancies. To be classified as a high implementing grade within a school, 100% of teachers at that grade level must have participated in at least one summer course and the RTOP scores at that grade level must have been 12.5 or above out of a possible 20 on each subscale. Low implementing grades within a school are those with no participation in summer courses or those where there was participation but RTOP scores were low (5 or below out of a possible 20 on each subscale). The remaining grades within schools were classified as moderately implementing. In this way, all grades within schools were divided into three groups: low implementing, moderate implementing, or high implementing. Since the basic unit classified is the grade and not the school, it is possible for one grade in a school to be high implementing and others to be moderate or low implementing.
To assess student achievement, normal curve equivalents (NCEs) on the mathematics portion of the Stanford Achievement Test (SAT-10) were used. From 2007 to 2008, SAT-10 NCE scores were compared from grades 4 to 5, 5 to 6, 6 to 7, and 7 to 8 for matched students (n = 24,026) in partner districts for which data was available. Six of nine systems have been analyzed to date, including the three largest districts in the partnership. The collection of students in these six districts is diverse in terms of race and ethnicity and socio-economic status (SES). This process of comparing matched student SAT-10 NCEs was also conducted for 2006 to 2007.