Background
The Comprehension of Written Grammar (CWG) Test was written as a test of an examinee’s ability to understand English grammar at the most basic levels (syntax and morphology) of acquisition. Dr. Susan R. Easterbrooks, a professor emeritus at Georgia State University, created this test. Dr. Joanna E. Cannon, a professor at the University of British Columbia, conducted the validity and reliability research and conducted revisions to the test based on the results and three funded grants from the Social Sciences and Humanities Research Council of Canada. Caroline Wellesley drew the line drawings for all test pictures. Elissa Miller modified some of the line drawings and created the pictures in the vocabulary test. Elizabeth Miller, Kristen Hudgins and Alicia Knox colorized the pictures.
Reliability and Validity Information
Three studies over the past five years have examined the reliability and validity of the CWG, across 135 participants (86 d/Deaf and hard of hearing; 49 hearing) between ages 7-12 years old. These studies have assessed the CWG for content validity (Cannon & Hubley, 2014); test-retest reliability and known-groups validity (Cannon, Hubley, Millhoff & Mazlouman, 2015), and convergent and discriminant validity (Cannon, Hubley, O’Loughlin, Norman, Phelan, & Finley, 2019, with promising results.
Content Validity Results
Using 10 Subject Matter Experts (SMEs) and the Content Validity Index (CVI), the quantitative findings showed general support for the content validity of elements of the CWG in assessing the comprehension of written grammar (Cannon & Hubley, 2014).
Relevance of Grammatical Structures
At least 80% of SMEs endorsed the relevance of 26 of the 30 grammatical structures. Narrative feedback indicated that many structures were seen as building blocks to language acquisition, were structures that help identify weaknesses, and/or are appropriate in the developmental order of language acquisition. Fewer SMEs endorsed the relevance of irregular plural nouns, medial relative clause, dative movement, and final relative clause stating that these structures may not be developmentally appropriate or may reflect vocabulary rather than grammar. SMEs suggested adding pronouns and anaphora as structures (Cannon & Hubley, 2014).
Representativeness of Test Sentences
At least 80% of SMEs endorsed the representativeness of the 60 sentences used to test the grammatical structures. Three important themes emerged from narrative feedback: (a) avoid irregular constructions (e.g., plural – ‘pens’ rather than ‘feet’), (b) vocabulary needs to be clear and explicit, and (c) importance of clarifying which grammatical structure is being assessed (Cannon & Hubley, 2014).
Test-Retest Reliability Estimate
Test-retest reliability of the Vocabulary pretest scores were r = .05, n.s. for the Hearing participants and r = .92, p < .001 for the d/Dhh participants. The test-retest reliability of the CWG test scores were r = .84, p < .001 for the Hearing participants and r = .96, p < .001 for the d/Dhh participants. A parametric F-test was used to compare the scores of the Hearing and d/Dhh participants. Results indicate that the d/Dhh participants (M = 35.7, SD = 10.69) performed significantly lower on the Vocabulary pretest and the CWG test than the Hearing children (M = 44.1, SD = 4.58), t(96) = 5.04, p < .001, d = 1.03) (Cannon, Hubley, Millhoff & Mazlouman, 2015).
Known-Groups Validity Results
A rank transformed Welch t’ test was used to compare the scores of the Hearing and d/Dhh participants (Ruxton, 2006; Zimmerman & Zumbo, 1993). This test found that the d/Dhh children (Mdn = 13.0) scored significantly lower on the Vocabulary pretest than the hearing children (Mdn = 14.0), t (70.5) = 3.18, p < .01, d = 0.76 .
A parametric F-test was used to compare the scores of the Hearing and d/Dhh participants and found that the d/Dhh participants (M = 35.7, SD = 10.69) performed significantly lower on the Vocabulary pretest and the CWG test than the Hearing participants (M = 44.1, SD = 4.58), t(96) = 5.04, p < .001, d = 1.03) (Cannon, Hubley, Millhoff & Mazlouman, 2015).
CWG Vocabulary Pretest and Test Completion Times
A subsample of 29 Hearing participants and 36 d/Dhh participants were timed to determine how long the CWG Vocabulary Pretest and Test together take to complete. The rank transformed Welch t’ test was used and found that the d/Dhh participants (Mdn = 11.0 minutes) took longer to complete the CWG Vocabulary Pretest and Test than the hearing participants (Mdn = 10.0), t (61.0) = -2.19, p < .05, d = 0.56. The d/Dhh participants showed more variability in completion times than the Hearing participants. The longest completion times in the subsample were 21 minutes for a Hearing participant and 25 minutes for a d/Dhh participant.
Comparison of Hearing and DHH Groups on 26 Grammatical Structures
The scores for the two items for each of the 26 structures were summed and then the scores on the 26 structures for the Hearing and d/Dhh groups were compared (Cannon, Hubley, O’Loughlin, Norman, Phelan, & Finlay, under review). Conducting this analysis does give us a clue as to where critical differences lie between the two groups. Independent samples t-tests results indicated significant differences between Hearing and d/Dhh groups on the following 14 grammatical structures, with the Hearing participants scoring higher than the d/Dhh participants:
NP + Vi
NP + Vi + Adv-p
NP + be + Adv-p
NP + be + Adv-a
are + ing
were + ing
Vh (idiomatic)
comparative
possessive ‘s
reversible passive
non-reversible passive
for/to complement
NP complement
perfect tense
Difficulty of Each CWG Test Item
The difficulty of each CWG test item was assessed by determining the percentage of children in each of the Hearing and d/Dhh groups that scored correctly on that item. The test items were then ordered to increase in difficulty from the start to the end of the test and presented in the same order for each testee.
Intervention Effects
A validation study conducted by Cannon, Hubley, O’Loughlin, Norman, Phelan, & Finley (2019), found that the CWG test appeared to be sensitive to changes in the comprehension of written grammar over time. Struggling elementary school readers’ performance on the CWG was collected pre and post an 8-week intervention period using a randomized control trial methodology. A statistically significant increase in CWG test scores was seen over time in both the intervention and control groups. This may indicate that the CWG can measure growth in grammar skills.
Convergent and Discriminant Evidence
The CWG was investigated for correlations between the CWG and (i) convergent measures of grammar skills and reading comprehension, and (ii) discriminant measures of vocabulary. Discriminant validity results indicated a high correlation between grammar and vocabulary assessments, and convergent validity results indicated high correlations between the CWG and a measure of reading comprehension (Cannon, Hubley, O’Loughlin, Norman, Phelan, & Finley, 2019).
Revisions to the CWG Test
Following the research findings, revisions were conducted on the CWG to improve test validity. The presentation of the pretest was changed in order to more closely match the format of the test itself. The updated version of the pretest consists of 14 of the more difficult vocabulary words used in the main test items. Each vocabulary word in the pretest is printed on a page with three images and the participant is asked to point to/select the image which correctly corresponds to the word presented.
The images in the main test were similarly updated to improve clarity. The success rate was calculated for each CWG item and this data was compared with the overall success rate for that grammar structure to help identify whether lack of image clarity may be influencing the success rate for each item. 37 items indicated possible image confusion and these were updated to remain more consistent and demonstrate greater clarity in the direction of movement, the sequence of events, etc.