2009 Aera Tica Instrument Validation

Validity and Reliability of Measures of Online Reading Achievement J. Greg McVerry & W. Ian O’Byrne University of Connecticut

Instrument: Identified frequency of Internet use both inside and outside school. Assessed knowledge and skill of Internet-specific reading and writing activities. Instrument: This instrument successfully developed a reliable scale for measuring online and Method: offline reading motivation in an online format. 1.Extracted factors with Eigenvalues greater than 1.25 using both PCA w/ Varimax Method: rotation and PAF with direct oblimin rotation. Instrument: 1.Extracted factors with Eigenvalues greater than 1.25 using both PCA w/ Varimax 2.Both extraction techniques suggested a nine factor solution. Extraction was then Measured online reading comprehension rotation and PAF with direct oblimin rotation. conducted with nine factors using PAF w/ direct oblimin rotation. performance using online quiz interface. A total 2.Both extraction techniques suggested a four factor solution. Extraction was then 3.Deleted multidimensional items and items that loaded alone on a scale, and re-ran of 15 items were scored using a rubric. conducted with four factors using PAF w/ direct oblimin rotation. extraction. 3.A reliability analysis was then conducted on all four of the subscales (Reading 4.We had an eight factor solution with more variance explained. Method: Online, School Literacy, Accuracy & Reliability, Reading Offline). 1.Ran initial PCA w/ varimax rotation of both treatment and control pre-tests. Results: Explained 56.245% of variance with a marvelous KMO (.906). 4.Of the four scales, one (Reading Offline) was shown to not have an α value 2.Ran PAF w/ direct oblimin rotation on both treatment and control pre-tests Scale Items α above the threshold established at 0.70 (0.58) 3.Both suggested five factors, thus we ran a PCA w/ varimax on pre and post control tests. Out of school Internet Leisure Use 11 .932 4.Calculated reliability of each scale. Results: Explained 50.89% of variance with a marvelous KMO (0.876). In school online content area reading 7 .902 5.Calculated frequency tables of each scale to check for adequate variance of scores. Out of school content area reading 8 .927 6.Collapsed the scoring rubric of some items, and re-ran reliability estimates. Scale Items α Internet Self-Efficacy 9 .926 7.Checked if collapsing scales threatened the overall construct validity of the item. Reading Online 6 .812 Pop culture communication in school 7 .771 8.Re-ran PCA w/ varimax to check item loadings with recoded scales. School Literacy 5 .756 9.Ran overall reliability of the instrument. In school Internet leisure use 5 .793 Accuracy & Reliability 3 .705 Discussion boards in school

2

.713

Discussion boards out of school

4

.875

Results: Scale

Items Collapsed item in bold Locating 2a, 2b, 2c Critical Evaluation 3a, 3b

Instrument: Determined which students were at the greatest risk for school dropout. Data gathered helped to determine the degree to which the Internet Reciprocal Teaching intervention yielded greater student engagement with, and attitudes towards school. Method: 1.Identified eight of 22 scales related to school engagement and attitude toward school (school engagement, learning climate, teacher support, selfconfidence, adjustment, social support trouble avoidance). 2.Ran PCA w/ varimax rotation and PAF w/ direct oblimin and both methods of extraction suggested 12 factors. 3.Ran PAF w/ direct oblimin using all subscales on the SSP School scale and extracted Eigenvalues >1.25. 4.Removed multidimensional subscales (trouble avoidance and school satisfaction) and re-ran PAF w/ direct oblimin. 5.Created a composite score to be used as a predictor of at-risk of dropping out. Results:

Initial Reliability Pre/Post α .758/.784 .905/.916

Collapsed Reliability Pre/Post α .762/.778 .905/.922

Critical Evaluation 4a, 4b, 4c,4d, 4e, 4f, 4g .631/.619

.631/.608

Synthesis and Communication

.710/.655

5a, 5b, 5c

.622/.605

Recoding items increased the reliability of the instrument, but also threatened construct validity of the item. For example, collapsing items 2c from a 3, 2,1,0 scale to a 1,0 scale would ensure almost any response would be scored correctly. Therefore we decided to check reliability of scales collapsing only the items that did not threaten construct validity (5a, 5b, and 4e). Reliability was highest with only items 5a and 5b collapsed.

No Collapsed Items all collapsed items items 5a and 5b collapsed Items 5a,5b, and 4e collapsed

Overall Pre-Test Reliability α

Overall Post-Test Reliability α

.705 .736 .793 .737

.705 .718 .725 .720

The three scale solution explained the greatest amount of variance (64.609%) and each scale had adequate reliability. The composite score was also comprised of the three scales the intervention could most affect. Scale

Items

α

School Engagement

10

.857

Learning Climate

3

.925

Teacher Supprt

6

.931

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305G050154 to The University of Connecticut. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Instrument: This instrument identified and measured the attitudes and aptitudes necessary for online reading comprehension. Method: 1.Identified five dispositions necessary for online reading comprehension (persistence, flexibility, collaboration, reflection and critical stance). 2.Three separate stages of content validation to establish item validity. 3.Ran PAF w/ direct oblimin while extracting Eigenvalues >1.25, and a parallel analysis. Both extraction techniques suggested five factors. 4.An EFA was conducted, using PAF and a direct oblimin rotation. 5.Five scales were identified through results from the EFA (Reflective Thinking, Critical Stance, Collaboration, Flexibility, Persistence). Results: The five factors were shown to explain 38.68% of the variance, with an achieved KMO of 0.939. In future iterations of the DORC, additional items need to be written to match the constructs and improve reliability. Scale

Items

α

Reflective Thinking

14

.907

Critical Stance

4

.686

Collaboration

3

.754

Flexibility

4

.623

Persistence

2

.700