RESEARCH AND SURVEY STATISTICS – STA3022F SOLUTION TO TUTORIAL #3 Week 4 2008 ITEM RELIABILITY ANALYSIS REPORT TO RESEARCHER – Reliability of constructs underlying attitudes towards women in senior managerial positions Aims: 1. To assess whether the sets of questions designed to test the three constructs underlying attitudes towards women in senior management positions are internally consistent (i.e. are reliable indicators of the underlying construct) 2. To assess whether any of the questions should be deleted in order to increase the reliability of the scales. Methodology: Initially, an exploratory factor analysis of all questions was performed to identify, without imposing any structure on the data, which questions loaded together on a factor. If responses to the researcher’s questions are consistent with the hypothesised constructs, one would expect 3 factors to be present, with questions loading on factors in accordance with the specified relationships (i.e. one factor comprising A4, A5, A8, A10; another factor comprising A1, A2, A3; and another factor comprising A6, A7, A9) Then, an item-reliability analysis was conducted on each of the three underlying constructs. For each of the constructs, the overall Cronbach alpha and the average item-total correlation was examined to assess the overall construct reliability. Then, for each of the questions within a particular scale, the Alpha if deleted and item-total correlations were examined to assess whether any particular questions should be deleted in order to increase the reliability of the measurement scale. Results: The initial factor analysis shows three significant factors, in agreement with the researcher’s expectation. The factor loadings show some (but not total) consistency with the hypothesised construct structure, with the following factors emerging: Factor 1: A1, A2, A3 Factor 2: A5, A8, A10 Factor 3: A6, A9
(Hypothesised: Decision making skills) (Hypothesised: Interpersonal skills) (Hypothesised: Conflict resolution skills)
Only the decision making skills shows total consistency with the data. Question A4 and A7 do not load on any one factor in particular. Question A4 loads most strongly on factor 2, in accordance with the hypothesised structure, but the loading is fairly weak at 0.56. Question A7 is even more unclear, and loads on factor 1 and factor 2 with roughly equal strength. Construct 1: Decision making skills (TABLE 3) Overall Cronbach alpha = 0.6286 Average item-total correlation = 0.3744 Both the overall Cronbach alpha of 0.63 and the average item-total correlation of 0.37 suggest that the overall internal consistency of the 3 questions designed to measure decision making skills is moderate but not high. Although the overall reliability fails the traditional cut-off of 0.7, the difference is fairly small, and the questions could probably be considered to adequately represent the underlying construct. From table 3, there are no improvements in reliability that would result from deleting any of the questions measuring the decision making skills construct.
1
Construct 2: Interpersonal skills (TABLE 4) Overall Cronbach alpha = 0.65067 Average item-total correlation = 0.3378 The interpretation is very similar to the previous construct. Both the overall Cronbach alpha of 0.65 and the average item-total correlation of 0.33 suggest that the overall internal consistency of the 3 questions designed to measure interpersonal skills is moderate but not high. Although the overall reliability fails the traditional cut-off of 0.7, the difference is fairly small, and the questions could probably be considered to adequately represent the underlying construct. From table 4, there are no improvements in reliability that would result from deleting any of the questions measuring the interpersonal skills construct. Construct 3: Conflict resolution skills (TABLE 5) Overall Cronbach alpha = 0.210124 Average item-total correlation = 0.0635 The overall Cronbach alpha of 0.21 and the average item-total correlation of 0.06 suggest extremely low internal consistency between the 3 questions designed to measure conflict resolution skills. Although table 5 shows that there is a substantial improvement in reliability that would result from deleting question A7 from the interpersonal skills construct, the overall reliability of the remaining two questions (A6 and A9) would still only be 0.40. Thus the questions designed by the researcher to measure conflict resolution skills appear not to be measuring the same underlying construct. Conclusions and recommendations: The aim of the analysis was to identify whether the questions in the pilot questionnaire adequately captured the three hypothesised constructs underlying attitudes towards women in senior management positions. The decision making skills and interpersonal skills constructs can be considered to be reliably measured by the sets of questions hypothesised by the researcher. However, the conflict resolution skills construct is extremely unreliable, with very poor internal consistency. Removing question A7 substantially improves the reliability, and is recommended, but even then the reliability remains low. The research should consider changing the questions designed to measure conflict resolution skills before proceeding with the main study, and possibly rerun the pilot study to ensure that the new questions are more reliable. If the same questions are used, any results based on the conflict resolution skills construct should be interpreted with caution.
2
QUESTION 2 Q2.1 Construct 1: – Composed of Q1, Q2, Q3, Q4, Q5 – Results: alpha = 0.80, avg inter-item correlation = 0.45 – Assessment: Reliable scale; could remove Q5 to increase the alpha to 0.82, but improvement is only marginal so might retain Q5 anyway (either outcome is fine but must have mentioned possibility of removal) Construct 2: – Composed of Q6, Q7, Q8 – Results: alpha = 0.84, avg inter-item correlation = 0.64 – Assessment: Reliable scale; don’t remove any questions Construct 3: – Composed of Q9, Q10 – Results: alpha = 0.76, avg inter-item correlation = 0.52 – Assessment: Reliable scale; don’t remove any questions To explain apparent differences between factor and item reliability analysis results: mention that item-reliability analysis only looks at the correlations between indicated questions, and not at correlations between those indicated questions and any questions outside the indicated construct (Key idea: correlations between questions inside/outside a user-specified construct).
3
QUESTION 3 Q3.1 For two attributes, var[Q1 + Q2] = var[Q1] + var[Q2] + 2 cov[Q1;Q2] Thus rewrite alpha as
α=
k 2 cov(Q1, Q 2) ( k − 1) var(Q1 + Q 2)
Now:
• •
Variances are always non-negative, so for Q1 and Q2 positively correlated, Cronbach alpha must be greater than 0 and increase with increasing covariance relative to variance (stronger association) Cronbach alpha could be negative if items were negatively correlated
Q3.2 var[Q1 + Q2 + Q3] = 5.452 = 29.7 var[Q1] + var[Q2] + var[Q3] = 5.9 + 5.1 + 6.3 = 17.3 k=3 Alpha = (3/2)(29.7-17.3)/29.3 = 0.68 Items are not internally-consistent according as the alpha is below the rule-of-thumb cutoff of 0.70. However, they are only very slightly below the cutoff and their reliability is thus not too bad. Q3.3 Recommend removing item Q3. By doing so, the internal-consistency reliability rises greatly from 0.68 to 0.95 which (1) pushes the “unreliable” construct over the 0.70 benchmark, and (2) is a substantial increase in reliability (0.95 – 0.68 = 0.27)
4