Session 22 Trevor Barker

The Development of a Verbal and Visual Skills Profile for use in the Configuration of Blended Learning Objects Trevor Barker, Asli Adisen University of Hertfordshire, [email protected]

ABSTRACT The research presented in this paper relates to the development and testing of computer software to measure visual and verbal skills and to investigate their relationship to Riding’s WAVI cognitive style dimensions. The context of this work relates to the development of psychological mental and student models to assist navigation and task performance on adaptive educational computer systems, blended learning objects. Based upon our experimental work, there was little evidence of correlation between the common visual skills measured in our test and Ridings WAVI dimensions as measured by his Cognitive Skills Analysis (CSA) test. Initial findings related to common verbal skills also failed to find significant relationships. We discuss possible reasons for this finding and attempted to relate it to the validity of the CSA test. We also present our ideas on a visual skills profile that we believe is a more useful measure of the visual abilities of learners than single numbers along Ridings VI dimension in our work. The potential for the skills profile for use in the configuration of adaptive learning objects is discussed.

KEYWORDS Student Models, Computer-Based Learning, Cognitive Style, Visual Skills INTRODUCTION The research reported here relates to understanding how individual learners interact with computers when as they learn (Adisen et al., 2004). We have developed in the past, a psychological student model based on the characteristics and skills of learners (Barker, 2006; Barker et. al. 2002a; 2002b). We have used these models to configure adaptive multimedia and other software to assist learners. The concept of cognitive style has been important in our models to measure verbal and visual skills of learners. This research investigates the potential of cognitive style as a descriptor in a psychological model of learners. Riding has described two bipolar dimensions of cognitive style, the Wholist/Analytic (WA) and Verbaliser/Imager (VI) dimensions. A simple computer based test, the

SOLSTICE 2007 Conference, Edge Hill University

1

Cognitive Style Analysis (CSA), has been developed by Riding that is able to classify learners according to their position along the WA and VI dimensions. Simply stated, Riding's WA dimension relates to whether individuals process information in wholes or parts. The VI dimension classifies whether individuals represent information, during thinking, in words or pictures. Riding and others have provided evidence to relate the WAVI cognitive styles to learning processes (Riding, 1997). Our own research over the last ten years has been concerned with the use of Riding’s cognitive style in understanding how people learn with computers (Barker, 2006). The increasing processing power of the computer has provided opportunity to create complex interactive visual environments within which we can work and learn. The research described in this paper is concerned with understanding how Riding’s WAVI dimensions relate to the skills required for a range of complex visual activities. These include the abilities to memorise and compare images, manipulate and analyse information held in image form and the ability to form and hold images in the mind. These visual skills will form part of a student model that will be used in order to configure the presentation of information in an adaptive computer-based learning application.

VISUAL SKILLS PROFILE An important question for our research relates to the validity of Riding’s test. We were interested in how Riding’s WAVI dimensions were related to simple visual skills, such as are required when students are interacting with computers to perform simple tasks like browsing the web, navigating in virtual space or working with a graphics application. To this end we identified a set of visual skills that we could measure simply and attempt to relate to the WAVI dimensions. The set of skills identified are shown in table one below. Other measures of visual ability have been included in addition to those objective skills listed above, such as the students’ perception of their visual ability, and their reported method used to complete visual tasks. This was because some published estimates of verbaliser-visualiser ability are based on self reporting using introspective methods. In our study, a visual skills test was developed to measure the skills shown in table one. The main independent variables used in the study were the accuracy of responses to questions in the above categories and also the time taken to respond. These are set out in table four. Two variables measured in the study were Riding’s WA and VI scores obtained by participants in the CSA test. VISUAL SKILLS TEST The Visual Skills Test was written especially for this study. In addition to recording participant information, the test consisted of 101 questions measuring visual skills, classified into the 14 skills classes shown in table one. To avoid any effect of fatigue, the visual skills were distributed in small groups throughout the test. Riding’s CSA test employed in the study was the standard computer-based test, delivered and administered as recommended (Riding 1991a; 1991b; 1997) and described in the next section.

SOLSTICE 2007 Conference, Edge Hill University

2

Visual Skills categories measured by the test. Remember an image and compare with the one seen earlier Compare an image with one present on the screen at the same time Remember details of an Image Not Present Ask students whether they adopt a V or I strategy - hypothetical Measure whether students adopt a V or I strategy – actual task Ask students whether they adopt a W or A strategy - hypothetical and actual Analyze information contained in a graph or chart Remember verbal instructions about an image not present and perform task Remember verbal instructions about an image present and perform task Remember and estimate the size of an image not present Hold a consistent map in their mind Rotate and manipulate an image present on the screen Remember objects in an image not present and compare to other similar images

Table 1: Common visual skills assumed to be important in computer-based learning: Independent variables. Method The sample consisted of 34 volunteers (19 male, 15 female) who were mainly university teaching staff, postgraduate students and experienced trainers from a large Government organisation. All participants were good English speakers with normal eyesight. Ages ranged from 18 to 54. All data was collected online by the software and held in files on computers. It was initially collated by software especially written for this purpose. All timings were made automatically from within the software and were accurate to approximately 0.1 second. Prior to the study, all testing and analysis software, instruments and procedures were tested in a small pilot study with four expert users. A few small problems were identified and corrected. Procedure The study took place on standard desktop computers in two equivalent locations, quiet computer laboratories with controlled lighting. Prior to each test a scripted introduction was read to each participant. After these instructions, participants were randomly allocated take either our visual skills test followed by Riding’s CSA test, or vice versa, with a short break in between. Additional on-screen information was provided for both tests which participants had to read both before and during the tests. The visual skills test took approximately 40-60 minutes to administer and Riding’s CSA test approximately 15-20 minutes. Results The summary of the scores obtained in Riding’s CSA test and in both WA and VI dimensions are shown below in table two.

SOLSTICE 2007 Conference, Edge Hill University

3

Table 2. Summary of the scores obtained in Riding’s CSA test The scores obtained for Riding’s CSA test are mostly in line with other reported studies (Riding 1997). To test the independence of the dimensions, Pearson’s PM correlation was performed on the data summarized in table two. The results of this correlation are shown in table three below. Correlations WA WA

VI

Pearson Correlation Sig. (2-tailed) N Pearson Correlation Sig. (2-tailed) N

. -.061 .751 34

VI -.061 .751 34 .

Table 3 Pearson’s PM correlation performed on the data summarized in table 2. The value of the correlation coefficient (r=-0.061, N=34, p=0.75) supports the assumption of independence between the WA and VI dimensions. Any relationships between then can be ascribed to chance alone. The mean scores and time taken to complete tasks in the visual skills test are summarized in table 4 below. Data analysis

The data summarized in tables two and four was subjected to further analysis to investigate the significance of any differences or relationships. A limitation of the data collected in the CSA test is the fact that it is based on ratio measurements (Riding, 1998). It is difficult therefore to justify the use of parametric data analysis, as some of the assumptions of parametric analysis are invalid with ratio data. The use of such parametric analyses with Riding’s test is commonly reported in the literature however, presumably on the assumption that the underlying dimensions being measured are normally distributed and linear. We have used the SPSS software application to perform parametric data analysis in this study, with a mind to the limitations described above. Table five below shows the relationship between the time taken to complete the visual skills test (TIME) and the score obtained (ACC), with the WA and VI scores obtained in the CSA test.

SOLSTICE 2007 Conference, Edge Hill University

4

VARIABLE DESCRIPTION Total Accuracy

( N=34)

Mean

(S.Dev).

51.76

(5.03)

Accuracy in remembering images

3.73

(0.70)

Time taken remembering images

10.24

(3.12)

Accuracy comparing images

7.38

(1.15)

Time taken to compare images

11.92

(4.51)

Accuracy remembering images

11.66

(2.50)

Time taken remembering images

7.44

(2.12)

Accuracy analysing graphical information

20.58

(2.82)

Time taken analysing graphical information

30.08

(8.45)

Score reporting on strategy

7.14

(1.50)

Time taken to undertake tasks requiring V or I strategy

30.38

(4.90)

Comparisons using a V or I strategy

6.82

(1.56)

Time taken making comparisons using a V or I strategy

27.85

(9.87)

Accuracy in estimating size of images

3.47

(1.81)

Time taken in estimating size of images

17.27

(4.99)

Accuracy manipulating images

6.29

(2.24)

Time taken in manipulating images

15.04

(6.56)

Accuracy answering questions requiring V or I strategy

1.41

(0.70)

Time taken answering questions requiring V or I strategy

8.96

(4.73)

Time taken in actual task used to estimate V or I strategy

40.00

(6.71)

Estimate of V or I strategy used in actual task

5.38

(3.19)

Report of W or A strategy used in WA questions

2.50

(2.23)

Accuracy answering questions requiring WA strategy

1.91

(0.51)

Time taken answering questions requiring WA strategy

6.61

(2.51)

Accuracy answering mind map questions

4.11

(3.36)

Time taken answering mind map questions

136.91

(62.53)

Table 4. Mean scores and times obtained in the visual skills test (N=34). Variable TIME

Pearson Correlation Sig. (2tailed)

ACCURACY

Pearson Correlation Sig. (2tailed)

TIME

ACC

WA

VI

1

.202

.290

-.099

.

.251

.097

.577

.202

1

.206

.143

.251

.

.243

.419

Table 5. Pearson’s PM correlation showing the correlation between the score obtained in the visual skills test, the time taken, WA and VI scores.

SOLSTICE 2007 Conference, Edge Hill University

5

The results of this analysis show a small and insignificant positive correlation between the WA dimension and the TIME variable (p two-tailed >0.05). No other significant correlations were found. Correlations WAVI & visual skills The following correlations were obtained between some of the dependent variables and Riding’s VI and WA dimension. One-tailed statistics were used, as it was possible in all cases to predict a direction for the relationships found. Most values of the correlation coefficient were small. Only those correlations that were significant (p<0.05) or of interest (p<0.10) are given. Verbaliser-Imager • Time to remember an image and answer questions about parts of it (p=0.009) r = 0.44 • Ability to remember an image and answer questions about parts of it (p=0.10) r = 0.22 Wholist-Analytic • • • • • •

Time to analyse graphical information (p=0.02) r = 0.35 Time taken to manipulate an image (p=0.045) r = 0.29 Reported strategy to solve problems (p=0.06) r = 0.27 Ability to solve problems relying on Analytical skills (p=0.025) r = 0.33 Mind map (time and distance related) (p=0.01) r = 0.40 Time to remember an image and compare with the one seen earlier (p=0.04) r = 0.3 • Ability to remember an image and compare with the one seen earlier (p=0.08) r = -0.25 • Time to remember an image and answer questions about parts of it (p=0.06) r = -0.26 • Ability to remember an image and answer questions about parts of it (p=0.085) r = 0.24 Analysis of Variance (ANOVA) - WAVI groups and visual skills

An ANOVA was performed on the data summarised table four. No significant differences were found between the performance of Verbalisers and Imagers, or between Wholists and Analytics. To allow for the effect of those classified as Intermediates (Riding, 1991a) the data was divided into three equal groups at appropriate points along each of Riding’s WA and VI dimensions. Intermediate values were removed from the data set and the remaining values were subjected to ANOVA to test for the significance of any difference in the means of the remaining extreme WAVI groups for all the dependent variables as before. With the middle values removed, some SOLSTICE 2007 Conference, Edge Hill University

6

significant differences in the means were found (p<0.05). These are summarised below along with some interesting differences that, although not significant, were approaching significance (p>0.05 <0.10). • Time taken to manipulate images (A greater than W p<0.05) • Reporting a Wholist Strategy to solve image problem (W greater than A p<0.07) • Ability to hold a mental map (A greater than W p<0.05) • Ability to remember an image (I greater than V p<0.065) • Time taken to compare images present (I greater than V p<0.05) • Ability to remember images (I greater than V p<0.07) It is interesting to note that even significant correlations were relatively small, with r in the region of 0.3; that the direction of correlations is sometimes difficult to explain and that many of the dependent variables relating to ability had no significant relationships with the WA or VI dimensions or showed no significant differences in performance between WA or VI groups. Our overall interpretation of these results is given in the discussion section of this paper. VERBAL SKILLS TEST In order to relate verbal skills to Riding’s Cognitive Styles test, a similar experiment was conducted using the set of verbal skills shown in table six below. Answer a question related to a passage that is present on the screen Answer a question related to a passage that is not present on the screen Answer questions regarding a passage listened to Complete the sentences with the most appropriate choice given Find the analogies between words Find the antonyms of the given words Find the synonyms of the given words Answer some questions that requires analytical skills Remember the words that were presented on the screen Remember the words that were listened to Answer some questions regarding verbal instructions

Table 6. Important verbal skills required to undertake complex tasks on computers. The results of this study are currently undergoing detailed analysis. Initial findings from this work however, suggest that Riding’s CSA test was not useful as a measure of the range of verbal skills identified in table five above. We were unable to find significant differences between those classified as verbalisers and imagers in performance of these skills, nor were we able to identify significant correlation between Riding’s dimensions and performance measures. The implications of the lack of a significant relationship between Riding’s WAVI dimensions and common verbal and visual skills is discussed in the next section of this paper.

SOLSTICE 2007 Conference, Edge Hill University

7

DISCUSSION It is claimed that those classified as Verbalisers according to the CSA test find “speech and text easier than diagrams” and “learn best from verbal presentations”. Imagers “learn best from visual displays” and “find pictures easier than words” (Riding, 1998). Although Riding’s summary of the relationship between visual and verbal skills and CSA test scores is necessarily a generalization, our results suggest that it is an oversimplification and this claim per se. certainly is not supported by our results. Few significant correlations and differences in performance were found. Most of those found related to time to undertake tasks, rather than differences and relationships in visual skill levels and the WAVI dimensions. In the research reported here we also considered the verbal skills end of the VI dimension, initial findings from a similar verbal skills test, producing a similar lack of evidence. It is surprising that many common visual are not performed better by imagers when compared to verbalisers, and that we found no relationship between skill level and scores along the VI dimension. Likewise we world expect verbalisers to perform better at verbal tasks than those classified as imagers. We might expected some sort of relationship, based on Riding’s claims. In the study reported in this paper, no significant differences were found in performance between those classified as verbalisers and imagers, when divided at the median point (1.02). Even when intermediate scores were removed, many verbals and visual skills failed to show significant differences between the performance of verbaliser and imager groups. It is interesting therefore, to consider how these findings might relate to the validity of the VI dimension of the CSA test. There have been criticisms of the reliability of Riding’s CSA test recently (Peterson et al.,2003a; 2003b; Coffield et al, 2004; Razaei and Katz, 2004). There has however been little evidence to suggest that the CSA test is not a valid measure of cognitive style. Riding suggests that a test may be valid yet not reliable (Riding, 2003). He further states that validity is determined by the relationship of a test to objectively observed behaviours. We have been unable to find any direct examples where complex visual skills have been tested in the way we reported and measured against the CSA test. The often reported correlation in the literature between learning, or learning preference and the WAVI dimensions is too coarse a tool we argue, to support the validity of the WAVI dimensions on their own. Learning after all, is a broad, multidimensional and poorly understood complex concept in its own right. The reported and observed independence of the WA and VI dimensions from themselves and from other factors such as intelligence, although necessary to determine validity, are not sufficient, as is claimed by Riding (2003). Other measures of validity, including predictive validity, content validity and criterion validity might be better employed to this end. Much of what is offered in support of the validity of the CSA test is little more than face validity, un-supported by direct objective measures against other tools. It is of course possible that the CSA test is valid in simple cases, but when visual tasks get difficult and complex it is not possible to relate real-word tasks to the VI dimension alone. There is also ample evidence from our work and from others for the interaction of the WA and VI dimensions in real-world tasks (Kozhevnikov et al., 2003). The WA and VI dimensions have consistently been shown to be independent in the CSA test. It

SOLSTICE 2007 Conference, Edge Hill University

8

seems likely to us that the form of the CSA test itself (VI dimension measured only verbally, and the WA dimension measured only with images) is an important factor or reason for this independence. If visual skills are measured by the ability to deal with images themselves, then interaction with the WA dimension is likely to take place. In real life tasks, the interpretation of images is a multi-dimensional problem, involving analysis and manipulation of whole and part images. Since the CSA test was first developed, information presentation has moved from a textual, monochrome, paperbased, non-interactive world to the world of full colour, real-time interaction in virtual, image based environments. Tasks involving images have got more complex and there are many more of them. Objective measures of the VI dimension such as the CSA test, in future may need to reflect this fact. Working with images is a complex multidimensional skill involving imagination, colour perception and skills, ability to remember, compare, form, analyse and manipulate two-dimensional, three-dimensional and moving images on a computer screen, on paper and in the mind. We are currently engaged in developing visual and verbal skills profiles that we will use to understand how well students perform many kinds of visual and related tasks (Barker 2006). We are making no claims as to the validity or reliability of such profiles. Indeed we expect such profiles to change with time, due to the effects of learning and the context of their use, unlike Riding’s dimension which are claimed to be ‘biologically wired-in’ (Riding, 1997). We are certain however, that such profiles will be more useful to us in our research than any single measure along the VI dimension for an individual to measure verbal and visual skills. We intend to measure, record and change these profiles as users interact with the visual and verbal applications on their computers. Since the Riding’s CSA Test failed to relate to the important visual and verbal skills required to interact with complex computer applications, such as is necessary when learning from computers, we decided to develop and investigate visual and verbal skills profiles based on the skills identified in the verbal and visual skills tests. A factor analysis was therefore performed on the data obtained in the visual and verbal skills tests in order to identify important components that might be useful in a psychological student model. Based on this factor analysis there are six visual factors and four verbal factors that might be included in the new psychological model. Table seven and eight shows the results of this factor analysis for visual and verbal skills respectively.

SOLSTICE 2007 Conference, Edge Hill University

9

Variable Remember an image and compare with the one seen earlier Compare an image with one present on the screen at the same time Remember the features of an image that is not present on the screen and answer some questions about it. Analyse the information presented with a graphical format Ability to estimate a size of an image that is not present on screen Ability to manipulate an image Self reporting on VI Strategy used for VI questions Self reporting on WA Number of items recalled in a photograph Number of removed items recalled in a photograph Noticing the items removed from a photograph

1 .546

2

Component 3 4

5

6

.608

.721

.659

.661 -.727 .563 .809 .664 .546 .590

Table 7: Component Matrix relating to the skills considered in Visual Skill Test Six important factors were identified for the visual skills profile which are as follows: • Factor 1= Analysing the information given in image or graph format • Factor 2= Predicting information based on an existing image • Factor 3= Self reporting on WA dimension • Factor 4= Recalling information about information seen earlier • Factor 5, 6= Self reporting on VI dimension

SOLSTICE 2007 Conference, Edge Hill University

10

Variable Answering questions about a passage read Answering questions about a passage listened to Completing sentences Finding the synonym of a given word Finding the antonym of a given word Answering analytical skill questions Recalling the words read Recalling the words listened to

1 .667

Component 2 3 .818

4

.760 .690 .805 .877 .637 .706 .742

Table 8: Component Matrix relating to the skills considered in Verbal Skill Test Four important factors were identified for the verbal skills profile which are as follows: • Factor 1= Finding the synonyms and antonyms of the given word • Factor 2= Analytical skills required to recall information • Factor 3= Completing sentences • Factor 4= Ability to recall the information heard More work will take place in future on the identification and testing of verbal and visual skills factors, as these are likely to be important components of the psychological student model. Other psychological descriptors that could be considered for inclusion in the new psychological student model for configuring blended learning objects are Riding’s WA dimension. This is because this dimension has been shown in our experiments to be important and related to the performance of students undertaking complex tasks. The other reason for potentially including this dimension in the model relates to the findings of the other studies reported in literature such that this dimension showed many correlations and differences in different approaches students take to interact with different computer applications. The two other features that will be included in the new student model are students’ language level which is an important indicator in terms of how well they can understand the textual information and their domain knowledge which is a good indicator of their existing mental model of the application. It is important for our research to capture valid, simple and accurate measurements of verbal and visual skills. It is expected that the factors identified for the visual and verbal skills profiles, a learner’s language ability and the student’s domain based mental model will change with time, due to the effects of learning and the context of their use. We intend to measure, record and change these profiles as students interact with the visual and verbal applications on their computers. In summary, the reliability of any of the measures employed to obtain those features used in our student model are far less important for this study than validity of those features. This may seem strange at first, but it is intended to adapt presentation of SOLSTICE 2007 Conference, Edge Hill University

11

visual and verbal information in learning objects based on performance on a computer. For this purpose the reliability of the test and re-test measures of these features are not important at all. The most important condition for the new student model is simple and accurate initial measurements that relate accurately to the important skills under consideration. To this end it is thought the visual and verbal skills profiles developed will be far more useful than a single value measure (such as Riding’s CSA measure) which tries to summarise all the skills in one or two measurements. The drawback of using a single measurement for multiple skills is the difficulty of deciding the relation between different skills when a change in the value to occurs. In the new visual and verbal skills profiles there are no such problems as different skills are represented in different factors.

REFERENCES Adisen, A, Barker, T. & Britton, C. (2004) Investigating the potential of mental models in adaptive user modelling, Proceedings of HCI 2004: Design for Life, Leeds Metropolitan University, September 6-10. Volume 2, 161-163. Barker T, Jones S, Britton C and Messer D J, (2000b), The use of the Verbaliser-Imager cognitive style as a descriptor in a student model of learner characteristics in a multimedia application, Proceedings of 5th Annual Conference of the European Learning Styles Information Network, (ELSIN) University of Hertfordshire June 26 & 27, 2000, Barker T, Jones S, Britton C and Messer D J, (2002a), The use of a co-operative student model of learner characteristics to configure a multimedia application, User Modelling and User Adapted Interaction, 12(2-3), 207-241 Barker, T. 2006, Attending to Individual Students: How student modelling can be used in designing personalised blended learning objects. Journal for the Enhancement of Learning and Teaching, ISSN 1743-3932, Vol. 3 (2) pp 38-48 Coffield F., Moseley, D., Hall, E and Ecclestone, K. (2004), Learning Styles and Pedagogy in Post 16 Learning: A systematic and critical review. Learning Skills and Research Centre. Kozhevnikov, M, Hegarty, M and Mayer, R.E, (2003), Revising the Visualizer–Verbalizer Dimension: Evidence for Two Types of Visualizers, Cognition and Instruction 20 (1), 4477 Peterson, E. R, Deary, I, J and Austin, E.J (2003a), The reliability of Riding’s Cognitive Style Analysis test, , Personality and Individual Differences 34, 881-891 Peterson, E. R, Deary, I, J and Austin, E.J (2003b), On the assessment of cognitive style: four red herrings, Personality and Individual Differences 34, 899-904

SOLSTICE 2007 Conference, Edge Hill University

12

Razaei A, R and Katz L. (2004), Evaluation of the validity and reliability of the cognitive styles analysis. Personality and Individual Differences 36, 1317-1327 Riding R, (2003), On the assessment of cognitive style: a commentary on Peterson, Deary and Austin, Personality and Individual Differences 34 Riding, R. (1991a), Cognitive Style Analysis, Birmingham Learning and Training Technology. Riding, R. (1991b) Cognitive Style Analysis, Users’ Manual, Birmingham Learning and Training Technology. Riding, R. (1997), “On the Nature of Cognitive Style, Learning Styles and Strategies”, Educational Psychology 17(1/2), 29-49. Riding, R. (1998), Cognitive Style Analysis, Research Applications, Birmingham Learning and Training Technology.

SOLSTICE 2007 Conference, Edge Hill University

13