03 Evaluation Of Ict Programmes

Evaluating the impact of ICT

Prof. Angela McFarlane University of Bristol

Impact 2 2 Preliminary studies Main team at Nottingham, MMU and OU 60 schools Teacher researchers Pupil researchers Ends July 2002 funded by DfEE

Impact 2 - Strands Strand 1: to develop and apply appropriate methodologies for evaluating the use of ICT in school and to analyse the relation between effective implementation of ICT and standards of performance in national tests. Strand 2: to develop and apply a variety of methods to establish how pupils use ICT out of school and what is gained from such use. Strand 3: to explore the nature of teaching and learning involving ICT in various settings, with a central focus on the perceptions of pupils, teachers, parents and managers.

Impact 2 - Questions ⌧(1) ⌧(2) ⌧(3) ⌧(4) ⌧(5)

What is the involvement of pupils with computers and the Internet at home and in school? Does curriculum centred usage have a measurable effect on performance and attitude ? Are these effects confined to usage in school? Are all kinds of computer usage equally productive of learning? If ICT based learning involves interactions between home and school, what are the attendant problems and how can these be met?

Impact 2 – Answers? 75% primary pupils, 90+% secondary pupils have a computer at home Use in school subjects – never or hardly ever 5 from 13 subjects show a small positive correlation between ICT use and added value of attainment Use at home seems to correlate to improved attainment in school The data on types of ICT usage was inconclusive Problems of home/school use subject of a separate study

Impact 2 – Answers? ⌧ •

Differences in attainment associated with strength of ICT involvement were never large, but were nonetheless clearly present in more than a third of all comparisons

⌧

• In none of the 13 comparisons was there a statistically significant advantage to groups with lower ICT involvement

⌧

• At Key Stage 2, there was a highly significant positive association between ICT and SATs for English

⌧

• At that key stage there were similar trends for Maths and Science, but these were less clear-cut

⌧

• At Key Stage 3 there were no clear-cut associations between ICT and SATs results, but there were some indications of a positive association in Science

⌧

• At Key Stage 4, there was a highly significant positive association between ICT and GCSE for Science and for Design Technology

⌧

• Also at Key Stage 4’ there were strong indications of a positive association in Modern Foreign Languages, and somewhat weaker in Geography.

Impact 2 – Answers? There is no consistent relation between the average amount of ICT reported for any subject at a given key stage and its apparent effectiveness in raising standards. It therefore seems likely that the type of involvement is allimportant.

ICT can make a contribution to educational outcomes •for some learners •under certain circumstances.

Perspectives on ICT ICT as a set of skills/competences ICT as a vehicle/tool set etc to do what we have always done - better/cheaper ICT as an agent of change which impacts in a revolutionary way

Aspects of ICT that can contribute to learning Feedback Representation of dynamic processes The ability to edit Multiple representations of knowledge Shared records of learning

All depend on the prevailing learning culture

What should we look for in terms of impact of ICT?

Classification by outcome:

• The teaching of specified skills or discrete content e.g. spelling, the use of a database, the three times table. • The exposition of personal knowledge e.g. writing an essay on coffee production, or authoring a multimedia presentation on drugs awareness. • The building of personal knowledge through the identification, collection and relation of relevant information sources e.g. collecting information on coffee production or drug abuse from a variety of sources. • Communication within an electronic community

Traditionally defined curriculum and learning culture

Traditionally defined ICT skills

Learner activity, using new technology and ICT skills

Direct Impact Model

Improved knowledge and understanding

Improved attainment

Common gaps in the data set Number of pupils Number of schools Age of pupils Number of LEAs Computer/pupil ratio Types of ICT used Context used Methodology Date study began Duration Funding source

Self-directed learning

Overlap

Institutional learning

Learner-defined Learner-defined curriculum curriculum and and culture culture

Traditionally Traditionally defined definedICT ICT skills skills

Traditionally Traditionally defined defined curriculum curriculum and and learning learning culture culture

Learner’s Learner’s personal personal representation representation of of the the task, and available task, and available resources resources

Home-based Home-based task task

Teacher Teacher//curriculumcurriculumspecified specifiedtasks tasksfor forthe the learner learner

Potential Potential benefits benefits for for the the learner: learner: knowledge knowledge and and skills skills (beyond (beyond those those validated validated or or approved approved externally) externally) autonomy autonomy and and confidence confidence in in learning learning extended extended awareness awareness of of the the social social benefits benefits of of the the ICT ICT

Improved Improved knowledge, knowledge, understanding understanding and and skills, skills, of of personal personal and and social social benefit benefit

Learner Learneractivity, activity,using using ICT ICTskills skills

Improved Improvedrelated related knowledge knowledgeand and understanding understanding

Improved Improvedattainment attainment

Socially Contextualised Impact Model

Two fundamental questions for the next five years -

What should be recognised as educational attainment? What are the implications of the wider context of school based education?

Evaluating the impact of ICT

Prof. Angela McFarlane University of Bristol