Chaired Panel 6 Jackie Haigh Christine Dearnley

The impact of mobile technology on students’ experience of learning and assessment in practice settings: a focus group study

Jackie Haigh University of Bradford [email protected]

Christine Dearnley University of Bradford [email protected]

ABSTRACT As part of the ALPS CETL (HEFCE 2007) 29 student midwives and their link lecturers were given an electronic version of a clinical portfolio on Personal Digital Assistants (PDAs). These devices were used during a seven week clinical practice placement to record tripartite assessment interviews and clinical experiences relevant to the performance indicators of the placement. Focus groups explored the impact of the electronic portfolio on the students’ experience of clinical practice and its assessment. Data was analysed from an activity theory perspective in that the electronic portfolio was viewed as an artefact mediating situated knowing about student assessment in a particular socio-historical context. Findings suggest that the changes made to the electronic version of the marking criteria mediated a shared understanding of the assessment process which was pragmatic and less contested by students; however this still required skilful facilitation by the link lecturer and needed to be sensitively introduced to clinicians. Changing the assessment tool has the potential to change the shared understanding of the assessment process. Before seeking to radically change the tool for all professional groups it is essential to understand current processes and how the introduction of mobile technology and new tools might impact on them. This paper analyses student experiences of using an electronic assessment tool on a PDA to facilitate self-assessment, feedback and grading in clinical practice. It explores the potential of the new tool to mediate a shared understanding of the assessment process and to enhance student learning from experience in practice. The significance

SOLSTICE 2007 Conference, Edge Hill University

1

of the project lies in its potential to illuminate the potential of new technologies and tools to change the shared meaning of professional practice assessment.

KEYWORDS Clinical assessment: activity theory, mobile technology

INTRODUCTION This paper reports on one aspect of the findings of a case study conducted as part of an Assessment & Learning in Practice Settings (ALPS) IT pilot project. ALPS is one of the 74 Centres for Excellence in Teaching and Learning (CETLs) funded by the Higher Education Funding Council for England. It is a collaborative programme between five Higher Education Institutions with proven reputations for excellence in learning and teaching in medicine, dentistry, health and social care. Practice settings in health and social care education refer to hospital and community based work placements for students. The findings of this study may also be relevant in other assessed work-based learning situations. ALPS aims to develop and improve assessment, and thereby learning, in practice settings for all Health & Social Care students and to develop the competence of people who support and assess Health and Social Care students in practice settings. Crucially, it is intended that assessment and learning in practice settings will be enhanced by electronic mobile delivery of learning objects and assessment documents. Exploring the feasibility and potential of mobile learning and assessment has therefore become a central tenet in the work of ALPS. This paper focuses specifically on the impact of assessment tools on the student experience, arguing that use of a mobile format of itself necessitates change. Such change if informed by clear pedagogical principles can enhance the student experience of being assessed.

THE PILOT A cohort of thirty first year student midwives was selected as the case study group to trial using a PDA (personal digital assistant) in a clinical assessment context. The midwifery programme already assessed practice through a paper portfolio document. At the beginning of each placement students have a preliminary interview with their clinical assessor and link lecture during which an action plan is agreed. The learning opportunities in the placement are identified and performance indicators to be achieved are discussed. An intermediate interview midway through the placement is a chance to review progress towards the performance indicators and amend the action plan if necessary. At the end of the placement the student discusses her achievement in all performance indicators and these are signed off by the assessor if an adequate standard of performance has been achieved. The assessor then grades the overall performance using a marking criteria grid to assess the level of student attainment of the module learning outcomes for the particular stage of the programme. Changing this

SOLSTICE 2007 Conference, Edge Hill University

2

well established process to include a PDA format was thought to be relatively unproblematic – it was already available electronically as a word document. The pilot group had completed one placement using the paper portfolio and would be in placement again within the time span of the project therefore they were ideally placed to compare the two tools. The assessment process included three clinical interviews involving student, link lecturer and clinician. This allowed the PDA to be tested over time as a tool to facilitate an assessment interaction between three people in a clinical setting. Our initial project plan was to create electronic performance indicator and assessment interview forms using Microsoft Word© which students could work on using their personal computers or the PDA. These forms would replicate, to the extent that it was possible, the paper based format with which students and staff were familiar. In addition, a new form would be created which amalgamated the learning outcomes and assessment criteria into a simple dropdown assessment form, since it was recognised that huge chunks of text and the necessity to scroll should be avoided on the PDA if at all possible. We anticipated that forms could be prepared in advance on the students own computer, could be added to by link lecturers and mentors during the interviews, signed electronically and beamed by blue tooth technology from the students PDA to the link lecturers PDA on completion of the interview. The link lecturer could then feedback the electronic form to a central data base held by the divisional administrator. Both the student and the Midwifery Division would have a complete and verified record of the students’ placement experience stored electronically and available for print.

TECHNICAL HITCHES As we tried to put this plan into action we came up against a few technical hitches. Pocket Word© did not support tables, drop down boxes or many of the features we had taken for granted in our own use of the professional word software. We soon realised that it was not sophisticated enough for our purpose. We spent a frantic few days searching for alternatives and chose Pocket Forms© as a programme more suited to our needs, however it was far from ideal. Our IT expert was able to create the required forms on a PC but the cost and complexity of providing the full Pocket Forms© programme to students was prohibitive and would not have adequately solved the problem of allowing students easy access to their records whether on PC or PDA. We were forced to compromise and offer a system which required all data to be added via the PDA. A second problem was bringing all these forms together to create a database of the placement assessments for the whole group. In the paper version the students had handed in their portfolio at the end of the year so that marks could be recorded. Creating a contemporary record of student achievement in placement was felt to be necessary for the security of the electronic version. This was not easily achieved through Pocket Forms© and necessitated the recruitment of a freelance computer

SOLSTICE 2007 Conference, Edge Hill University

3

software consultant to design a bespoke central database which would accept the Pocket Forms©. Several forms were needed to suit the different purposes of the portfolio. We had a form for students to record their experience in each performance indicator and to note any relevant references. Another form recorded the details of each placement interview. Finally an assessment form integrating learning outcomes with assessment criteria was created. At this point we found another limitation of pocket forms in that it was unable to calculate an average mark from a series. We overcame this by creating an excel form for the link lecturer to use to record the marks. This left our designed assessment form somewhat redundant so we decided to use it as a student self-assessment form instead. This idea was not purely opportunistic but rather grew out of existing practice. Students had been asked to self-assess in the paper version but only as a comment on the interview sheet, they had not been asked to grade their own performance using the assessment criteria as a guide. However in academic assignments these students have been grading themselves using assessment criteria for some time.

ACTIVITY THEORY The new forms were used during the students’ second placement at the end of which three focus groups were conducted to explore the student experience of using mobile devices to support assessment in clinical practice. My interest in this project was in discovering how if at all the new PDA forms impacted on student understanding of clinical assessment. I therefore used an activity theory framework to analysis the focus group data.

Activity Theory Mediating artefacts: paper/electronic portfolio, assessment criteria, learning outcomes

Subject: student midwife

Rules: QAA benchmarks, NMC standards, competencies etc

Object: valid and reliable assessment of clinical competence

Community: Clinicians Academics Students

Division of Labour Tripartite system: student, mentor, link lecturer

Activity theory (Engeström 1994) rejects the idea that thinking and learning are best studied as individual cognitive processes; rather it argues that learning is a social process of reaching a shared understanding or meaning and should be studied as SOLSTICE 2007 Conference, Edge Hill University

4

embedded collective activities. Thus we can see from the diagram that what we mean by assessment of clinical practice is embedded in a framework of rules, community and division of labour and mediated by the tools we use to articulate that meaning. In our case study the main tool or artefact used to mediate understanding of the assessment process was the clinical portfolio. By changing this in the process of converting it for PDA use there was the potential to change the shared meaning of assessment for the actors involved.

FINDINGS Findings suggest that the new tools i.e. the electronic portfolio and particularly the new assessment forms, created a new understanding of assessment for students and that they felt it helped develop a shared understanding with their assessor. The new tool seemed to encourage a more detailed appraisal of student performance. In particular discussion based on the student’s self assessment of grade in each criterion grounded the discussion in how the student felt she had performed. It’s just a much better way of getting a grade The way I was graded from my first placement to my second placement was a lot better…it was ‘what do you think to this? where would you put her within this banding? This process of discussing each criterion in turn seemed fairer to the students and also appeared to ensure that full consideration was given to each aspect by the assessor. Students had complained about the previous process using the paper forms where it was easier for the mentor to give an intuitive assessment of student overall performance without necessarily pinpointing strengths and areas for development. I think the new grading of assessment is better it’s fairer, because you can be very good in some things for example communication skills and not as good in others But that makes them have to do it, because I know the grades are in the book but it doesn’t say you have to go through each one and give a percentage for them, it just says look at them – you know what I mean?

DISCUSSION Interviews with link lecturers confirmed the favourable student response and made the lecturers rethink how tools can influence a particular interaction. They were so impressed with the new assessment form that even though the PDAs were not available for subsequent placements they translated the form back to paper copy trying to preserve as many advantages as they could i.e. full student self assessment and the marking of each criterion. Thus it could be argued that the design of the forms rather than the electronic format was the important factor; however without the PDA acting as a catalyst these changes would not have been made. Tools are not neutral. They can change the way we think and act and have the potential to improve the practice of learning and assessment. We have reported elsewhere (Dearnley and Haigh 2006) that this pilot of mobile technology was not an easy process for staff or students to undertake, mainly because

SOLSTICE 2007 Conference, Edge Hill University

5

of the changed practice involved in using a new device – some of these students still shudder at the very mention of a PDA! However the pilot was feasible and we did succeed in developing our practice in ways unanticipated at the beginning. I feel that this success was only possible because we were not imposing a completely new assessment process but building on current best practice thus minimising associated stress to students and staff. Technology moves on and since last year there have been developments in making web-based personal learning systems accessible from PDAs (Pebble Learning Ltd 2006 personal communication). This has the potential to make future work in this area much less logistically challenging but will still necessitate some rethinking of paper based formats.

CONCLUSION We live in exciting times. New technologies allow for the development of learning and assessment tools that change the way we think and act. Tools are not neutral; they are by definition interactive and therefore stimulate interaction, which is multi-dimensional in nature and has the potential to impact on relationships between the student, mentor and lecturer and the assessment process. New technologies demand new approaches to learning and assessment. It is not the case that we simply replicate old and trusted paper friends in digital format, the rewards of such an exercise would barely prove worthwhile in terms of value added. To obtain such rewards we must allow new technologies to work for us. This case study clearly demonstrated that to do this successfully, the new approaches need to be acceptable to the actors involved by building on current best practice and thus minimising associated stress to students and staff. Crucially, pedagogy must drive technological advancement. We must build on what we know about how people learn and how people respond to assessment and be creative in our technological responses.

REFERENCES Dearnley, C. and Haigh, J. (2006) Using mobile technology for assessment and learning in practice settings: the Bradford Pilot. In Assessment for Excellence, Northumbria EARLI SIG Assessment Conference Northumbria University. Engeström, Y. (1994) Teachers as collaborative thinkers: Activity-theoretical study of an Innovative Teacher Team. In Teachers' minds and actions: research on teachers' thinking and practice: 6th International conference: Selected papers(Eds, Carlgren, I., Handal, G. and Vaage, S.) Falmer Press, London, pp. 43-61. HEFCE (2007) Complete list of funded CETLs http://www.hefce.ac.uk/Learning/tinits/cetl/final/cetllist.pdf updated 1st March 2007 accessed 9th April 2007 Pebble Learning Ltd (2006) PebblePAD. www.pebblepad.co.uk

SOLSTICE 2007 Conference, Edge Hill University

6