Mosvold-fauskanger, Aera 2009 Paper

Challenges of translating

CHALLENGES OF TRANSLATING AND ADAPTING THE MKT MEASURES FOR NORWAY

Reidar Mosvold and Janne Fauskanger University of Stavanger, Norway

1

Challenges of translating Abstract This paper reports on the process of translating the U.S. measures of MKT into Norwegian. The main questions addressed in the paper are (a) what challenges were encountered in the process of translating the U.S. measures into Norwegian and adapting them to a Norwegian context? (b) which of these challenges are general, and which appear to be specific to the Norwegian culture? The PISA Technical Report supports the idea of using double translation, and we have used double translation with extensive documentation of the changes that were made during the translation process. These changes were placed in different categories, and we had to put extra emphasis on issues concerning the translation and adaptation from English to Norwegian.

2

Challenges of translating Challenges of Translating and Adapting the MKT Measures for Norway In the Learning Mathematics for Teaching (LMT) project, measures were created in order to analyze teachers’ Mathematical Knowledge for Teaching (MKT). In this paper, we present some of the challenges involved in our attempt to translate and adapt these measures for use with Norwegian teachers. The measures were originally created for use in a U.S. context, and a number of differences between the two countries contribute to increasing the difficulty of using the U.S. measures in Norway. At this point, we have carried out the translation as well as a prepilot study. The main focus of this paper is to address the following research questions: • What challenges were encountered in the process of translating the MKT measures into Norwegian and adapting them for use in Norway? • Which of these challenges are of a general nature, and which appear to be specific to the Norwegian culture? We start by presenting the theoretical background for our project, followed by a short presentation of the Norwegian school system. In the methods section, we present the methods that were used in the translation process as well as in the pre-pilot. This is followed by a presentation of results and a discussion. Towards the end of the paper we present some concluding remarks and suggestions for the road ahead. Theoretical background Research from the last 15 years indicates that “the mathematical knowledge of many teachers is dismayingly thin” (Ball, Hill, & Bass, 2005, p. 14). When analyzing 700 1st and 3rd grade teachers (and almost 3000 students), researchers found that the teachers’ knowledge had an

3

Challenges of translating effect on the students’ knowledge growth (Hill, Rowan, & Ball, 2005). Stigler and Hiebert (1999, p. 10) claim that: “Although variability in competence is certainly visible in the videos we collected, such differences are dwarfed by the differences in teaching methods that we see across cultures”. But even though research indicates that teachers’ knowledge might have a positive influence on students’ learning, it is not obvious what the content of this knowledge is. There are also no clear guidelines for what to focus on in in-service education. Our study focuses on Norwegian teachers’ mathematical knowledge for teaching, and it is closely related to the LMT project1. Theoretically, it follows Shulman’s (1986) efforts to define the theories concerning subject matter content knowledge and pedagogical content knowledge. The categorization of the various components of teacher knowledge has evolved from Shulman’s original proposal, where he distinguished between subject matter knowledge (SMK), pedagogical content knowledge (PCK), and knowledge of curriculum. In the LMT project, this model evolved into a model of mathematical knowledge for teaching (MKT).

1

For more info about LMT, see http://sitemaker.umich.edu/lmt/home

4

Challenges of translating Figure 1. Model of MKT (based on Hill, Ball, & Schilling, 2008). Hill, Ball and Schilling (2008) point to a discussion about how effective teachers have a unique kind of knowledge. This includes knowledge about students’ mathematical ideas and thinking. This domain of teacher knowledge has been identified in the U.S., but there is a possibility that it might differ from the domain of knowledge that is held by effective teachers in Norway. The MKT measures have been developed over several years, and the research team at the University of Michigan has spent a lot of time and money on this. It would therefore have been of great interest if we could build on their efforts and use the same material with Norwegian teachers. Similar attempts have been done in Ireland, and Delaney (2008) points to some possibilities as well as some problematic issues. The process of translating these items is far from straightforward, and there are several issues to be aware of when going into this (cf. Delaney et al., 2008; Mosvold, Fauskanger, Jakobsen, & Melhus, 2009). In our Norwegian project, we do not aim at comparing the knowledge of U.S. and Norwegian teachers. The measures were not built for that purpose. Our aim is to learn more about the mathematical knowledge for teaching that Norwegian mathematics teachers have, and the knowledge they need in order to become (more) effective teachers. Such information would be useful for pre-service as well as in-service education. Although it is not our intention to compare teachers in Norway with teachers in the U.S., it is necessary to investigate whether or not our translated and adapted measures work in a way that will provide meaningful data about Norwegian teachers’ mathematical knowledge for

5

Challenges of translating teaching. It is therefore necessary to go into an analysis of the scores in order to find out if the items that were difficult for U.S. teachers are more or less difficult for Norwegian teachers. If there are significant differences, we have to figure out whether these differences are related to the translation process, to cultural differences, or to other aspects. The Norwegian school system The Norwegian school system is different from that in the U.S., and we point out some issues here. Norwegian schools are divided into three main categories or levels: • Primary and lower secondary education (children aged 6-15) • Upper secondary education (three years after 10th year of lower secondary education) • Tertiary education (influenced by the Bologna Process) In 2007, 616,388 children were attending primary and lower secondary education in Norway (Statistics Norway, 2009). All public education is free of charge, and the Norwegian school system has some important overall aims and precepts: Education for all is a basic precept of Norwegian educational policy. Children must have an equal right to education, regardless of where they live, gender, social and cultural background or any special needs (MER, 2007, p. 5). Based on these more overall aims, the goals and frameworks for Norwegian schools are then defined (and decided) by the Norwegian Parliament and the Government. The Ministry of Education and Research has a particular responsibility for carrying out the national educational policy. National standards are ensured through curricula and framework plans along with laws and regulations (MER, 2007). When it comes to teacher education, one formerly thought that if teachers knew enough

6

Challenges of translating mathematics, their teaching would be good and their students would learn mathematics. The content of in-service education was therefore pure mathematics (Cooney, 1999). On the other extreme, there appeared to be a consensus in Norway that it was possible to become a good mathematics teacher without knowing any mathematics at all (Haaland & Reikerås, 2005). Nowadays, there are several worldwide attempts to pursue excellence in mathematics classroom instruction, and the focus is moving away from the traditional in-service courses. Exemplary lesson development is one example (Huang & Li, 2009), and there have also been recent examples of studies that make use of an inquiry-based approach in Norway (Jaworski, Fuglestad, Bjuland, Breiteig, Goodchild, & Grevholm, 2007). The Ministry of Education and Research (KD, 2006) refers to TIMMS 2003, which draws attention to the fact that Norwegian teachers’ education in mathematics and in mathematics education is below the international average. Another issue is that Norwegian mathematics teachers rarely participate in relevant in-service education (Pedlex, 2008; Grønmo et al., 2004). It therefore appears evident that a focus on strengthening in-service as well as preservice education of mathematics teachers is necessary. Mathematics is a compulsory subject for all pupils in Norwegian schools. In years 1-10, the pupils are supposed to work on the following subject areas (Utdanningsdirektoratet, 2008):

7

Challenges of translating

When compared with the content of the MKT items that are used in our studies, it is important to notice that algebra does not appear as a main subject area in years 1-4. (In our penultimate curriculum, algebra only appeared in years 8-10.) Functions only appear in years 8-10. Throughout elementary school (years 1-7), Norwegian pupils should have altogether 812 hours of mathematics. In lower secondary school (years 8-10), they have 313 hours of mathematics1. Normally, teaching in Norway is organized in a combination of the teacher addressing the whole class and individual work by the pupils (Haug, 2004; Bachmann & Haug, 2006; Alseth et al., 2003). Norwegian classroom research also indicates that the time that is actually spent on mathematics in reality is much lower than the hours presented above (Haug, 2006; Skorpen, 2006). The Norwegian Ministry of Education and Research recently published a strategy for teachers’ in-service education: “Kompetanse for kvalitet” (in English: “Competence for quality”) (KD, 2008b). They underline that knowledgeable teachers are important for students’ learning. The argument is that high mathematical and pedagogical competence among teachers contribute to better results among the students. The strategy document does, however, provide little 1

Norwegian lessons are normally organized in 45-minute units, although these numbers are presented in 60-minute units

8

Challenges of translating evidence for the claim that knowledgeable teachers are important for students’ learning. In order to operationalize the strategy, part of our rationale might be to find out more about the knowledge that is currently held by Norwegian teachers. The Ministry aims at implementing a permanent system for teachers’ in-service education, and 60 ECTS in mathematics is recommended. Little is said, however, about the content of these 60 ECTS. Methods This paper reports on issues that arose in the process of translating and adapting the MKT measures into Norwegian, and on the results from interviews with teachers that were conducted in a pre-pilot study. In this section, we mainly describe methodological considerations regarding the translation of the measures as well as the organization of the pre-pilot study. Translation According to the PISA 2003 Technical Report (Adams, 2005), translation errors are known to be a major reason why some items function poorly in international tests of students’ knowledge. At the same time, studies normally provide little information as to how measurement instruments are adapted for use from one country to another and in the different publications little information is given about translation issues arising in the research, particularly in the case of measures of teachers’ knowledge (Delaney et al., 2008). Translating the MKT measures into Norwegian is not only a matter of word choice. It is also a matter of adapting the measures for use in a cultural context that is quite different from the original. This is particularly important with the MKT items, since they were not originally created for use outside the U.S. The items are strongly grounded in the practice of teaching

9

Challenges of translating mathematics, which may vary across countries, and it is therefore necessary to include experts of teaching in the process of translating and adapting the items. After the translation process, an instrument should continue to measure the same characteristics it was intended to measure (Geisinger, 1994). An important methodological goal for translating the MKT measures into Norwegian therefore is to ensure equivalence at the level of context and opportunity. Various terms are used in cross-cultural research to describe different aspects of equivalence. According to Johnson (1998), the terms are not always well defined and considerable overlap exists among them. An attempt to adapt the U.S. measures to an Irish context (cf. Delaney, 2008) emphasized the need to establish whether the MKT construct is equivalent in different settings. To focus on construct equivalence is thus an important aspect of our validation process. Singh (1995) outlines six steps that contribute to construct equivalence, three of which should be studied even before using the measures to collect data: functional equivalence, conceptual equivalence and instrument equivalence (Singh, 1995; Delaney, 2008). We therefore focus on these terms at this point. Functional equivalence relates to whether or not the MKT construct serves the same function in Norway as in the U.S. In order for students to acquire knowledge, the teacher must have some kind of knowledge related to teaching (in this case, MKT). MKT is the mathematical knowledge needed to teach mathematics. This construct has a universal function, and thus satisfies the requirements of having functional equivalence (cf. Delaney, 2008). Conceptual equivalence relates to the question of whether the construct of MKT means the same in Norway as in the U.S. or not. To answer this question in an Irish context, Delaney

10

Challenges of translating (2008) examined the construct more closely by studying the work of teaching in Ireland. He compared that work to conceptions of the work of teaching that informed the development of MKT. Delaney also studied literature about the construct, and he analysed items based on the construct. He found relatively minor differences in his analysis. One possible explanation might be that these two countries share a common language. This could make it easier for ideas and conceptions about teaching to travel back and forth between the U.S. and Ireland. Norway and the U.S. do not share a common language, so it is possible that more differences may emerge if the tasks that informed the MKT were compared to tasks of teaching in Norway. Since we have to take into account the added complexity of a different language, attempts to ensure conceptual equivalence will be important in our work, but is not included at this point Instrument equivalence might be related to both the format and the contents of the items. If the multiple-choice items are equally interpreted in Norway and the U.S., we have instrument equivalence (cf. Delaney, 2008, referring to Singh, 1995). When translating a set of items into a different language, it is important to focus on what Peña (2007) and others call linguistic equivalence. The translation needs to be of high quality. In our case, there is a strong connection between what some researchers refer to as linguistic equivalence and what others (cf. Singh, 1995) refer to as functional equivalence, conceptual equivalence and instrument equivalence. In order to assure linguistic equivalence, we used a double translation procedure (Adams, 2005). The translation of the items took about half a year. Towards the end of this period, we had a working seminar where the translations were finished. In this seminar, we worked in pairs (or sometimes three) and translated all the items. Two groups

11

Challenges of translating of two researchers would work separately on the same set of items. The translations were then discussed in the larger group, and the entire process resulted in one translated set of items. Peña (2007) claims that it is not enough to use certain translation techniques in order to ensure linguistic equivalence. In our case, it was a matter of ensuring whether the construct of MKT meant the same to Norwegian teachers as to American teachers. It was also important for us to address issues related to the format as well as the content of the items. All the researchers who participated in this work are involved in teacher education. Some have background from teaching in schools at different levels, and some have a more purely mathematical background. Pre-pilot As part of our work on translating and adapting the MKT measures, we conducted a prepilot study. The aim was mainly to have a quality check of our translation before the pilot study, but the pre-pilot study can also be seen as an important part of the process of adapting the items to a Norwegian context. We conducted two focus group interviews. Our plan was to conduct one focus group interview with a mixture of experienced and inexperienced teachers, but it was difficult to set a time that fit into everyone’s schedule. We therefore ended up having two interview sessions instead of one. In both sessions, the participants replied to the measure items individually before the interviews. They were asked to write down comments they might have on items as they were going through them, and they were especially asked to be conscious about whether or not the items represented contexts that would be familiar to Norwegian teachers.

12

Challenges of translating When everyone had finished the test, we had a small break before the focus group interview started. In the focus group interviews, we asked the participants to give general comments about the test. Did they find the items difficult? Were there some elements of mathematical knowledge for teaching (as they have experienced it) that they thought were missing from the test? Did they have comments regarding the format of the questions? After a round of such more general questions, we went through every item in the test, one by one, and asked them to make comments. The focus-group interviews were recorded with a digital video camera and an digital audio recording device, in order to provide ourselves with a material that could be used for analyzing more than the discussion alone (e.g. their use of gestures). One of the groups consisted of three newly certified teachers, who were students in a masters program in mathematics education. All three were female students. The second group consisted of two experienced teachers, one man and one woman. They both work in elementary school, but the woman also has spent some years as a teacher in lower secondary education. The male teacher finished his teacher education 25 years ago, and he has worked a lot with students with special needs. The female teacher also has 25 years of experience, and she has been a teacher in lower secondary school for several years. She only has a small unit of mathematics in her teacher education (15 ECTS in our terms). The last 10 years, she has worked exclusively with students in the first years of elementary school. These teachers were selected because we knew them, and we believed that they could provide interesting feedback for this phase of our study. We also thought it would be interesting to see examples on how both experienced and inexperienced teachers might react to the

13

Challenges of translating measures. Results and discussion In this part of our paper, we will present and discuss the results from our translation process as well as from the pre-pilot. Issues regarding translation Throughout the translation process, we carefully documented all changes that were made to the items (other than direct translation from U.S. English to Norwegian). This was done because we suspect that these changes might influence the teachers’ responses to the items. Delaney and colleagues (2008) report on a similar study that was carried out in Ireland, and they summarized their changes in the following categories: 1. Changes related to the general cultural context 2. Changes related to the school cultural context 3. Changes related to mathematical substance 4. Other changes Delaney and colleagues (2008) recommended their own results as working guidelines for others who attempt to adapt the items, so we decided to use these categories in our translation process as well. They included altering spellings to reflect differences between American and British English in category 1 above (changes related to the general cultural context), but we decided to have the translation from U.S. English to Norwegian as a separate category. The translation and adaptation from American English into Norwegian was far more complex than the process of adapting the items for use in Ireland (as described by Delaney,

14

Challenges of translating 2008). We therefore had to develop the list of categories further, and we ended up with two new, in addition to the original four categories above: 1. Changes related to the translation from American English into Norwegian in this particular context 2. Changes related to political directives In this paper we will focus on the two categories that evolved from our translation process because these challenges may be specific to the Norwegian case. The example below is from the set of released items. This particular item was not included in our test, but it is presented here as an example of the types of changes that were made during our translation process.

This is how it looks in the Norwegian translation: 10. Elevene til Hans har arbeidet med å sortere desimaltall i stigende rekkefølge. Tre av elevene, Anders, Klara og Kristin, sorterte desimaltall slik: 1,1 12 48 102 31,3 0,676. Hvilken feil er det disse elevene gjør? (Marker ETT svar.)

15

Challenges of translating a) De ignorerer plassverdi/posisjonsverdi. b) De ignorerer desimalkomma. c) De gjetter. d) De har glemt at det fins tall mellom 0 og 1. e) De gjør alle feilene ovenfor.

We discussed several issues in relation to the translation of this item. First, Norwegian students are referred to as pupils (or elever in Norwegian) as long as they are in compulsory school, and students when they enter university. We also changed the name from Mr. Hayes to Hans, which is a common Norwegian first name. In Norway, it is common for pupils and colleagues to address teachers with their first name only. This might vary somewhat according to the teachers’ age and the level in which they teach, but in lower secondary school the pupils would normally address their teacher as Hans rather than Mr. (Hans) Hayes. If we decide to keep the more formal American setting, most Norwegian teachers would find this different from what they are used to, and they might therefore not experience this as a familiar setting. The result of this might be that they were distracted from the substance of the question. When making changes from the American names like Mr. Hayes and Ms. Wilson to more common Norwegian first names like Hans and Marianne, we are also adding a potential complexity to the item in that it becomes more difficult to distinguish between the teacher and the pupils in the problem context, since both are referred to by their first names. We therefore had to change some of the items and sometimes include some extra information in order to clearly distinguish between the pupils and the teacher. When making these changes in names, we were conscious about preserving the teacher’s gender. When changing the first names (of pupils) to names that are more common in Norway, we tried to find names that were somewhat similar, but this was not always done. Another issue is that the original sentence referred to the students in Mr. Hayes’ class, 16

Challenges of translating and you are not supposed to refer to a class of students in Norway. In 2003, the Norwegian Education Act (Opplæringsloven) §8-2 (1998) was changed into a statement that students can be organized in groups according to their needs. Prior to this, the law stated that Norwegian students should be organized in classes. Schools are still allowed to organize their students in traditional classes, but the Education Act no longer use this term, and most official documents refer to groups of students rather than classes (KD, 2008a). We therefore decided to translate class into group, or simply rewrite it somewhat. As described above, we have added a category about political correctness to Delaney’s categories, and changing class into group is one example. This decision to use the word group instead of class in Norwegian schools might appear trivial, but there is more to it than what we have described above. It appears that a large number of teachers continue to use the word class, although the Ministry has decided to avoid it, and for these teachers the word group as a replacement to class could be both confusing and misleading. If we decided to go for the traditional term class, which is no longer the officially correct term, we would probably be faced with a large number of teachers who would argue that our measures were not up to date, and not in line with the official guidelines. In order to be politically correct, we have chosen to rewrite the sentences that originally referred to class and use pupils (elever) instead. We could have used group instead, but that could lead to confusion in some instances, because the same word is also used when we refer to group work. Sometimes we therefore ended up rewriting the item somewhat. For example: “Mr. Hayes’s class” is translated into “Elevene til Hans” (English: Hans’s pupils). Another example we have chosen to put in this category is use of the verb to learn. In a Norwegian context, we normally refer to learning as the outcome rather

17

Challenges of translating than the process. As a result, we find it inappropriate to say that “Mr. Alder’s students are learning about...”, since we cannot know if they have actually learned it. In items that refer to the learning process, like in the example just mentioned, we therefore decided to rewrite it somewhat. A Norwegian translation would be: “Elevene til Anders arbeider med...” (in English: “Andy’s students are working with...”). The passage about putting decimals in order was also discussed. It is more common to sort (sortere in Norwegian) numbers than to put them in order. We also talk about decimal numbers (desimaltall) rather than decimals in Norway. To make this passage sound better in Norwegian, we moved some of the information from the second sentence to the first. As a result, it seems as if Mr. Hayes’ students only worked with ordering decimals from least to greatest, whereas the original U.S. idea might have been that they worked with ordering decimals in different ways, although the students in this particular example ordered the numbers from least to greatest. The first sentences in the stem of this item were difficult to translate directly into Norwegian, and we decided to rewrite them somewhat. When doing this, there is always a possibility of interpreting the sentences in a way that has removed or added information to the item. In Norway, we use a decimal comma rather than a decimal point, and since comma was used to separate the different numbers that were presented, we had to change this to avoid confusion. One possibility could be to represent the numbers like this: 1,1 - 12 - 48 - 102 - 31,3 0,676. From a linguistic point of view, this might be a proper solution, but in a mathematical setting there might be a danger of confusing the - with a subtraction sign. We also discussed the

18

Challenges of translating possibility of using semi-colon instead of comma to distinguish the numbers, but we decided that this would result in too much clutter. We therefore ended up representing the numbers separated by extra space. In addition, we had to change .676 into 0,676, because decimal numbers lower than one are never written without the zero in Norwegian. In the alternative solutions, we spent some time discussing alternatives a) and d). In a), there is a reference to place value, and we might use the similar word plassverdi in Norwegian. Several teachers would rather prefer to use posisjonsverdi instead, and we decided to include both alternatives to avoid confusion. Both these words mean the same, and they do not add any information to the item. Alternative d) was more problematic to translate. When Americans talk about forgetting your (or their) numbers, this is hard to translate directly into Norwegian. Our translation therefore had to be an interpretation rather than a direct translation. After some discussion, we agreed that the meaning of this sentence must be that the pupils have forgotten that there are numbers between 0 and 1. Another interpretation might be that they did not know this, and a translation into Norwegian might then be: De kan ikke tallene mellom 0 og 1 (They don’t know the numbers between 0 and 1). Such a translation might, however, indicate that the pupils have never been taught this, and we believe that this is not the correct understanding of this alternative solution. We ended up with the following translation: De har glemt at det fins tall mellom 0 og 1 (They have forgotten that numbers exist between 0 and 1). The mathematical language used in Norwegian schools of course differs from the language used in schools in the U.S. In most cases precise translations of the terms were possible, but the mathematical language that is used in Norwegian schools is often translated into

19

Challenges of translating a more everyday language. For example, hexagon does have the Norwegian translation heksagon; polygon could be written the same way in Norwegian as in English; and congruent might be translated into the Norwegian word kongruent. Our impression is that these more precise mathematical terms are rarely used in Norwegian schools. This impression is supported by the national curriculum, where one of the competence aims after Year 4 is that the pupil shall be able to “recognise and describe characteristics of circles, polygons, spheres, cylinders and simple polyhedrons” (Utdanningsdirektoratet, 2008). The strange thing is that the original Norwegian version of the curriculum does not include the word polygon, although the official English translation does. In the Norwegian version, the word mangekant (English: multi-edge) is used. In our study, these terms were translated in the following way: hexagon – sekskant, polygon – mangekant and congruent – helt lik (English: exactly the same). This could be problematic, because there is a risk that these changes make the items easier for the Norwegian teachers. The masters students also found it difficult when precise mathematical concepts are used and our impression was emphasized when one of them in the pre-pilot reacted to the word tessellate: FS1: “I tessellate”, then I had to (…) That didn’t tell me anything. It didn’t provide me with any information. JF: The word tessellate? FS1: Mm. FS3: I reacted to the word myself. I would guess that not so many know what it is.1 1

The interviews were held in Norwegian, and the transcripts were translated by the authors. FS1, FS2 and FS3 are used to distinguish the three female students. JF are the initials of Janne Fauskanger, who was conducting the

20

Challenges of translating It is interesting to notice that this term was so unfamiliar to these masters students, and it was as a result of their suggestion that we decided to add an explanation to this particular term. This is a significant change to the original, and it is likely that this item will be somewhat easier for the Norwegian teachers because of this. The multiple-choice format The multiple-choice format has not been widely used in Norwegian schools and might be unfamiliar to the Norwegian teachers. This is an issue that should be discussed, because it may cause validity problems. We have seen indications that this might be changing in Norway, and this appears to be related to the increased use of digital tools in particular. It is conceivable that in a culture where multiple-choice formats are unfamiliar, one may have to change the format. But changing the format may be problematic as well, because it could influence the item’s level of difficulty, and it might also make scoring the items more difficult (Delaney et al., 2008). It could also make the items more or less discriminating or change how effectively they measure the underlying constructs. We have decided to keep the multiple-choice format for now and evaluate the matter after the pilot study. But our pre-pilot study points our attention to the format, and we have to be prepared to change it. When interviewing the three masters students, we started by asking them how they felt about doing a multiple-choice test. One of them said that the only experience she had with this format was in connection with the theoretical exam she had to take in order to get a driving license. She found the multiple-choice format difficult and said:

interview. Likewise, RM are the initials of Reidar Mosvold. When (…) is used in the transcripts, it indicates that something has been omitted, and when [something] is marked like this, it represents a comment or addition by us.

21

Challenges of translating You become uncertain, because everything is kind of similar, partly right. And then you have to choose something. I think it’s easier when you can simply make your own answer, instead of having to choose some pre-produced [answer]. One of the other students did not agree. She said: I have nothing against multiple-choice (...) not in these kinds of tasks and surveys, but if it occurs when I am supposed to sit down and produce something on an exam, I wouldn’t be so happy about it. Because then I feel that I sort out the possible answers, and I would have thought in a different way, or done things differently… The students were told that the reason why we conducted the pre-pilot study was to “test the test” and not to “test their knowledge”. When we asked them if they would look at the format in a different way if they were teachers taking the test, they all said “Yes”. One of them stated the reason for the “Yes” by saying: “You are kind of forced into someone else’s way of thinking”. The other students said “mm” to indicate their agreement. The interview continued by discussing each of the items. In this discussion especially one of the students had statements showing that she was not comfortable with the multiple-choice format. She said: Multiple-choice makes me [uncertain], like everything is right (laughter) ... No, I won’t [do that]. What she said also indicates that the format controlled her response. When she was explaining one of her responses she started to laugh and said: “I guess it was because I didn’t think there would only be ones”. The idea that there would be a pattern, or that the correct alternatives would be spread, therefore appeared to have an impact on her response. This is a problematic issue, and it might indicate that the responses that the participants make not only illustrate their thinking or knowledge about the issues described, but also their anticipation of how the correct

22

Challenges of translating alternatives are distributed. If they were more familiar with multiple-choice tests, this might not have been an issue. Are the MKT measures suitable for the Norwegian tradition? Both the experienced teachers and the masters students suggested that some of the items would be too difficult for Norwegian teachers, especially for the teachers working at the lower grades. One of the masters students repeatedly directed our attention to the items including algebra. For example she said: I think that from page 17 and onwards, the tasks become much more difficult, and I immediately think [they fit for teachers in] lower secondary school. First there are fractions and calculations and stuff, but from page 17 and onwards there are lots of theories that belong in lower secondary school. So perhaps questions become harder because of that. (…) There are many who teach mathematics in elementary school, who might not have so much knowledge concerning the mathematics that comes after [elementary school]. The two other students agreed, and one of them recommended us to intersperse the algebra items throughout the form instead of having all towards the end as in the original: ... so that you don’t get the cramps in the end. That you feel it becomes harder. Especially if this test is supposed to be for teachers in elementary school. The experienced teachers also found the items too difficult, especially for teachers that took their initial teacher education more than 20 years ago. One of the teachers said: It might be that recently educated [teachers] have been through a teacher education that enables them to do this, but those who have taught for ten years and more, I am sure they are not going to feel at home with this or feel competent. The other teacher agreed with this. Even though the interviews revealed that some of the items would be difficult for Norwegian teachers, they also indicated that several items were quite easy. RM: Item 11? 23

Challenges of translating FS2: I think that one was easy enough. RM: Is it something in particular that made it easy? FS2: I don’t know. FS1: Pizza is kind of... FS2: (…) I think like this: Okay, so I imagine a task, and I ask myself: “What is the answer to this task?” And to help me find out if the problem is right or not, then I think about what the answer to the problem might be. So there is a little answer book for my own part, and in addition to just trying: could this work, does it seem logical, then I think like this. Although the teachers and the masters students suggested that some of the items would be difficult for Norwegian teachers, we decided to avoid changes that might influence the difficulty of the items. We also decided to include the complete set of items in the same order that they were in the original measures, although the respondents suggested that we might spread the algebra items to make it appear less difficult towards the end. Another perspective concerns the relevance of the contexts included in the items. The masters students found some of the contexts irrelevant for the Norwegian context. Baking cookies represents a context that is familiar in an American setting, whereas this was not conceived as a familiar activity in an Irish context. In their article, Delaney and colleagues (2008) changed this activity into one of baking scones. Neither of these are familiar activities in a Norwegian setting. The challenge is to find a good alternative for the translation, and at the same time avoid changing the problem in a way that influences the mathematical challenges involved. We decided to use the activity of baking chocolate cookies/biscuits in the pre-pilot study to see how this worked out, although this is an activity that few Norwegians find familiar.

24

Challenges of translating The masters students reacted to this. FS1: I wonder how many kids bake chocolate cookies. [...] FS2: If it had been 3 cups of flour and they were making pancakes, would that be okay? FS1: Yes, it might be better. Chocolate cookies are so American. There aren’t many (…) who have baked chocolate cookies. Christmas cookies maybe, or sweet buns or waffles, or ... But chocolate cookies, that’s something you buy rather than bake. After the pre-pilot study we decided to change this context from the chocolate cookies to the more familiar Norwegian sweet buns with raisins. This is a context that more Norwegians would find familiar, and we did not have to change any of the numbers or mathematical considerations that were involved in the item. The masters students did not always agree on the relevance of the contexts, and the item that included a context where a paper frog was moved along the number line was particularly troublesome for some. Since the pre-pilot study did not show agreement among the participants at this point, we decided to keep the item unchanged. The item that included a reference to the Tetris video game also confused the master students: FS1: (…) I was annoyed by that one. How did you get that one? FS2: Because if you rotate, then it is that one. FS1: Are those the ones you’re supposed to... FS2: Yes, it’s these figures up here you’re supposed to rotate. FS1: Oh, I rotated these [figures]. Then all of them matched! 25

Challenges of translating FS2: If you rotate that one, anyway, then it will not be similar to that one. FS1: (…) I played an awful lot of Tetris when I was little, so I only imagined which would fall down on that one to make it disappear. When the item included a reference to Tetris, this masters student made use of her knowledge of this video game and interpreted the item in a way that was not originally intended. The intention was that the figures that were rotated were similar to the figures that appear in Tetris, but the Tetris game strategy was not meant to be applied in the item. This distinction was apparently unclear to this masters student. The amount of text included in many of the items was also a problematic issue that appeared in the interviews in the pre-pilot study. One of the master students described herself as a slow reader, and she said that the amount of text in some of the items made them difficult for her. FS2: In item 3 I had to read the text three or four times before I (…) FS1: There was a lot of text. FS2: Yes. But I guess that was part of the idea as well, that it should uncover misconceptions, I don’t know… RM: Was it something in particular that resulted in you having to read it several times, or was it simply the amount of information? FS2: I guess it was just that there was a lot of information in the text compared with what you are used to when working with these kinds of tests. To keep our translated MKT measures as close to the original as possible, we decided to keep the amount of text unchanged. The amount of text is closely related to the time it takes to complete the items. The participants did not have a time limit for the test, but we indicated that they would

26

Challenges of translating normally use 60-90 minutes on the entire form. Although there was no time limit, one of the masters students said that she had felt a time pressure simply because the others had finished before her, and we were sitting there, waiting for her to finish. FS1: I had little time. Cause there was too much to read. (…) I would have preferred to have another half an hour. (…) And I can tell you this: I feel it in my body, even when this wasn’t an exam or anything. I don't like to be the last one to finish these tests. I think it sucks to be the last one, when you two are sitting there waiting for me to finish. (…) It stresses me out. I would only wish that I could sit in peace and quiet, you know. A last but not least important issue is the concern about how we as researchers are going to use the results. One of the masters students refer to a test she and her fellow students were given when they started their own teacher education. This test had no relation to the MKT measures. The student told us that the test was given without any reason or explanation, and some weeks later the newspaper headlines proclaimed that the teacher students did not know their mathematics. FS1: They [teacher students] didn’t know the 10th grade curriculum and stuff. JF: What do you think about that? FS1: (…) I do think it’s sad to be presented like that, because there is a limitation to what these kinds of tests can say about your knowledge. One of the masters students also recommended us to change one aspect about the information that was given as an introduction to the test. FS1: I would like to say another thing. In the introduction, you shouldn’t say that those who perform well on this test will have good results in their class. That would seem like... The teacher who sits there and might not get any answers right, and who is thinking: “What about me then?” FS2: Yes, like we said before, yes. 27

Challenges of translating FS1: At least not if you want them to play along nicely. We are aware of the fact that the American tradition in which the MKT measures are developed may be different from the Norwegian tradition. We may experience that teachers are unwilling to participate in our study as the following transcript from our interview with the experienced teachers indicates. FT: [My] first impression is that this was generally quite hard. Lots of [teachers] in Norwegian schools would be shocked by this, and they would refuse to participate in a course if they were faced with this beforehand. Norwegian teachers do not enjoy the feeling of being tested, and this might at least be partly related to the fact that Norwegian newspapers have been writing lots of negative articles about the quality of Norwegian teachers. Every time an international test reveals that Norwegian students’ performance in mathematics is worse than we would like it to be, newspapers write about how bad our schools are. There is also an ongoing debate in Norway about the possibility of paying successful teachers more, and this is another difficult issue that is partly related to the idea of testing or evaluating the teachers. The issues presented in this section are some of the aspects we intend to investigate further in our pilot study. Conclusion The questions this paper intended to answer was on the one hand regarding the nature of the challenges involved in the process of translating and adapting the challenges for use in Norway, and on the other hand we wanted to find out more about which of these challenges were general and which were special to the Norwegian case. The latter question is arguably more

28

Challenges of translating difficult to answer than the former. Another issue that has arisen is even more difficult to approach: • What criteria will guide a decision as to whether or not our translation and adaption has been successful? This question will be important to pay careful attention to in relation to our pilot study, and we are going to present a beginning of this discussion towards the very end of this paper. First, we are going to pay attention to the initial questions. Translating and adapting the MKT measures for use in Norway is both complex and difficult, as the issues we have brought up in this paper indicate. It is not simply a matter of making a literal translation, it is not even about making a high quality translation, and several important aspects can be lost in translation. Even changes that appear to be trivial have the potential of making the items more complicated, easier to misunderstand, or even easier to understand than was the original intention. Delaney and colleagues (2008) reported that they changed some of the names in order to make them more familiar in an Irish setting. Still, they continued to refer to the teachers as Mr. or Mrs. In Norway, it is more common to address the teacher with his or her first name, and we had to change the items according to this. Because of this, it was no longer evident which names belonged to teachers and which belonged to students. In several instances, we therefore had to rewrite the item somewhat in order to make this clear. Although this might be considered as a minor issue, it could potentially make the items more complex. This issue is special to the Norwegian context, but it might also be relevant in some other countries.

29

Challenges of translating An issue that is more special to the Norwegian context is that regarding the use of the word “class” when referring to groups of pupils. According to the Norwegian Education Act, schools are no longer obliged to organize their pupils in traditional classes, and most official documents refer to “groups of pupils”, or they use both “groups” and “classes” to indicate that there is no longer a demand that students should be organized in classes. In several schools, they are very conscious about not using the word “class”, and in order to avoid this issue we decided to rewrite the items that referred to this word. This issue is likely to be more related to the Norwegian context in particular than many of the other issues that we encountered. In the pre-pilot study, both the experienced teachers and the more inexperienced masters students agreed that several of the items would be difficult for Norwegian teachers. Some of them also stressed the fact that several items included a lot of text. These issues might be rooted in differences in curriculum content or other issues related to school culture. We can only assume that American teachers did not feel the same way about this, but to our knowledge American teachers have not been consulted about the items in the same way that we are doing, and we therefore do not know if these issues are special to the Norwegian context or not. In our pilot study, we intend to evaluate the success of our translations and adaptations by using focus group interviews in addition to analyzing the results from the survey. We plan on asking a selection of the teachers in detail about whether the items appear authentic to them or not, and whether the mathematical content of the items is of a kind that Norwegian teachers encounter in their regular teaching practice. The teachers will also be asked to comment on each particular item in the measures. Data from these interviews might help explain errors or

30

Challenges of translating responses that we would not expect based on previous analysis of the results from the American teachers. These data might also help us formulate alternative questions or response options in a future adaptation of the items, if such an adaptation proves to be necessary. In the Irish study, an analysis of the interview data was fruitful in order to identify which items may cause difficulties for the teachers, and whether the situations and characters described appeared authentic to teachers. After we have carried out and analyzed the results from the pilot study, we are going to find out if the issues from our translation and from the pre-pilot study that have been pointed out above are actual problems or not. If some of the items appear to be significantly easier or harder for Norwegian teachers than for American teachers, we have to go into an analysis of the interviews to try and figure out why this is. There is a possibility that the MKT construct in the Norwegian setting differs so much from the American that we have to make more serious adjustments to the measures, and there is also a possibility that we may have to develop our own measures from scratch. The really difficult question is still this: When and how can we decide whether or not the translations and adaptations that we have made are successful? If some items appear more difficult or easier to Norwegian teachers than to American teachers, we have to analyze why that is the case. An analysis of the interviews or an analysis of curriculum papers, textbooks or classroom practice might be necessary in order to do this. Norway and the U.S. have different types of curriculum frameworks and standards, and such an analysis would probably be rather complicated to make. Even if the results of the Norwegian teachers are more or less similar to the American teachers, it does not necessarily imply that the MKT construct is

31

Challenges of translating the same. Since the construct of MKT is so closely related to teaching as a practice, the use of these measures alone is probably not enough. Our suggestion is that comparative studies are necessary in order to reach a conclusion about this, and such comparative studies should probably include approaching the translation, adaptation and testing of the measures in different countries along with interviews and analysis of classroom practice (e.g. video studies of some kind). Although this risk of failure is actual and present, we believe that it is important to try. By going into such a study with a critical view, we might learn something important about the constraints and possibilities that are entangled in the process of translating, adapting and using measures and assessments across language and cultural barriers. The potential rewards from such an endeavor appear to outweigh the risks that are involved, and we find it important for us as researchers to shed light on these issues in order to prevent uncritical use (and abuse) of such measures. On the other hand, it would have been useful for the research community if some common or more universal construct of MKT could be found. If this is impossible, it would still be worthwhile to investigate the possible differences between the different cultures. References Alseth, B., Brekke, G., & Breiteig, T. (2003). Endring og utvikling ved R97 som bakgrunn for videre planlegging og justering: matematikkfaget som kasus. Notodden: Telemarksforsking-Notodden. Adams, R. (2005). PISA 2003 Technical Report. Organization for Economic Co-operation and Development. 32

Challenges of translating Bachmann, K., & Haug, P. (2006). Forskning om tilpasset opplæring. Forskningsrapport nr. 62. Volda: Høgskulen i Volda. Ball, D. L., Hill, H. C., & Bass, H. (2005). Knowing Mathematics for Teaching. Who Knows Mathematics Well Enough To Teach third Grade, and How Can We Decide? American Educator (Fall 2005), 14-17+20-22+43-46. Cooney, T. J. (1999). Conceptualizing teachers’ ways of knowing. Educational Studies in Mathematics (38), 163-187. Delaney, S. (2008). Adapting and using U.S. measures to study Irish teachers’ mathematical knowledge for teaching. Unpublished PhD-Thesis. Delaney, S., Ball, D., Hill, H., Schilling, S., & Zopf, D. (2008). “Mathematical knowledge for teaching”: adapting U.S. measures for use in Ireland. Journal of Mathematics Teacher Education, 11(3), 171-197. Education Act [Opplæringsloven] (1998). Lov om grunnskolen og den vidaregåande opplæringa (opplæringslova) av 17. juli 1998 nr. 61. Retrieved March 4, 2009, from http://www.lovdata.no/all/hl-19980717-061.html Geisinger, K. F. (1994). Cross-Cultural Normative Assessment: Translation and Adaptation Issues Influencing the Normative Interpretation of Assessment Instruments. Psychological Assessment, 6, 304-304. Grønmo, L. S., Bergem, O. K., Kjærnsli, M., Lie, S., & Turmo, A. (2004). Hva i all verden har skjedd i realfagene: norske elevers prestasjoner i matematikk og naturfag i TIMSS 2003. Oslo: Institutt for lærerutdanning og skoleutvikling, Universitetet i Oslo.

33

Challenges of translating Haug, P. (2004). Resultat frå evalueringa av Reform 97. Noregs Forskingsråd. Retrieved February 12, 2009, from: http://www.forskningsradet.no/servlet/Satellite? c=Page&pagename=reform97/Hovedsidemal&cid=1231248838727 Haug, P. (2006). Begynnarundervisning og tilpassa opplæring. In P. Haug (Ed.) Begynnaropplæring og tilpassa undervisning – kva skjer i klasserommet? (pp. 19-54). Bergen: Caspar Forlag. Haaland, I., & Reikerås, E. (2005). Matematikkfaget ved lærerutdanningen i Stavanger. In M. Lea (Ed.), Vekst og utvikling. Lærarutdanninga i Stavanger 50 år (pp. 55-65). Stavanger: Universitetet i Stavanger. Hill, H., Ball, D. L., & Schilling, S. (2008). Unpacking “pedagogical content knowledge”: Conceptualizing and measuring teachers’ topic-specific knowledge of students. Journal for Research in Mathematics Education, 39(4), 372-400. Hill, H. C., Rowan, B., & Ball, D. L. (2005). Effects of teachers’ Mathematical Knowledge for Teaching on Student Achievement. American Educational Research Journal, 42(2), 371406. Huang, R. and Li, Y. (2009): Pursuing excellence in mathematics classroom instruction through exemplary lesson development in China: a case stuy. ZDM. Retrieved January 29, 2009, from: http://www.springerlink.com/content/j271g68518161338 Jaworski, B., Fuglestad, A.B., Bjuland, R., Breiteig, T., Goodchild, S., & Grevholm, B. (Eds.) (2007). Læringsfellesskap i matematikk - Learning Communities in Mathematics. Bergen: Caspar Forlag AS.

34

Challenges of translating Johnson, T. P. (1998). Approaches to equivalence in cross-cultural and cross-national survey research. In J.A. Harkness (Ed.), Cross cultural survey equivalence (pp. 1-40). Mannheim, Germany: Zentrum für Umfragen, Methoden und Analysen (ZUMA). KD (2006). Et felles løft for realfagene. Strategi for styrking av realfagene 2006-2009. Oslo: Kunnskapsdepartementet. KD (2008a). Kvalitet i skolen. Stortingsmelding nr 31 (2007-2008). Oslo: Kunnskapsdepartementet. KD (2008b). Kompetanse for kvalitet. Strategi for videreutdanning av lærere. Midlertidig versjon. [Electronic Version]. Oslo: Kunnskapsdepartementet. Retrieved December 19, 2008, from http://www.regjeringen.no/upload/KD/Vedlegg/Grunnskole/Kompetanse %20for%20kvalitet.pdf. MER (2007). Education - from Kindergarten to Adult Education. Oslo: Ministry of Education and Research. Retrieved February 10, 2009, from: http://www.regjeringen.no/en/dep/kd/ Documents/Brochures-and-handbooks/2007/Education-in-Norway-2007.html?id=475839 Mosvold, R., Fauskanger, J., Jakobsen, A. & Melhus, K. (2009). Translating test items into Norwegian - without getting lost in translation? Manuscript submitted for publication. Pedlex Norsk Skoleinformasjon [Pedlex] (2008). Kunnskapsløftet. Fag og læreplaner i grunnskolen. Oslo: Pedlex Norsk Skoleinformasjon. Peña, E. D. (2007). Lost in translation: Methodological considerations in cross-cultural research. Child Development, 78(4), 1255-1264. Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational

35

Challenges of translating Researcher (15), 4-14. Singh, J. (1995). Measurement issues in cross-national research. Journal of international business studies, 26(3), 597-619. Skorpen, L. B. (2006). Kunnskapstypar og arbeidsformer i matematikk i begynnaropplæringa. In P. Haug (Ed.) Begynnaropplæring og tilpassa undervisning – kva skjer i klasserommet? (pp. 115-152). Bergen: Caspar Forlag. Statistics Norway (2009). Facts about education in Norway 2009 - key figures 2007. Statistics Norway, Division for Education Statistics. Retrieved February, 10, 2009 from: http://www.ssb.no/english/subjects/04/02/facts/ Stiegler, J. W., & Hiebert, J. (1999). The Teaching Gap. Best ideas from the world’s teachers for improving education in the classroom. New York: The Free Press. Utdanningsdirektoratet (2008). Mathematics subject curriculum. [Official English translation of the subject curriculum for mathematics] Retrieved March 4, 2009, from http://www.utdanningsdirektoratet.no/upload/larerplaner/Fastsatte_lareplaner_for_Kunns kapsloeftet/english/Mathematics_subject_curriculum.rtf

36