Online Environment Evaluation

Evaluating students' performance in discussion boards in Thinking and Creativity blended course at Sultan Qaboos University

Hiba AL-Julandani

Tamadher AL-Za'abi

u$%&%'()s*u+edu+om

u$,-..%)s*u+edu+om

Instructional and Learning Technologies Department, College of Education, Sultan Qaboos University May (, .$$(

Abstract This paper examines students' performance in asynchronous discussion in Thinking and Creativity blended course. Specifically, students' participation in terms of number of messages posted in different discussion boards. Moreover, the quality of messages posted is also examined. Students' participation is observed throughout the course to address the number of messages posted by each individual student. The quality of messages is examined according to a checklist that contains items about message start, opinion responses, clarity and topic context. Introduction As we are moving fast in the digital age, here at Sultan Qaboos University many applications have been implemented by academics. The entrance of online learning through moodle and WebCT has influenced students learning and performance. Using moodle allows asynchronous communication between students and instructors. In the discussion boards, an instructor and students in the course share their thoughts and ideas with each other and essentially engage in "conversation.” This is an important aspect of fully online courses. But, what about blended courses. Usually, the more interaction between and among the students, instructor, and course content, the more successful the distance course. In the case of blended courses, there should be a balance between class activities and online activities. Repeating same activities makes learning process boring for both instructor and students. Participants and context of the study Participants are 240 students. The participants are enrolled in three sections of an elective two-credit course entitled Thinking and Creativity, spring 2009 at Sultan Qaboos University. This course addresses the development of human intelligences and discusses the recent researches in this filed. It aims to improve students' creativity through helping them to identify their learning styles, types and through helping them to draw concept maps that leads to improve their cognition and meta-cognition abilities. Moreover, this course addresses some success techniques. This course is a blended course. The e-part of the course is delivered using moodle. Students have to have an enrolment key from the course tutor to access the course.

Course Home Page

Forums Page

Topic (4) page Purposes of the study This short paper aims to: 1. Specify the degree of student participation in discussion forums in terms of number of messages posted by each student. 2. Analyze the quality of messages in two topics of Thinking and Creativity blended course posted by students in terms of Self-introduction, Repeating information in another message, responding to accept or reject others’ views and opinions without explanation, responding to accept or reject others’ views and opinions with more explanation, context related, clarity, grammar and provocative. Questions of the study: 1. How many messages are posted in the whole discussion boards and by each individual student? 2. For which extent is the quality of messages posted by students in terms of Selfintroduction, Repeating information in another message, responding to accept or reject others’ views and opinions without explanation, responding to accept or reject others’ views and opinions with more explanation, context related, clarity, grammar and provocative?

Evaluation Methodology These researchers will utilize a quantitative method in this study to find out the total number of messages posted by student. Moreover, a qualitative methodology will be applied to evaluate the quality of messages posted by students. A checklist will be used to analyze messages' contents. Checklist consists of seven items that evaluate the message from different aspects. First, the checklist tends to examine if the student introduces him/herself at the beginning of the post. The second item examines repeating information in another message. The third and fourth items examine the student's response to accept or reject other's views with or without explanation. The rest examine the message clarity, context related and correct grammar. Moreover, the checklist examines massage proactive. Proactive means that the message prompts other student to respond. Results and Discussion Question 1 finding: Question 1 was: How many messages are posted in the whole discussion boards and by each individual student? For answering this question the discussion boards have been observed throughout the course. The average number of messages sent by students during studying the whole course includes:

1. The total number of messages posted to the discussion boards In this course, seven discussion topics were addressed by the course tutor. Results from observing the discussion board showed that the total responses of students' messages are 878 through out the course. Assuming that each student from those who enrolled in the course (240 students) should participate at least by posting one message to each topic, this number (878) represents only 52.3% of the expected total number of messages (1680) that should be sent to the discussion boards (Table 1). The total number of messages posted to the discussion boards (Table 1) Percent Number of Total number of messages Actual total number of assumed to be sent messages sent by students students 240

7*240= 1680

878

52.3%

2. The total number of messages posted to each topic, number of posts and number of replies to discussion board: The number of messages posted to each topic differs from topic to other topics. This difference may be as a result of how much students like the topic. In addition, the degree of students' motivation to reply and post differ from topic to another according to their interests. Table 2 shows the total number of messages posted to each topic. It is clear that Success Secrets topic has the highest number of posts (132) and the highest number of replies (317). However, topic No.6 that asks students to critique a video clip for Dr. ALFaqi has the lowest number of replies.

The total number of messages posted to each topic, number of posts and number of replies to discussion board (Table 2) Topic Number

Topic title

Number of posts Number of replies

1.

FGVWIJU ‫در‬STU - FMNOPK‫ ا‬FGHIJK‫ا‬

1

45

2.

Z[‫\را‬K‫ق ا‬WYJK‫ا‬

95

140

3.

`GHJK‫ح وا‬S_IK‫ار ا‬N[‫أ‬

132

317

4.

`GHJK‫ح وا‬S_IK‫ ا‬bTc

1

38

5.

‫ح‬S_IK‫ ا‬eMNf dOYK‫ا‬

13

19

6.

ZlYK‫ ا‬hG‫اه‬Nj‫ إ‬.‫ د‬WM\Gg

13

12

7.

‫ح‬S_IK‫دئ ا‬SPU ‫ل‬Wr ‫ت‬SPM‫\ر‬n : ٦ mM‫\ر‬n

1

51

256

622

Sub Total Total

878

3. The minimum and maximum number of messages per topic As discussion boards have been observed, we found that the minimum number of messages per topic posted is 25 messages, whereas the maximum number of messages per topic posted per topic is 449 messages as shown in table 3.

The minimum and maximum number of messages per topic (Table 3) Number of N Minimum Maximum Mean messages per topic 7 25 449 125.42

4. The number of messages posted per student Students' responses varied from ignoring all topics or some topics in the discussion boards to positive involvement. The minimum number of messages per student sent to the discussion boards was zero (Table 4). Table 5 shows that the majority of students posted between one and thirty messages. Around 112 students choose not to respond to any topic in discussion boards. The minimum and maximum number of messages per student (Table 4) Number of N Minimum Maximum Mean messages per 240 0 74 3.56 student The number of messages per student (Table 5) Number of Frequency Percent of messages per students student 0

112

47%

1

20

8%

2

27

11%

3

15

6%

4

9

4%

5

6

3%

6

9

4%

7

2

1%

8

8

3%

9

6

3%

Number of Frequency Percent of students messages per student 10

2

1%

11

4

2%

12

2

1%

13

3

1%

14

2

1%

15

2

1%

16

1

0%

17

2

1%

20

2

1%

22

1

0%

25

1

0%

29

2

1%

30

1

0%

74

1

0%

Question 2 finding: Question 2 was: For which extent is the quality of messages posted by students in two topics in terms of Self-introduction, Repeating information in another message, responding to accept or reject others’ views and opinions without explanation, responding to accept or reject others’ views and opinions with more explanation, context related, clarity, grammar and provocative? Topic1: Topic one asks students to post references of human resources development. This topic has 45 replies. Table 6 shows frequencies of messages after evaluating posts quality.

Table 6

Freque ncy Freque (Gramm ncy er) (Prov) yes 36 36 no 9 9 Tot 45 45 al

Frequenc y (Related)

Frequenc y (Clear) 45 22 23

Frequenc y (Int-Acex)

Frequenc y (Int-Acc)

Frequenc y (Int-Rep)

Frequenc y (Int-sel)

1 44

44 1

1 44

9 36

45

45

45

45

45

INT_SEL 40

30

Frequency

20

10

0 yes

no

INT_SEL

Chart (1) Number of messages that contain self introduction in topic No (1) Around nine messages contain self introduction before the real post. Some examples for the replies that have self-introduction are: ,-./01‫ ا‬3456‫ل ا‬89 /4:‫أ‬3< =>1‫@>? ا‬1‫ ا‬ABC












































‫ت‬SPGJwK‫ ا‬xyj ‫أت‬Nc zMNOPK‫ ا‬FGHIJK‫{ ا‬V |}~j S‫أ‬

INT_REP 50

40

30

Frequency

20

10

0 yes

no

INT_REP

Chart (2) Number of messages that contain repeated statements from another messages in topic No (1) The chart shows that one message only contains repeated statement. That is because students tend to post their responses to the original topic directly without reading or commenting on others posts.

INT_ACC 40

30

Frequency

20

10

0 yes

no

INT_ACC

Chart (3) number of messages that shows students replies to accept or reject others’ views and opinions without explanation in topic No (1) The chart shows that most students responded to accept or reject others' views and opinions without explanation. Example of message that accept others' view without explanation:

€VWWU ‫ر ع‬WwOU

INT_ACEX 50

40

30

Frequency

20

10

0 yes

no

INT_ACEX

Chart (4) number of messages that shows students replies to accept or reject others’ views and opinions with explanation in topic No (1) The chart shows that most students responded to accept or reject others' views and opinions without explanation. Example of message that accept others' view with explanation: hKSyK‫ ا‬FMS„ …GK dOYK‫ ان ا‬dyYKSj

RELATED 30

20

Frequency

10

0 yes

no

RELATED

Chart (5) Number of messages that are related to topic context in topic No (1) Around 22 messages are related to the context of the topic posted by the tutor. However, around 23 messages are not related to the topic context.

CLEAR 50

40

30

Frequency

20

10

0 yes

CLEAR

Chart (6) Number of clear messages in topic No (1) The chart shows that all messages posted by students are clear.

PROV 40

30

Frequency

20

10

0 yes

no

PROV

Chart (7) Number of messages that prompt students to respond in topic No (1) Around 36 messages are provocative, whereas the rest are not.

GRAMMER 40

30

Frequency

20

10

0 yes

no

GRAMMER

Chart (8) number of grammatically correct messages in topic No (1) The chart shows that around 36 messages are grammatically correct. That is may be because most messages are copied from different sources without any changes. Topic2: This topic asks students to post some stories about successful people. This topic has 38 replies. Table 7 shows frequencies of messages after evaluating posts quality.

Table (7)

Freque Frequenc Frequenc Frequenc Frequenc Frequenc ncy Freque Frequenc y y y y y (Gramm ncy y (Related) (Int-Acex) (Int-Acc) (Int-Rep) (Int-sel) er) (Prov) (Clear) yes 35 30 32 28 4 2 9 no 3 8 6 10 38 34 36 29 Tot 38 38 38 38 38 38 38 al

INT_SEL 40

30

Frequency

20

10

0 yes

no

INT_SEL

Chart (9) Number of messages that contain self introduction in topic No (4) Most of the replies does not have self introduction. Some examples for the replies that have self-introduction are:

/KL/M NBًP 3-Q‫ وآ‬T.‫ي وا‬NK: V-W‫ و‬/WN‫ ه‬/M/YM‫ أ‬Z.‫ي إذا و‬3]M ,4L‫ _^ و‬/M`W * ,W‫د‬/41‫ ا‬b>c@M ‫ أن‬/B-0L /d-lm ‫ا‬k1 ,:‫و‬/j>_ ‫ت‬/L‫ر‬NC ^@1‫ن و‬gKL/M /B-0L /dM‫ إ‬V1g< NP‫ أر‬/_ /‫زه‬tBd1 ‫ة‬gp1‫ط ا‬/pM‫ و‬/4KW/@d1 nBq1‫ط ا‬/pM *

F‹MWf ‫ة‬NJg IU mJwK‫\ ا‬r‫ أ‬Zg - Š‫ ا‬zHr‫ ر‬- ‫ان‬W„j ‰GOK‫ ا‬FTc ‫أت‬Nc*

bŽ {V FTc |yH[* =‫ه‬/Q>M‫ت ا‬N5 ,u< 3‫*اذآ‬

INT_REP 40

30

Frequency

20

10

0 yes

no

INT_REP

Chart (10) Number of messages that contain repeated statements from another messages in topic No (4)

There are about 2 repeated replies in this topic and this may be by mistakes from the students because they were done by the same students and at the same time with the post which is repeated from.

INT_ACC 40

30

Frequency

20

10

0 yes

no

INT_ACC

Chart (11) number of messages that shows students replies to accept or reject others’ views and opinions without explanation in topic No (4)

INT_ACEX 40

30

Frequency

20

10

0 no

INT_ACEX

Chart (12) number of messages that shows students replies to accept or reject others’ views and opinions with explanation in topic No (4)

There are about 4 students who comments for other replies by agree or disagree without explanation and no replies by agree or disagree with explanation. This may because of the nature of the topic which is written real stories of people success in their life. Because the stories are real there is no agree or disagree. Almost of the replies are different stories rather than comments for others. Some example for those replies is below:

...‫ أروع‬bTc‫آ“م را‘ و‬ ..W‹~K‫ع ا‬WWHK‫‹• ا‬V ‫ء‬Sg‫ا و‬Nw ...€nSI˜r ‫`ان‬GU Zg Š‫ ا‬z‹y—

RELATED 30

20

Frequency

10

0 yes

no

RELATED

Chart (13) Number of messages that are related to topic context in topic No (4)

Ten replies from (38) replies are unrelated to the main topic. One example of the unrelated replies is below:

y1/B1‫= ا‬W ‫ا‬N-Bx w-B: ‫إذاأردت أن‬ 3--z>1‫{ ا‬0m‫{ أ‬C y1/B1‫ آ{ ا‬3--z: ‫ول‬/K: 8W . bYjM =W

This reply is not related to the topic and it is not related to others replies. Most of the unrelated replies are taking about tips in how to success in life which is interfered with the main topic. So, it maybe the misconception of the topic is the reason.

CLEAR 40

30

Frequency

20

10

0 yes

no

CLEAR

Chart (14) Number of clear messages in topic No (4)

Most of the replies in this topic are clear accept about 5 of them. They are not clear because they are not related to the topic and not related to other replies which means that they are not meaningful.

PROV 40

30

Frequency

20

10

0 yes

no

PROV

Chart (15) Number of messages that prompt students to respond in topic No (4)

Most of the replies are provocative which means that they encourage others to reply and read them. This may be because of the nature of the topic. The most provocative replies are those which talk about stories about successful Omani people. This may be because they are more related to the participants' context and culture.

GRAMMER 40

30

Frequency

20

10

0 yes

no

GRAMMER Chart (8) number of grammatically correct messages in topic No (1)

Most of the replies in this topic are written in good grammar and using correct Arabic language.

Pros and Cons of using discussion boards in blendes courses Instructors in face-to-face universities should think carefully before integrating online learning for their students. One advantage of using such technology is that it allows asynchronous communication between classmates and course tutor. However, students who prefer face-to-to-face communication can not respond effectively in discussion boards. Moreover, instructors should specify the evaluation criteria for students at the beginning of the course to lead the discussion messages posted by students. In addition, course instructor should participate on the discussion boards and comments on students' posts which will influence and motivate students to respond.

References: 1. 2. 3. 4.

http://www.mirandanet.ac.uk/ejournal/uploads/48/Critique.pdf www.nnu.edu/fileadmin/Elearning/bb_tips/BbAssess_EvaluateDiscussion.pdf www.stjohns.edu/media/3/c84d5c3eb0f54c19baba0ba5e59af116.pdf http://www.surveyconsole.com/console/TakeSurvey?id=570745

Appendices Appendix I : checklist N Item 1. The student introduces him/herself at the beginning of the post 2. The student repeats information in another message. 3. The student Responds to accept or reject others’ views and opinions without explanation. 4. The student Responds to accept or reject others’ views and opinions with more explanation. 5. The student replies to the post in the same context. 6. the student reply is clear 7. The student reply stimulate others responses. 8. The student reply is grammatically correct.

Yes No