Ask

A$kAn$: An Online Marketplace for Q & A N. Sriram, Ph. D.

As we read, we often engage in an implicit conversation with the absent author, recognizing novel facts, delighting over fresh approaches, asking questions, and becoming aware of the gaps in our understanding. A similar internal dialogue unfolds when we are absorbed by a compelling presentation. Metacognition or “thinking about thinking” is a term that encapsulates the diverse components of this mental activity. Research indicates that metacognition is a crucial catalyst in learning. Not only do competent individuals have good domain knowledge and skills; they are also acutely aware of their knowledge gaps and skill deficiencies. Paradoxically, less competent individuals, suffering from metacognitive blindness, have overly optimistic and inaccurate beliefs about their skills and knowledge1. Educators, going back to Socrates2, have advocated questioning the learner as a superior pedagogical strategy, especially when compared to the alternative of feeding the learner with a litany of unmotivated facts. It is reasonable to assume that considerable metacognitive activity occurs, particularly among motivated learners. However, due to a variety of reasons, this metacognitive content remains implicit, personal and is soon forgotten. How often have we heard a speaker request “Any questions?” at the end of a presentation? And the typical response? Deafening silence! Few can summon the chutzpah required to express imperfect intuitions in public. On occasion interesting questions are articulated but, more often than not, are drowned out by the torrents of consciousness gushing from the mouths of vocal incompetents. Student assessment exclusively operates from the vantage point of students answering questions posed by teachers. Nonetheless, it can be extremely gratifying for the teacher to have the opportunity to address challenging questions posed by students. Sometimes novices pose novel questions of fundamental importance. Even if questions are not of fundamental importance, they inject a new dimension to the discourse surrounding the domain concept and can deepen the teacher’s appreciation of the subject matter. And if questions arise from misunderstandings then the learner can be quickly disabused of the misconceptions that were blocking learning. Despite the apparent importance of student questioning, few systems allow students to have repeated opportunities to ask questions and few students take advantage of these opportunities when they are presented. This article describes my experience with A$kAn$3, a system that rewards students for asking and answering questions. The system was conceptualized in late July 2002 and, following the initial implementation, refined with the participation of a handful of volunteers during August 2002. In early September, the system was opened to general use by 230 students in an introductory psychology course that had the usual complement of weekly lectures, group discussions, midterm

quizzes and final exams. Students used the system for about two months and participation in A$kAn$ accounted for eight per cent of the final grade. Question asking systems in the domain of knowledge management first made their appearance on the internet in late 1999 and early 20004 and continue to proliferate as the internet evolves and consolidates5. These systems encouraged experts to answer questions by rewarding them with real money or virtual ratings that increased their prestige in online communities. Novices are not rewarded for asking questions—indeed, on some systems, they pay money to obtain an expert answer. Once the question is answered, the expert answer is evaluated by the novice and the interaction is incorporated into a searchable database. The concept of rewarding experts for their answers is quintessentially Skinnerian6 --- behaviour that is reinforced by money and/or ratings is strengthened, leading to a positive feedback loop. While such models are suitable for tapping the expertise of individuals on the internet or the intranet of far flung corporations, they cannot be applied wholesale to the context of tertiary learning. For one thing, while students make generalized evaluations of instructors at the end of each semester (this process has been critically evaluated7), they cannot consistently and accurately rate answers to questions in a domain in which they are novices. In fact, such ratings would carry very little credibility with their peers. A$kAn$ makes a radical departure in that both questions and answers are rewarded. Furthermore, this valuation is solely done by the teacher. Students play the dual role of novices and experts, with the teacher as evaluator and critical commentator. Students registered with the system using an existing email address and occupied a slot in one of 10 virtual cities. Each city was represented as a 5 x 5 matrix with 25 cells. Each student chose a nickname and an icon to be associated with the student questions and answers. Students could change their appearance, nickname or location; most of them quickly settled in one of 10 cities, adopting a personal nickname and an icon.

Students ask questions by clicking A$k in their highlighted cell. The question is placed in one of 12 chapters of the introductory psychology text. The teacher, using a special interface, values the question and assigns a maximum dollar value for the answers to the question. The valued question is accessible to other students who can now answer it. Each answer is invisible to other students till it has been valued by the teacher. The author of a question or an answer can delete it at any time prior to the valuation.

I valued questions and answers when I was online, typically during the evenings, and on average, spent about 30 minutes each day. Sometimes I would comment on the question or answer. Very occasionally I asked questions; these can be seen by clicking on my cell in Bangkok. The teacher can close a question, disabling further answers to the question. I closed questions whenever the marginal value being added by answers became insignificant. The teacher can also make a question “private” so that students cannot see others answers to the same question until such time the question is made “public”. This feature was used only very occasionally. The black $ value on each cell represents the total value earned by the student and the sum of these values represents the contribution of each city. The white $ value represents the earnable potential of each cell—this is the sum of the max dollar values for answers to the questions posed by the student. Students could interact with each other in private and public discussion forums. “Pals” is a custom matrix consisting of the student’s friends. Menu items at the top allowed navigation of questions that had not been answered (NewQ), had answers with high value (HiQ) and by chapter (SeeQA). A keyword search by questions, answers and nicknames completed the functionality. The system can display tens of thousands of items across multiple pages in all views. The rewards for questions and answers can be adjusted by the teacher. I observed that some students were asking too many questions of the “hit-and-run” quality; this led to my valuing questions more critically while simultaneously increasing the rewards for answers. Over a period of two months, 2098 questions and 3369 answers were constructed and valued on the system. 1629 questions received at least one answer and 469 remained unanswered. Questions, on average, were valued at $1.67 and answers at $2.26. The final market capitalization was $11,084. The number of questions and answers are an order of magnitude higher than the quantum elicited by general-purpose discussion forums. A$kAn$ is database driven and opens up the possibility of many kinds of reuse including course redesign. A$kAn$ is being generalized to multiple courses and multiple teachers within each course who are assigned to one or more course content domains. Team based activity, chat and other features are also in development. The goal is to make the system engaging, effective and scalable; both from the technical8 as well as the cognitive perspective. A few outstanding students, who consistently produced high quality questions and answers, were given special recognition during the last lecture session of the term. It is interesting to note that while these students did well on traditional evaluations, there were several students who excelled on exams but the quality of their questions and answers were not outstanding. This suggests that the kind of learning and accomplishment exhibited on the A$kAn$ is distinctive from that measured by traditional evaluations. Student feedback was encouraging, detailed and useful as can be seen from a couple of the more than fifty responses at the end of the semester. Students frequently reported being amazed by the depth of understanding in some of their peers. The experience of the past semester indicates that the token economy approach is well suited to awakening metacognitive awareness during learning.

References 1.

Kruger, J. & Dunning, D. (1999). Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments, Journal of Personality and Social Psychology, 77, 1121-1134. http://www.apa.org/journals/psp/psp7761121.html

2.

Garrett, E. (1998). The Socratic Method, http://www.law.uchicago.edu/prospective/headnotes/socratic.html

3.

http://www.comunaware.com/matrix/index.htm Use the Nickname=cita and Password=nus123 to Browse.

4.

http://www.askmecorp.com/ is a Vendor of Question-Asking systems for Knowledge Management.

5.

http://answers.google.com/ is Google’s Answering Service.

6.

http://www.bfskinner.org/Operant.asp is an Overview of the Skinnerian Perspective on Learning.

7.

http://faculty.washington.edu/agg/bytopic.htm#sri Contains Papers on the Validity of Student Ratings.

8.

A$kAn$ is hosted on a J2EE application server running on Linux.