Research Proposal

Research Proposal Online Student Satisfaction Susan Wilson University of British Columbia

Introduction In his 2006 research article, Herbert stated that the rate of discontinuance in online, post-secondary courses is ten to twenty percent higher than that of traditional classroom-based courses. As online education moves into secondary education institutions, the problem of retention will compound. Post-secondary institutions work with students who have already successfully completed one tier of education; students who are motivated to learn and have a financial interest in successful course completion. Public secondary schools educate students of varied abilities, attitudes, and backgrounds. I believe that the rate of discontinuance in online, secondary courses is even greater than that of online, post-secondary courses.

According to the 2008 Saskatchewan Education Indicators Report, Saskatchewan’s population of 1 015 985 on July 1, 2008 was made up of approximately 40% of people aged 45 and over and only 27% of people aged 20 or younger. In addition to the relatively small population, the report states that the number of people living in rural communities has declined at a higher rate (11% decrease from 1971 to 2001) than the Canadian average (4% decrease from 1971 to 2001). Saskatchewan’s large geographical area and dispersed population make it necessary to operate small schools, some of which have fewer than 50 students.

Online learning is a practical solution to challenges presented by rural depopulation and a large geographical area. It is also a viable learning environment for adult students, students who are required to work to support themselves financially, and

students who have significant family responsibilities. Saskatchewan’s 2005 rate of teen pregnancy was the highest among the provinces and more than double that of the Canadian rate (Saskatchewan Indicators, 2008).

The 2008 Saskatchewan Indicators report suggests that approximately 20% of youth do not graduate. Retention of secondary courses is an important consideration in Saskatchewan and everywhere. Having a Grade 12 certificate increases the chances that a person will be employed (Saskatchewan Indicators, 2008) and increases the probability that a student will continue their education. Having a post-secondary certificate, diploma or degree increases the average income level to an average of 33% (statistic from 2005) higher than that earned by those without the post-secondary training (Saskatchewan Indicators).

As an online educator in rural Saskatchewan, I am motivated to learn how to best implement online education opportunities into our high-schools. Rural depopulation and economic factors translate to teacher cuts in our school division. If we are to remain viable, we must offer and accept online education into our school. What factors must we be cognizant of when designing online courses? How can we best support our students before and during their involvement in online education?

Online education from K-Post Secondary reaches millions of students every day (Corry, 2008). If online courses are being offered to students in situations like ours, then students are not making a voluntary choice to take a class online, they are required to take

online education to complete graduation requirements. At times, this is because small rural schools do not have the staing to offer all required courses, they do not have the staing to offer various electives, or they do not have the staff to offer multiple forms of a class to accommodate students who have missed credits or changed schools. If courses are not otherwise be available, we, as educators, must ensure that students are satisfied with their online experience and that they are receiving quality education.

Education is for everyone and online education should not be any different. It is imperative that reliable instruments are developed to measure student success and provide quality assurance to all stakeholders of public education. Educators who have access to an instrument that can predict the probability of student success will be able to design pro-active interventions and supports to facilitate student success thus widening the doors to online education.

With this in mind, this research proposal is designed to answer the following questions: 1. What factors do students perceive to aid in their satisfaction and success in online courses at the secondary level? 2. What factors do online educators perceive to aid in student satisfaction and success in online courses at the secondary level? In answering these questions, I hope to collect data that will aid in the development of a predictive instrument for use with Saskatchewan high-school students entering into online education. The data will also be used to aid in the development of a student

support system that will be used to target areas of student weakness in an effort to increase their probability of success in secondary, online courses. The data will not be used to limit or restrict enrolment in online, secondary courses.

Quantitative and qualitative data will be collected by using student and instructor survey responses. I hypothesize that student demographics (age, gender, location of residence) will not be identified as factors correlating to student success. I believe that institutional factors such as instructional design, hardware and human infrastructure, communication, time, and course organisation will be identified by both students and instructors as being significantly correlated to student satisfaction and success. I expect that personal student factors such as ability, experience and motivation will have an effect on satisfaction and success, but that they will not be as significant as institutional factors. While circumstantial variables may present themselves and influence both satisfaction and success, they are not controllable by student, instructor or institution and will therefore not be considered in this study.

Student satisfaction will refer to the degree to which students enjoy the educational experience and are engaged and interested in the course. Student success refers to a student’s passing the course and retention refers to finishing the course.

Literature Review Studies relating to factors influencing student success and satisfaction in online education are becoming more prevalent as online education undergoes rapid and continuing growth (Dailey-Hebert et al, 2005; Herbert, 2006; Leong et al, 2006; Roblyer & Davis, 2008; Yukselturk & Bulut, 2007). Expressing a concern with growing rates of discontinuance, researchers (Dailey-Hebert et al, 2005; Herbert, 2006; Leong et al, 2006; Roblyer & Davis, 2008) have attempted to identify factors relating to student success and student satisfaction in online courses. Articles collected for this review study factors affecting online student satisfaction (by perception) and online student success (passing the course), with emphasis on retention (completing the course). Research surrounding factors affecting student satisfaction in online education is extremely important as students who are satisfied with their online experience are more likely to retain the course (Herbert, 2006; Swan et al, 2006). Assessing student satisfaction can be valuable in terms of program and course improvement (Herbert). The literature review is organized according to guidelines by Gay, Mills & Airasian (2009) using the headings Post-Secondary Student Satisfaction, Post-Secondary Student Success and High-School Student Success. Post-Secondary Student Satisfaction 1. Staying the course: A study in online students’ satisfaction and retention.

Herbert’s (2006) cross-sectional study was designed to identify institutional variables affecting retention. He used the Noel-Levitz Priorities Survey for Online Learners™, which was e-mailed to every student enrolled in an online course at a small, Midwestern university during the 2005/06 school year. Herbert cited evidence that student variables, among other factors, can be measured to predict the degree to which a student will complete an online course. He also referenced research findings that institutional and demographic variables are predictive as well.

Analysis of survey results was done by the Noel Levitz company. Of the 4100 students, 122 completed surveys were returned representing 40.1% of the students who did not successfully complete their online course. When asked for the reason for discontinuance, 61.3% cited time constraints and 16.1%, personal problems.

The most significant variable identified was faculty response to student needs followed by the quality of online instruction, timely feedback, quick institutional response to requests, and frequency of student-instructor interaction. Student-student interactions scored lowest. There was no statistical difference based on institutional variables between course completers and non-completers. However, results of an independent samples t-test showed a statistically significant difference between satisfaction levels of completers and non-completers. Students who were satisfied with their online experience were more likely to retain the course.

2. An empirical investigation of student satisfaction with web-based courses.

This quantitative study surveyed 128 students enrolled in 29 University of Hawaii, online courses to examine the relationship between demographic variables, online course experience and satisfaction. The researchers’ rationale was to identify factors to increase student satisfaction.

Based on previous research, Leong et al (2006) produced a 47-item survey to address instruction, instructor characteristics, management, technology, interaction, experience, workload and fairness of grading. Two additional questions based on overall course satisfaction and comparisons to face-to-face instruction were asked. A response rate of 25.2% (128 surveys) was achieved and results were calculated using factor and regression analysis. Also, t-test and univariate analysis of variance were used. Results showed that overall student satisfaction was influenced by four dimensions: instructor, system-wide technology, workload/difficulty and interaction but not by demographic factors or students’ prior experience.

Post-Secondary Student Success

3. Learner attribute research juxtaposed with online instructor experience: Predictors of success in the accelerated online classroom.

Dailey-Hebert et al (2005) compared characteristics of learner attributes with online instructor experience. The rationale was to develop practice-orientated understanding of

factors predicting student success. Literature exists on online student retention, their attributive influences and their reasons for choosing online delivery but Dailey-Hebert et al posited a need for more information about predictors of success as research based on internal learner attributes is of limited use unless paired with external data elicited from online educators.

The study surveyed a self-selected sample of 96 online educators with an average teaching experience level of 3.5 years. All educators offered accelerated, 8-week courses and no demographics were collected on the sample however future studies may compare educator characteristics with factors influencing success. An e-mail solicited their opinion on the five factors most likely to predict successful completion of an online course and a survey identified skills, strategies or factors they perceived to lead to student success. A content analysis of the responses identified 23 relevant factors which were then grouped into six themes, student competence, student initiative, student personal issues, time, technology, and instructional factors.

The four most predictive issues were time [timely, active involvement in the course (67.71%), effective time-management (67.71%) and timely access to instructional resources (19.79%)], initiative [personal initiative (52.08%), asking questions and seeking help

(37.50%), self-motivation (22.92%) and a positive attitude (4.17%)], technology [access to efficient computer and internet literacy (40.63)], and competence [reading comprehension (23.96%), writing skills (22.96%), communication (17.71%), awareness of expectations, environment and workload (16.67%), and organizational skills (13.54%)].

4. Predictors for Student Success in an Online Course

Yukselturk and Bulut (2007) designed a correlational study to analyse student characteristic variables (gender, age, education level, locus of control, learning style), motivational beliefs (intrinsic and extrinsic goal orientation, task value, self-efficacy, and test anxiety) and self-regulated learning components (cognitive strategy use and selfregulation) that predict online student success. They also examined instructor views on predicative factors. They hypothesized that the variables do not explain variance in student success.

The researchers posited a need for the inclusion of instructor perception. Their rationale was to fill a void in the research and to identify procedures for the design of high-quality online learning environments. In a well-written, extensive literature review, Yukselturk & Bulut provided history on online education and stated a need for maintaining quality. The authors defined terms used in the study.

The sample of two course instructors and 80 voluntary participants enrolled in the Data Structure and Algorithms with C course at the Middle East Technical University in Ankara, Turkey in the 2005/06 school year. All students were computer literate with intermediate English. There were more males (N = 56) than females (N = 24) and the majority were between 19 and 29 years of age. In addition, most had an assimilator learning style and a university degree.

Quantitative and qualitative data was collected through four online questionnaires: Demographic Survey, Internal-External Locus of Control Scale, Learning Style Inventory, and Motivated Strategies for Learning Questionnaire. Semi-structured interviews were conducted; the ten questions were developed around central themes then examined by two experts in the field of instructional technology.

Intrinsic goal orientation, task value, self-efficacy, cognitive-strategy use, and selfregulation were significantly positively correlated with success. Educational level and external locus of control were negatively correlated. Varied student characteristics and general personal characteristics, did not directly affect success.

5. Online student success: Making a difference

Beyrer’s (2005) study compared online students enrolled in Online Student Success (OSS), a course designed to prepare students for online learning, and online students who have never taken this class. He also compared online performance of students before and after completion of OSS. Beyrer hypothesized that students who take OSS will be more successful than those who do not and they will be more successful after completion of OSS than they were previous.

Beyrer conducted a well-written, comprehensive literature review stating that most research focused on instructional design factors affecting student success, tips to aid

student success, and technology and interface characteristics. Beyrer defined the terms used in the study.

The convenience sample included all students enrolled in fully online classes from Cosumnes River College, Sacramento CA from 2003 to 2005. They were divided into 4 subgroups: those who had never enrolled, those who enrolled and passed, those who enrolled and did not pass and those who enrolled but dropped. Historical data (academic performance, enrollment, and demographic) was collected and analyzed and an online survey of students was conducted. Twenty-one respondents completed the 18-question (open and closed answers) follow-up survey.

Compared to students who did not take OSS, the ones that successfully completed OSS had a higher success rate in their online classes. Those who were unsuccessful or dropped OSS had a lower success rate. Though only 11 students took online classes before and after enrolling in OSS, the improvement in their success rate was dramatic. Follow-up survey responses from all three OSS groups were positive.

High-School Student Success

6. Predicting success for virtual school students: Putting researchbased models into practice.

Roblyver and Davis (2008) posited that prediction models should be developed support school-age student success in online courses. They explained that the model should be based on the combined factors that contribute to predicting success as identified by research and should provide efficient measurement and implementation in virtual school settings. The article contains an example of a prediction model, the Educational Success Prediction Instrument (ESPRI) and a description of data collection and statistical processes used to derive it. They outlined procedures for implementation to increase the accuracy and utility of predictions.

Model testing was conducted using 4,110 students in the Virtual High School Global Consortium (VHS) who were enrolled in 196 VHS courses in 2006. An electronic version of the instrument was placed in course spaces and students were offered 10 points extra credit on their first week's assignments to complete the survey. A completed ESPRI survey, demographic data, and course scores and status were obtained for 2,162 students or about 53% of the total school population. Reliability with the 25-item instrument was .92. The model correctly predicted 93% of those who were successful, but only 30.4% of those who failed.

Common Themes

In general, research methods have consisted of survey instruments distributed online to students enrolled in online courses at the post-secondary level with few conducted at the high-school level. All studies in this literature review are cross-sectional in nature

indicating a need for longitudinal research which can identify trends from which appropriate revisions can be made (Herbert, 2006).

Herbert’s literature review found demographic factors to be predictive of satisfaction and retention but the studies compiled by the other researchers showed conflicting conclusions or no evidence that demographics can predict success. Dailey-Hebert et al and Yukselturk and Bulut identified a prevalence of reseach based on the effects of student characteristics. Both research studies widened their examination of predictive factors to include the perception of experienced online instructors. Linking student responses with faculty perceptions can be powerful in creating a practical understanding of the influences on student satisfaction and success.

Faculty response to student needs and student-instructor interaction were significant factors identified in the research studies. Time constraints and management issues were also significant predictors of satisfaction and success. Yukselturk and Bulut were the only researchers to focus on student characteristics such as motivation and selfregulation.

Summary

All researchers indicated a desire to inform educators of desirable factors to influence student satisfaction and student success in an online environment. Most stated a desire to counter-act course discontinuance at the post-secondary level. Studies of student and

institutional factors produce varied and somewhat contradicting results. Longitudinal research ventures may aid in identifying trends in student success that will better inform improvements. Predictive instruments of student success will be helpful if they can be refined to identify student supports needed and not used to limit access. This idea that research findings provide actionable information is supported by Beyrer (2005) and Roblyer (2008). Both projects involved the development, testing or implementation of an instrument, one predicative and one instructive, used to determine student suitability to online education and support students in need. Beyrer’s work focused on the benefit of enrolling students in an Online Student Success (OSS) course designed to improve probability of student success in online courses. Combining this idea with Roblyer’s focus on predicative models for online student success would allow institutions to offer pre-assessment and targeted support to potential students. It is important to note that the purpose of these instruments is to increase enrolment and success in online courses, not to rank, stream or prohibit students. Methodology Elaine Strachota (2008) developed a 27-item survey instrument, The Student Satisfaction Survey, based on student satisfaction with respect to online interaction (learner-content, learner-instructor, learner-learner, and learner-technology), and general satisfaction. Strachota defined each construct. Adult and distance education experts edited the survey questions to establish content-validity. Construct validity was established by conducting a pilot study of 249 online students at a Midwest Technical College in the US. Data from the pilot underwent factor analysis to verify that items loaded on the intended constructs. Those that did not load were eliminated.

This research project will use survey question ideas from Strachota’s Student satisfaction Survey as well as student readiness survey question ideas from Slick (2004) to develop an instrument suited to secondary students. Content validity will be established through review by the Instructional Technology Department Consultant and Coordinators of the SouthEast Cornerstone School Division, SK. This will take place at our next scheduled meeting on April 29, 2009. Construct validity will be established through a trial with online students in our division which will take place in June, 2009. Statistical analyses on data received will inform decisions about question retention or removal. Each survey item will require a Likert-response consisting of a 4-point scale: strongly agree, agree, disagree, strongly disagree. Once finalized, the survey will be distributed to a convenience sample of students enrolled in online education courses through the SouthEast Cornerstone School Division through their course management system. Course instructors will notify students of the survey by e-mail and will send up to three reminders over the two week implementation period. Completion of the survey will be expected as part of the assigned course work and will be given to students in the final two weeks of their course. Parental permission will not be required as the research will be done through the students’ attending school division for the purpose of improving instruction.

Data for this longitudinal study will be collected and analysed in January and June of each year for two complete years. Once complete, a trends analysis will be performed to determine factors affecting student satisfaction and success. Results will be quantified

by converting frequency of responses to percentages. Results from open-ended survey questions will be compiled and categorised. Data from respondents indicating course satisfaction and feelings of success will be evaluated for factor correlation. Data from respondents who report dissatisfaction and lack of success will be evaluated for factor correlation.

This data, analysed and interpreted over a two year period, will be used to aid in the development of a predicative instrument for student success in online environments. It will also be used to develop a student support system that targets factors influencing student success.

Significance Even though online education reaches millions of students every day (Corry, 2008), the field of online learning is still in its developmental stage with respect to having valid and reliable measurement instruments (Strachots, 2008). Creating a reliable survey instrument, developed for and piloted on secondary school students will add to the field of distance education at the secondary level. Further work with survey results to develop a predictive instrument and student support component will directly benefit online students.

Student feedback is an important component in the provision of quality education when data received is used to improve student instruction, satisfaction and chance of success. Many secondary teachers are new to online delivery. Survey results will be an

important component of professional development for online teachers. Data obtained will help inform department or even provincial policies and standards created to ensure quality in online course delivery. Data obtained from online instructors will provide good fodder for conversations around instructional design, pedagogy and best practices.

Continuing the research over the course of two years will provide more reliable and normally distributed data allowing for trend-analysis of student satisfaction, success, and retention at the secondary level. This information will provide a significant contribution to the field of distance education at the secondary level.

Conclusion Canadian school divisions are developing internal capacity for online course design, delivery and implementation. Current research around student satisfaction and success concentrates on post-secondary students. Research needs to be done on factors affecting the satisfaction and success of online, secondary students.

As evidenced in the research, surveys are effective tools for gathering both quantitative and qualitative data from students and instructors. Embedding the survey into the course management system ensures that all involved are presented with the opportunity to respond. Managing the survey distribution and data collection through such technology reduces errors and strengthens the reliability of results. Field testing and content validation will also increase reliability.

Research results will be used to develop a more refined survey instrument that will predict student success in an online learning environment. The instrument will identify individual student strengths and weaknesses for the purpose of providing student support. Student satisfaction and success factors will also be considered in the development of a student support instrument that will enable students to improve their weaknesses thus improving their chances of retention and success in online courses.

References

Beyrer, Gregory M. D. (2005) Online student success: Making a difference. Cosumnes River College. @ONE Carnegie Scholar. 2005-06. Retrieved from http://www.cccone.org/scholars/05-06/GregBeyrerMonograph.pdf

Corry, M. (2008). Online student success: Six categories of importance to the future. In K. McFerrin et al. (Eds.), Proceedings of Society for Information Technology and Teacher Education International Conference 2008 (pp. 312-317). Chesapeake, VA: AACE. Retrieved from www.editlib.org/index.cfm/files/paper_27178.pdf?fuseaction=Reader.DownloadF ullText&paper_id=27178

Dailey-Hebert, Amber, Donnelli, Emily, and Mandernach, B. Jean. (2006). Learner attribute research juxtaposed with online instructor experience: Predictors of success in the accelerated online classroom. The Journal of Educators Online, Volume 3, Number 2, July 2006. Retrieved from

http://www.thejeo.com/Volume3Number2/MandernachFinal.pdf

Gay, L.R., Mills, G.E., & Airasian, P.W. (2009). Educational research: Competencies for analysis and application (9th ed.). Upper Saddle River, NJ: Merrill Prentice Hall.

Herbert, Michael PhD. (2006). Staying the course: A study in online students satisfaction and retention. Online Journal of Distance Learning Administration, Volume IX, Number IV, Winter 2006 University of West Georgia, Distance Education Center. Retrieved from http://www.westga.edu/~distance/ojdla/winter94/herbert94.htm

Leong, Peter, Ho, Curtis P., Saromines-Ganne, Barbara. (Winter, 2006). An empirical investigation of student satisfaction with web-based courses. Online Journal of Distance Learning Administration, Volume IX, Number IV, Winter 2006. University of West Georgia, Distance Education Center. Retrieved from www.editlib.org/index.cfm/files/paper_9442.pdf?fuseaction=Reader.DownloadFul lText&paper_id=9442

Roblyer, M.D., Davis, Lloyd. (2008). Predicting success for virtual school students: Putting research-based models into practice. Online Journal of Distance Learning Administration. Winter 2008. Volume 11. Issue 4. University of West Georgia. Retrieved from http://www.westga.edu/~distance/ojdla/winter114/roblyer114.html

Saskatchewan Education Indicators: Pre-Kindergarten to Grade 12. 2008. Available from http://www.education.gov.sk.ca/adx/aspx/adxGetMedia.aspx?DocID=609,135,107 ,81,1,Documents&MediaID=6109&Filename=Web+Version+2008.pdf

Slick, J. (2004). Using Student Readiness as a Predictor of Satisfaction in the Online Environment. In G. Richards (Ed.), Proceedings of World Conference on ELearning in Corporate, Government, Healthcare, and Higher Education 2004 (pp. 2130-2136). Chesapeake, VA: AACE. Retrieved from www.editlib.org/index.cfm/files/paper_11492.pdf?fuseaction=Reader.DownloadF ullText&paper_id=11492

Strachota, Elaine. (2008). The use of survey research to measure student satisfaction in online courses. Online Journal of Distance Learning Administration, Volume XI, Number IV, Winter, 2008. University of West Georgia, Distance Education Center. Retrieved from http://www.umsl.edu/divisions/conted/education/mwr2p06/pdfs/D/Strachota_Use _of_Survey_Research.pdf

Swan, Karen; Schenker, Jason; Lin, YiMei; Shea, Peter;; Aviv, Reuven (2006). Student satisfaction with online learning: A concept analysis. Proc. Annual Meeting of the American, Educational Research Association, San Francisco, April 2006. Retrieved April 13 from http://tinyurl.com/d2twn2

Yukselturk, E. & Bulut, S. (2007). Predictors for Student Success in an Online Course. Educational Technology & Society, 10 (2), 71-83. 71 ISSN 1436-4522 (online)

and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). Retrieved from http://www.ifets.info/journals/10_2/7.pdf