Etec 565a Assignment One Kelly Nichols

SEAtS Software Evaluation Report

Learning Analytics Tool

2

RATIONALES Learning Analytics Tool- SEAtS Software SEAtS Software promotes itself as a student success platform. It appears to combine both a Student Information System (SIS) and Learning Management System (LMS) into one. Slade & Prinsloo (2013) identify the importance of student as agents who can “voluntarily collaborate in providing data and access to data to allow learning analytics to serve their learning and development” (p. 1529). Consequently, I actively searched for a Learning Analytics (LA) tool that would provide this affordance. Another aspect that lead me to select SEAtS software to evaluate was that the company highlighted how multiple stakeholders (student, faculty, and administration) could use and benefit from this software and also boasted of several reputable customers across Canada, America, and the United Kingdom, including Athabasca University. There was also a lack of online reviews from current users about this software. Consequently, for these reasons, I decided to provide an evaluation of the learning analytics of SEAtS Software.

The Evaluation Framework My learning analytics evaluation framework is modelled off both the frameworks from Cooper (2012) and Scheffel et al. (2014). This framework was developed for K-12 educators and administrators to support their search and evaluation for a learning analytics tool. The Cooper framework chose to step away from “soft factors”, elements with subjectivity like cultural context which featured previously in the original framework by Greller & Drachsler. Their rationalization was to focus on hard factors first and to debate on the emerging soft factors later. Scheffel et al. (2014), categorized outlier statements into a cluster called Acceptance and Uptake because of their inability to fit clearly with any other cluster. This led to an ad hoc criterion called organizational aspects which would feature some of these soft skills. My belief was that the soft skills should not be delayed or ignored but instead must be intentionally woven within the framework because although they do not necessarily contribute to the selection and analysis of a learning analytics tool, they do need to occur in order for the successful initiation of the search. Consequently, this resulted in the creation of a check list to be used before commencing in the evaluation of a learning analytics tool using the reworked framework.

3

Before commencing in the evaluation of learning analytics tools, a high school educational institution should first identify what LA software’s are available and their cost to see if the school budget could afford to implement them should they be selected. Next the infrastructure of the school would need to be analyzed, specifically the school network. Most schools operate on Local Area Networks (LAN) and connectivity should first be inspected. Next, there would need to be a check of what devices were available that the selected learning analytics software or browser-based program would be downloaded or accessed on. Access and compatibility for all intended users would need to be assessed. Before the searching and implementation of a LA software, acceptance from a variety of stakeholders should be sought out. Scanning the educational culture to ensure that a growth mindset is present would afford mass organizational change. Moreover, stakeholders would need to be convinced of the benefits of LA in order to be motivated to use it once implemented. Lastly, a check that LA could be implemented into the high school would be necessary. Stakeholders would need to be provided both time and training on how to use the new LA tool. To achieve this transition, both a team or individuals to provide IT support for the network, software, and devices issues as well as a team or individuals to provide LA support on how to use the new software would be needed. Once the checklist is complete, the school would be ready to commence their search of a LA tool. Once the search for an LA tools has begun, a framework is then needed to help evaluate and differentiate the available LA tools. Below is the proposed framework.

4

This framework is a condensed version of the Cooper framework which provides the context for evaluation. The key difference is that this framework includes the Scheffel et al. (2014) quality indicators of both importance and feasibility. It is proposed that after Analysis and Data Origin of an LA tool has been evaluated, that it also receives a rating of the three identified quality indicators on a scale of one to three, one being weak and three being strong. •

Transparency- refers to both what information is provided to all stakeholders and how

accessible that information is (i.e. presentation).

5 •

Data Ownership- refers to who has access and control to how data is created,

recorded, collected and monitored. It would also look at any potential data outsourcing or transfers to third parties. •

Privacy and Ethics- refers to what kinds of data is collected and where it is stored, for

example, Canadian versus American data centers. Also, if it follows the Freedom of Information and Protection of Privacy Act (FIPPA). For example, what kind of deidentification tools are available or do stakeholders have the ability to provide consent or opt out. After Orientation & Objectives and the Technical Approach of an LA tool have been evaluated the following would be rated on the same three-point scale: Actionable Insights- The computer methods used to determine at risk students and both the

computer and human generated recommendations that result from these detections. Student Agency- refers to the motivation of students to use the LA tool for their own regulation

and accountability. Transformative- Refers to how useful the LA tool would be to stakeholders. Is it comparable

to other LA tools or does it provide new information and opportunities? The question should be asked, how effective, efficient and helpful will this potential LA tool be for all stakeholders?

6

Evaluation: Student Retention

Analysis Subjects: Students Analysis Clients: Academic & Administrative Staff and Students Analysis

(Students are loosely classified as Analysis Clients as they can be a part of the alert system and consequently could adjust their own learning without further external intervention.)

Analysis Objects: Students

Private Data: SEAtS uses reactive and non-reactive data, and integrates and processes them to identify and isolate risk indicators that drive current retention, progression and graduation metrics. SEAtS collects data into a single repository. This supports teachers in data literacy to better collect and interpret the data (Ifenthaler, 2016). Data Origin Data Quality & Source- SEAtS does not collect new data, it only “sits above and draws information” from whatever student records, timetabling and other systems are already in place. Quality will depend on the size of the educational setting, for most high schools this would likely be small to modest. Educational settings follow FIPPA policies on personal data use and sharing.

Ratings:

Transparency

1

2

3

Data Ownership

1

2

3

Privacy & Ethics

1

2

3

Orientation

7 Past: [Diagnostic] to look at student attendance, engagement and performance attrition. Present: creates real time Early Warning Systems (EWS) alerts Orientation & Objectives

via e-mail and SMS to both academic and administrative staff on populations of students who may be ‘at risk’ such as those displaying disengagement with the course. EWS alerts can also be expanded to include the ‘at risk’ student as well. Future- [Extrapolation] data manipulation can be presented to students about future potential outcomes. For example, showing students both their current data and making minor changes to attendance or grades can demonstrate and visualize potential future results. Objective Type: Performance management (to increase retention, progression, attainment, and graduation metrics).

Traditional Descriptive model that uses rules-based analytics to generate data sets. Data is harvested from existing SISs, VLEs, and LMSs. Tolerance levels can be set by individual institutions for different educational contexts and breaches of tolerances can create and assign calls to actions to both staff and student. Technical Approach

Machine Supervised Machine Learning- once enough historic data is collected, models can be trained to create refined predictive profiles of potential ‘at risk’ students. Simulation- Although not entirely clear, SEAtS sites a benefit of their software to be able to model risk factors to build a Student Retention and Completion model that reflects the realities of within a given educational institutions.

8

Presentation- A campus wide and customizable dashboard that can unify both physical and digital data to create personalized student profiles of engagement, retention and achievement. Data is presented with visual aids like static and live charts.

Ratings

Actionable Insights

1

2

3

Student Agency

1

2

3

Transformative

1

2

3

SEAts states that “Research has shown that engagement and Embedded Theories and Reality

attendance monitoring identifies[sic] students most in need of support and enables university staff to intervene to improve the chances of a student’s successful course completion”. (SEAtS Software, 2019). However, no direct evidence of any theory has been provided and no supporting literature has been provided on their website or any supporting publications.

SEAtS software perpetuates the dominant mindset of current learning applications by focusing on performance management specifically with the ability to improve or increase positive behaviours in students. It achieves this through the usual means of harvesting data from the current educational system and presenting it to analysis clients. Moving through the framework we see that SEAtS strategically piggybacks on existing LMS, SIS, and VLE’s of the educational institution it partners with. This is strategic in that the responsibility of the data is shared and owned by the school and pre-existing third Comment

parties it works with. Depending on individual school goals and objectives, which in this scenario is performance management, SEAtS

9 suggests which data they will need to collect and store in their data repositories. One component of the SEAtS software that separates it from traditional business intelligence is the use of assisted machine learning. Initially, the SEAtS software will only serve to provide past, present, and future information through the form of reporting, alerts and extrapolation. However, after time and once historic data sets have been built, SEAtS can begin to build models that can be trained to create refined predictive profiles of potential ‘at risk’ students that provide some modelling and recommendation insights. This company makes bold claims but provided little specifics on how their software worked. It repeatedly highlighted the different types of educational data it utilized and how the LA could support increased student performance but failed to mention the following:

•

Differentiation between their software use for learning and academic analytics

•

Connection to learning theories

•

Focused highly on Outcome data with limited mention of input data

•

Examples or case studies of metric data used

•

What types of methods are used for predictive analytics

•

Discussion on privacy or ethical use of data

•

No reviews or testimonials could be found from current or past clients on their website or in a general web search.

10

RECOMMENDATION I would not recommend this tool for myself or colleagues because there are too many unanswered questions. The key hesitance is the lack of grounding in learning theory as well as knowing what methods for predictive analytics are being used (Simple? Linear? Logistic? etc.). The SEAtS Software platform is promising and has an appealing UI/UX design. It also could transform current teaching and administrative practices should the claims be sound. I would recommend that further investigation be completed and perhaps a trial use of their software first before purchasing.

11

REFERENCES Cooper, A. (2012). A framework of characteristics for analytics. CETIS Analytics Series, 1(7). Bolton, JISC CETIS. Ifenthaler, D. (2016). Recorded for the MOOC Analytics for the Classroom Teacher offered by Curtin University, Australia. [YouTube, 3 mins.] SEAtS Software | Student Success Platform. (n.d.). Retrieved February 22, 2019, from https://www.seatssoftware.com/ Scheffel, M., Drachsler, H., Stoyanov, S. & Specht, M. (2014). Quality Indicators for Learning Analytics. Journal of Educational Technology & Society, 17(4), 117-132. Sclater, N. (2017). Chapter 9. Metrics and Predictive Modelling, in Learning Analytics Explained (pp. 88-98). New York, USA: Taylor & Francis. Slade, S., & Prinsloo, P. (2013). Learning Analytics: Ethical Issues and Dilemmas. American Behavioral Scientist, 57(10), 1510-1529.