Produced with Scholar

Technology Mediated Learning Analysis

Project Overview

Project Description

'Parse' a learning technology - what is its underlying theory of learning and how is this reflected by the way it works in practice? When discussing the theory of learning read and cite (with links) the theorist works (Work 1) of other course participants.

Icon for Learning Analytics in Education

Learning Analytics in Education

Overview

As online learning environments increasingly accommodate hundreds of thousands of learners, instructors, administrators, policymakers and researchers are looking at ways to gather and analyze data around student enrollment, demographics, engagement and interactions, and academic performance with technology tools.

Big data have been used on commercial websites to personalize users’ experience. Capturing and analyzing user activities, identifying spending trends, and predicting consumer behavior have changed how decisions are made and resources are allocated in businesses. Data drives increased organizational efficiency and better strategies with greater clarity and confidence.

Media embedded September 6, 2015

With the current shift in educational settings to blended and online learning and the introduction of learning management systems, it is not a surprise to see that big data has found its place in education and will soon be extensively implemented. The educational application of big data is called learning analytics (LA) and the term can be defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens & Long, 2011).

One of the factors leading to the recent emergence of learning analytics is the increasing quantity of analyzable educational data. In educational settings, technological advancements in allow numerous learner information to be stored, organized, accessed and available for analysis. LA offers a promising approach for gathering, analyzing and reporting these educational big data and advancing our understanding of the learning process. With the aid of LA provides teachers, students and other stakeholders insight into learner interactions and performance; it also enables universities, schools, and corporate training departments optimize the learning environment so that the quality of learning can be improved.

Media embedded September 7, 2015

 

'Parse' the Educational Technology

Chatti et al. (2014) proposed a reference model for learning analytics to explain the technology from four dimensional:

  • What? What kind of data does the system gather, manage, and use for the analysis?
  • Who? Who is targeted by the analysis?
  • Why? Why does the system analyze the collected data?
  • How? How does the system perform the analysis of the collected data?
Learning Analytics Reference Model (Chatti et al., 2014)

What: Learning analytics involves a number of types of data. The data can be general individual information such as students' demographic information, personal preferences, past academic performance in other institutions or previous courses. The data can be more context specific, for example, a record of students’ key actions in interaction with learning management systems (LMS), such as logging on, posting and viewing messages, and accessing materials, etc. The data can also include information generated in the context, such as the content in students' assignments and postings, and their progress through a specific interactive activity such as a quiz. Combining these data from different resources can contain many variables that data mining can explore for model building and give algorithms better predictive power.

Who and Why: Different stakeholders may hold different perspectives, goals, and expectations for LA. Students will probably be interested in how analytics might support their progress and improve their performance. Teachers might be interested in how analytics can augment the effectiveness of their teaching practices or support them in adapting their instruction to the individual needs of students. Educational institutions can use analytics tools to support decision making, identify potential students at risk, and improve student retention.

How: LA applies different methods and techniques to detect interesting patterns hidden in educational data sets such as statistics, information visualization, data mining, and social network analysis. Most existing LMS integrate reporting tools that provide basic usage statistics of the students' interaction with the system. Information visualization helps to prepare the information in a more user-friendly format which might facilitate the interpretation of educational data. Data mining helps to discover useful patterns from the data and social network analysis visualizes and analyzes the social structures between individuals.

Here is an infographic that might help to understand how learning analytics work.

 

The Underlying Learning Theory

Learning technologies are only useful when they are based on ground theories. The emerging field of learning analytics is at the intersection of numerous academic disciplines, and therefore draws on a diversity of methodologies, theories and underpinning scientific assumptions. The interdisciplinary field integrates research and methodology related to statistics, data mining, social network analysis, data visualization, machine learning, psychology, artificial intelligence, and educational theory and practice.

The three learning theories associated most with educational technology are cognitivism, behaviorism and constructivism. At first glance, behaviorism might be the most closely related paradigm associated with learning analytics as a lot of behavioral data are gathered. However, one criticisms against behaviorism is that it does not account at all for inner mental states, thoughts, feelings, cognition, meanings ascribed to events by learners, etc. (Here is a great resource about the theory and it's criticisms from Stanford Encyclopedia of Philosophy).  If there is no difference in behavior, then there is no difference between the learners. In other words, if learning analytics only captures the students' behaviors such as the click patterns, it will miss other components such as thoughts, feelings and attitudes, which are hugely important to the educational process.

Therefore, it is also important to design learning analytics from the constructivist's perspective. Constructivists argue that a learner should be an active participant in the learning process and construct his or her own individual body of understandings. From a constructivist position, learning analytics should focus on tracking progress to facilitate learner's self-awareness, instructional improvement and personalized learning.

The Technology in Practice

The best known application of LA in higher education is Course Signals developed at Purdue University (Arnold & Pistilli, 2012). Using data collected by Blackboard and data from the institutional Student Information System (SIS), Course Signals uses a data-mining algorithm to identify students at risk of academic failure in a course. Specifically, Course Signals identifies three main outcome types: a student at a high risk, moderate risk, and not at risk of failing the course. These three outcomes are symbolically represented as traffic light where each light represents one of the three levels of risk (red, orange, and green respectively). The traffic lights serve to provide an early warning signal to both instructor and student. This signal is designed to prompt a form of intervention that is aimed at improving the progression of the student identified as at risk of failure. Early studies of Course Signals showed high levels of predictive accuracy and significant benefits in the retention of the students who took at least one course adopting the early alert software versus those who took a course without the Course Signals tool (Arnold & Pistilli, 2012).

Through a quick search, I found that most active players in the field of learning management systems or adaptive learning technologies have integrated learning analytics capabilities or learning analytics tools. It could be a good starting point to those institutions who have already implemented LMS and want to take more effective use of existing resources. The following table lists a few of them:

COMPANY/INSTITUTION DESCRIPTION
Blackboard Analytics Combining the extensive data from Blackboard Learn™ with student and course attributes to create comprehensive reports and dashboards
Canvas Analytics Canvas Analytics delivers intuitive dashboards for students, educators and administrators to aid in making data-driven decisions that improve learning.

Coursera Course Dashboard
 

A “Google Analytics”-style Course Dashboard to give teaching staff a top-level view of what was going on in their courses: Who is taking my course? Where are they coming from? How are they doing? What are the trouble spots, where learners fall off track?
Desire2Learn Insights A data mining and reporting service to create and customize a range of reports that you can use to monitor user progress, course access, engagement, etc.
edX Insights EdX Insights provides course teams with data about student background and activity throughout the course. Insights includes data on student enrollment, student demographic information, and student engagement activities
Google Analytics A freemium web analytics service offered by Google that tracks and reports website traffic
Moodle

Analytics & Reports

Engagement Analytics provides information about student progress against a range of indicators and student activities
SmartSparrow Knowledge Analytics Smart Sparrow’s Knowledge Analytics gives teachers insight into their students’ minds: what they’re learning and how, what’s not working and why.
SNAPP Used for conducting real-time social network analysis using data visualization techniques

 

Critical Reflection

The use of learning analytics offers tremendous potentials to transform education from a one-size-fits-all delivery system into a proactive and responsive framework adapted to meet individual needs and interests. Learning analytics enables commercial companies or education institutions to gather data on students’ learning experiences to improve learning and teaching experiences, enable personalized learning, identify learning problems and at-risk students, and assess courses and programs. (Horizon Report, 2014)

Despite its potential, the implementation and use of learning analytics is not a risk-free proposition. Although the concept of using these forms of analytics for informing education practice has been well received, the majority of educational institutions seldom make optimal use of their available data and analytical resources. One possible reason might be the field is still in its relative infancy stage and the level and sophistication of the data analysis performed are quite limited.

There are also legal, ethical and political challenges involved, such as data ownership, student privacy etc. If students feel their privacy is being invaded, they may be reluctant to allow their data to be used for research and analysis.

There might be some other more fundamental reasons, like research hasn't clearly identified which variables should be tracked and what types of outcomes these data could have impact on. Therefore, researchers like Hendricks, Plantz and Pritchard (2008) urge stakeholders to define goals or objectives to determine what data to capture before "drowning in data" (Snibbe, 2006). Probing questions should be asked such as: What do we want to achieve? Are you measuring what we should be measuring? How will the information be used? How can we create innovative metrics to illuminate deeper outcomes? (Elias, 2011). These are critical questions that educational researchers and practitioners as well as commercial vendors need to consider to attain full adoption of learning analytics.

 

Conclusion

With growing interest in learning analytics in the education sector, it is important for educators, administrators, policy makers and researchers to understand how learning analytics work, when they work and what the available tools are. Data collection and methodology are critical to determine what insights are gained and lost with analytics. The complexity of a social process in learning cannot be adequately assessed through basic metrics such as logins, time online, and clicks. The identification of student characteristics or contextual variable is equally important.

Learning analytics hosts many opportunities and challenges which require stakeholders' efforts to develop an understanding of the potentials and limitations of learning analytics and build capacities for implementation and interpretation of learning analytics systems and solutions. Those institutions considering a learning analytics project should review the literature thoroughly and consider compatibility, identification, and availability of data elements when selecting or designing their tools as what is reasonable to one institution may not be reasonable to another. Most importantly, stakeholders involved in the educational decision making process should analyze the current existing system and consider the following questions to guide their future work (Lepouras, Katifori, Vassilakis, Antoniou, & Platis, 2014):

(a) How the existing system should be used to maximize the benefits from its functionality?

(b) Which additional information and/or structural changes should be attached to curricula and
courses, so that they can provide the information needed for the analytics to be effective?

(c) How variables or metrics can be exploited (including the highlighting of proper and improper
interpretations of available analytics)? and

(d) How results from the analysis process can be used to improve courses and curricula?

 

References

Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). New York, NY, USA: ACM. doi:10.1145/2330601.2330666
Chatti, M. A., Lukarov, V., Thüs, H., Muslim, A., Yousef, A. M. F., Wahid, U., Greven, C., Chakrabarti, A., Schroeder, U. (2014). Learning Analytics: Challenges and Future Research Directions. eleed, Iss. 10. (urn:nbn:de:0009-5-40350)
Elias, Tanya (2011). Learning Analytics: Definitions, Processes and Potential. Retrieved from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf

Hendricks, M., Plantz, M.C., & Pritchard, K.J. (2008). Measuring outcomes of United Wayfunded
programs: Expectations and reality. In J.G. Carman & K.A. Fredricks (Eds.),
Nonprofits and evaluation. New Directions for Evaluation, 119, pp. 13-35.\

Lepouras, G., Katifori, A., Vassilakis, C., Antoniou, A., & Platis, N. (2014). Towards a learning analytics platform for supporting the educational process (pp. 246–251). Presented at the IISA 2014 - 5th International Conference on Information, Intelligence, Systems and Applications. http://doi.org/10.1109/IISA.2014.6878750
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30-40.

Snibbe, A.C. (2006). Drowning in data. Stanford Social Innovation Review, Fall 2006, pp. 39-
45.