e-Learning Ecologies MOOC’s Updates

Essential Update #3: Multimodal meaning and a VR piano


Multimodal meaning, as defined by Cope and Kalantzis, is meaning which is distributed through more than one modality, such as text, audio, imagery, data representations, or the tactile sense.


Though multi-modality predates the digital era - as in Cope's example of a children's picture book, which carries its message both in text and pictures - the digital age enhances the ability of the teacher or learner to use multi-modal messaging. The great difference between this and previous ages is that a single individual now has the ability to create multi-modal messages relatively easily; Previously each modality tended to be confined to a specific technology: print and ink for text or images, specialised recordings for audio, and so on. Now, however, pixels on a screen can be used to generate text, images, and unique data representations.


Multi-modality can be considered an affordance of technology for three reasons: firstly, as stated above, an individual with a modest amount of equipment, and little specialised know-how, can generate multi-modal messages with ease. Secondly, and particularly in conjunction with the affordances of ubiquity and active knowledge-making, students are able to benefit from much richer learning experiences which engage a number of different senses, and increasingly are being given the opportunity to produce assessed work in more than a single modality.


Finally, using a non-clinical definition of synesthesia, Cope and Kalantzis posit that this switching between modalities helps to boost the effectiveness of learning, and so this can be considered a further benefit of this affordance. This bears a superficial resemblance to the theory of learning-styles, but there is a significant difference between them: where notions of learning styles assign a single preferred learning style to each learner, a synesthesic view holds that all individuals learn best when presented with mixed-modalities. This view appears to be supported by a growing body of research which suggests there is little evidence to support the claim for individual learning styles. (See for example Scott, 2010.)


One example of multi-modality afforded particularly by technology which I think is worth exploring is the use of haptic feedback with virtual reality. Haptic feedback devices give the wearer of specially-designed equipment the impression of force upon their body. Earlier generations of VR equipment allowed the visual-spatial and audio experience of the virtual world, but their potential for use in certain training fields was limited by the inability of the user to feel the effect of their interactions with the world: a VR training package for surgeons, for instance, would not be much use unless the doctor understood how much force to apply with a particular tool.


One recent example of this is the use of a haptic glove with VR technology to helps students learn to play the piano, experiencing the resistance of the 'keys' on their fingers through force applied through the glove (Huang et al, 2008). While this might seem trivial - why not just go to a school with a piano and try the real thing? - it exemplifies how the different affordances of technology have impacts in combination. In this case, the use of haptic gloves and VR (multi-modality) combines with the affordance of ubiquity to allow a student to practice playing the piano without having regular access to a physical instrument.

References:
Huang, K., Do, E. and Starner, K. (2008). PianoTouch: A wearable haptic piano instruction system for passive learning of piano skills. Available online at: http://ieeexplore.ieee.org/abstract/document/4911582/?reload=true
Scott, C. (2010). The Enduring Appeal of ‘Learning Styles’. Available online at: http://journals.sagepub.com/doi/abs/10.1177/000494411005400102
 

  • OleksanDeR OleksandeR