ql-align-justify">

Enhancing Story Comprehension through Multimodal Knowledge Representations

In contemporary digital education, meaning is no longer constructed through text alone. A powerful yet often under-discussed multimodal meaning concept is Interactive Simulations as Multimodal Knowledge Representation. This concept emphasizes that learning occurs through the integration of multiple modes such as visuals, text, movement, symbols, and interaction rather than through reading or listening only. In this approach, learners do not simply receive information; they actively build meaning through exploration, manipulation, and observation.

Multimodal meaning refers to how knowledge is created through the combination of different forms of communication and representation. In interactive simulations, meaning emerges from the relationship between what learners see, read, anddo. This aligns with Richard Mayer’s Multimedia Learning Theory, which suggests that people learn better when words and relevant visuals are presented together because the brain processes information through separate verbal and visual channels. When both channels work together, understanding becomes deeper and more meaningful.

Multimodal Knowledge Representation involves digital environments where learners can control variables, observe dynamic visual changes, interpret graphs or models, and connect these with written explanations. Knowledge is not passively absorbed but actively constructed through engagement with multiple modes of representation. This makes learning more interactive, participatory, and cognitively rich.

A practical example is the digital storybook “The Fantastic Flying Books of Mr. Morris Less more”. In this story, students read text while watching animated visuals of characters and settings. Audio narration guides the reading process, adding voice intonation and emotion to the story. Students can interact with certain parts of the story, such as clicking on objects to reveal animations or extra details. This combination of text, visuals, audio, and interactivity allows learners to construct meaning actively, rather than passively absorbing information. Another example is the use of platforms like Storybird or Book Creator, where students can create their own multimodal stories by integrating drawings, text, and recorded narration, allowing them to synthesize knowledge across different modes.

Reference:

https://www.researchgate.net/publication/249738784_Writing_in_Multimodal_Texts_A_Social_Semiotic_Account_of_Designs_for_Learning

https://youtu.be/a6qOkvDEXdI?si=1u7n8a5wz9vCgg2H

Enhancing Story Comprehension throEnhancing Story Comprehensugh Multimodal Knowledge Representations