https://d2ng68cf36kxsk.cloudfront.net/store/public/post/832/post_images/Slide251.JPG">

However, for their monosensorial possibilities, text and speech could not be more different. This is is another of the reasons we have arranged the array of meaning forms in the way that we have, maximally separating text and speech.

This is also the reason why, in this grammar, we have found it necessary to abandon the notion of “language,” replacing it with two fundamentally different forms of meaning – written text closely aligned in its form to image and space, and speech, closely aligned in its form to sound and body. There are connections between text and speech, to be sure, because the graphemes in some languages are phonemic as well as ideographic. However, phonemic reading is impractical – most reading in alphabetically represented languages captures morphemes as units.^§MS0.2.1b Then there are primarily ideographic languages, notably Chinese. In the era of digital text, most graphemes in Unicode are ideographic, rather than phonemic.^§MS0.2.1a The syntax of text and speech are fundamentally different.^§MS0.2.7a And much “vocabulary” in digital text is practically unspeakable – for instance, the linked names of webpages,^§MS1.1.1d finely differentiated colors named by number,^§MS1.3.1h or abstract mathematical or algorithmic relations.^§MS1.3.2 From the point of view of a multimodal grammar, the connections between text and speech are no stronger than those between text-image-space or speech-sound-body. In the digital era, the separations between the forms of text and speech have become greater than ever.

Fig. 0.3: Meaning forms and human senses

However, by technological sleight of hand, digital media also bring together these most disparate of forms, text-image and speech-sound. Digital media are unlike the analogue media that preceded them in the degree to which they can record and render both text-image^§MS1.2.2e and sound-speech^§AS1.4.2d by the same underlying mechanism, decomposing them into quantities (naming by number, quantification, calculation) and recomposing them at the point of rendering.

This contradictory pulling-away of text from speech as meaning forms, at the same time as their material coming together in digital media, is one of the main reasons why there is a certain urgency for a grammar of multimodality. Any such grammar must show a commitment to classifying forms in order to analyze their distinctive affordances, and the changes in our capacities to mean that are under way in the era of digital media, when these diverging meaning forms are brought together onto the same platforms for their representation and communication. Paradoxically, it is the very media that bring them together that also tear them apart.

One of the few things in the history of media that has come close to digital media in bringing together a range of meaning forms was sound film, but this it did awkwardly, in juxtaposed channels, literally sticking side-by-side fundamentally different recording technologies. Now the technologies for creation and distribution are the same, the technologies of binary calculability.

As well as the digital convergence of text-image and sound-speech, there are many other such conjunctions. Space and object can be represented in computer aided design, and recomposed as assembly instructions and 3D printing. Body can be recorded and represented in video and holographic representations. So-called 4D cinema even adds gimmicky haptics to cinema, and virtual and augmented reality a sense of bodily movement.^§AS1.3.2a Of the human sensorium, only smell and taste cannot (yet) be digitized, either as recording or rendering. Nor can the sensorial roots of emotion – feelings of fear, anger, disgust, pleasure, or happiness, for instance.

---

Kalantzis, Mary, and Bill Cope. 2020. Adding Sense: Context and Interest in a Grammar of Multimodal Meaning. Cambridge UK: Cambridge University Press. pp. 13–15. https://doi.org/10.1017/9781108862059

Multimodality §AS0.3