As I was viewing this week's lectures, I started thinking about the relevance of multimodal knowledge representation to the fileds of learning.

As a first example, let us consider painting. In my mind I have pictures, e.g. the picture of the painting "Guernica", I have text, e.g. Picasso's statements regarding the background to painting the picture, and also have video clips showing the bombardment of the City of Guernica by the Nazis, and so on. In a sense I have a network of nodes, each node being something important, like Picasso, or Spanish Civil War, and then the bits and pieces that are linked to each node, like a painting Picasso made, or the City of Guernica. Each of these bits and pieces in turn may come in different modes: text, image, video, sound. I will not get into the technicalities of how all of these are actually stored and represented in the mind. Let us just assume that they are stored somehow, and they have this structure of a network with nodes and many bits and pieces hanging off the nodes. 

Let me now present my second example: the Russell - Zermelo paradox. I suggest that my knowledge of the paradox does not include video and sound. It may only contain some images of graphical depictions of sets. 

As we can see from these two simlistic examples, it appears that not all fields of knowledge are equally open and/or affected by multiple modalities. If this is the case, what are the implications?