Language consists of signs: words or morphemes. The sign does not refer to a
concrete object. "Cow" does not mean a specific cow standing in a specific
field. The sign must be part of an utterance to get a specific meaning.
The linguistic sign consists of a content (sense/meaning) and of a expression
(letters or sounds/phonemes) These smaller units have no sense, but they are
the condition for making sense: Thus the changing of one letter makes the
sign(word) change its meaning: cat vs. hat. Language has a double articulation,
the first is the division in words and/or morphemes, the second is the division
in letters or phonemes.
The sense does not depend on the letters/phonemes. The concepts of "cat" does
not stem from c + a + t. If this was the case the sense of "cash" should be
close to that of "cat" which is not the case! The meaning of the linguistic
sign is arbitrary (not given by nature), it is due to a convention, but this is
what makes language stable.
The linguistic sign are combined into syntagms (sentences or utterances). We
communicate only in sentences. If we use a single word it represents a whole
sentence. The syntagm is the concrete entity referring to the "world". The
entities that could be placed/put at the same place in the sentence form a
paradigm. The speaker has to make a choice within the paradigm.
The rules of combinations and selection are called the code.
Thus language is governed by rather strong codes or rules. It is a bit more
complicated when we come to the visual communication. Genuine visual
communication is rare. The combination of visual and linguistic communication
is not. But what is meant by visual communication is mostly a visualisation of
linguistic communication (~ a translation from linguistic to visual
expression).
Like the linguistic sign the visual sign consists of an expression and a
content (sense). But with some differences. Visual sign can be divided into
forms, indexes and icons. The problem is that the same sign can be form, index
or icon according to the context. The forms are arbitrary in the same way as
the linguistic sign (they are symbols), the indexes are traces and the icons
have a more or less clear resemblance with the thing they represent, they are
motives. Icons are called motivated signs. There is anyhow a gliding scale from
purely motivated signs to totally arbitrary signs.
This group of entirely arbitrary signs is made of graphics, organnigrams,
decorations, signs in non-figurative art, symbols like the swastika. Some
traffic signs: triangles, parking prohibited, red-green-yellow lights are
purely conventional
The icons consist of all kinds of pictures representing an object. Some are
clearly motivated: photos, naturalistic drawings and paintings. But as soon as
the sense begins to depend not on resemblance but on convention we are in a
twilight zone between the arbitrary and the motivated signs: pictograms
(traffic signs with two children), geographical carts, plans or schemes (of
houses or machines). And where should we place caricatures?
Many pictures have a double meaning a concrete visual and a symbolic
(conventional - and arbitrary) A picture of an old man with a key in his hand
may be the picture of an old man, but it is more likely to represent Saint
Peter (we know by convention that the key is the symbol of Saint Peter). Modern
advertising and the medieval churches are filled up with signs having this
double status
The indexes function in a different way, an index is a trace of the production
situation: You see the stroke of the brush on the wall painting, it is an index
of the painter's working process. The strokes represent a girl. The painting is
an icon.
The phonemes are clearly defined (structural) entities, they are stable. The
visual sign can only be defined in a context. A simple line may in one context
be a part of a roof (and has thus no meaning in itself), in another context it
is the top of the roof and gets sense within the whole representation of the
roof and finally in a third context it represents the horizon - and needs
almost only the frame of the picture to get sense.
Many ways of systemizing the visual elements as signs have been made. The
simplest is to divided the elements in points, lines, plans and more
complicated forms - but this is only a practical way of ordering
Some authors have attempted to make 4 levels of articulation:
- Syntagms and motives (more or less naturalistic), that are perceived as
surfaces or volumes. These signs have normally a referent.
- Closed forms, they may too be perceived as surfaces and volumes having a
referent.
- Non-closed forms (different combinations). They have normally no sense or
referent.
- Open and simple forms (points, lines or curves). They have normally no sense
or referent.
But the problem remains: What is a simple open form in one picture, may be a
motive in another.
The visual sign is thus polysemic and we do not always agree about the sense of
a given picture/photo/drawing. And as we have seen the visual signs have the
same variation as the linguistic signs between a first and concrete meaning
(picture of a rose) and a secondary and more symbolic meaning (the rose as sign
of love or youth). The first meaning is normally called denotation, the
secondary connotation. Many secondary meanings are conventionalised, they have
become fixed symbols (the key for Saint Peter, the specific hat marking the
cowboy).
In all these cases we should notice that the visual meaning is a translation
from linguistic meaning or could be translated into linguistic meaning. When we
talk about visualisation we have obviously a case of translation from
linguistic expression to visual expression.
Normally it is thought that language (as compared to pictures) is clear,
rational, easily made monosemic whereas the visual expression is emotional,
fluctuating, polysemic or even worse: ambiguous.
That is why the visual expression needs a linguistic anchorage to get rid of
superfluous meanings. That is why you almost always have some kind of
linguistic message related to the picture in an advertisement - the message in
the advertisement has been established before the picture is added. The
sense/meaning was already there - which is perfectly true.
The anchorage by montage is normally considered as another way of anchorage, a
more subtle - and more ambiguous way.
All this may be right in many cases, but as we shall see things are probably a
bit more complicated - and if you want to build up a multi media system based
on the use of the visual expression you had better look at the meaning of
pictures once more. What do pictures express , what can they express?
You have to look at the visual expression, you have to look at the pictures as
an autonomous entity.
Talking about the relationship between pictures and narration in the religious
paintings (the life of the saints) and in comics, Fresnault-Deruelle, professor
at Paris I, Sorbonne, says:
"I am not far from thinking that any comic is double/has a double meaning.
There is a surface story/narrative (that can be resumed as a scenario) serving
as an alibi for a quite different story......The discourse that could seemingly
be reduced to a classic narration, could it hide another story that subverts
the first one?" Fresnault Deruelle talks about the latent story, the frozen
picture that opens up new and unforeseen meanings, the image-carrefour, cross
road pictures that are the conjunctions of meanings and tensions. The picture
makes new stories possible, that is what he calls narrativity as opposed to
narration.
As specific examples of these cross road pictures, pictures that are promises
of story and latent meaning in spite of the official narration,
Fresnault-Deruelle quotes/indicates the paintings of Vermeer, Fragonard or
Hopper.
It should be noted that this distinction between official narration and latent
story/narrativity is not confined to painting and drawing. Even photography
knows this distinction. Many famous photographers have made photos with this
latent meaning - best of all is probably Cartier-Bresson in opposition to Capa,
whose pictures were normally clearly narrations that illustrate a story.
Pictures (alone or mostly combined with text) can be either narrative
(narration or narrativity) or they are what you might call discursive, that is
the transmitter gives a message to the receiver who is implied not only as
onlooker but as spoken to, the transmitter wants the receiver to make or to
realise something. This kind of discursive picture is a variant of the
narration-pictures. The narrativity is again in this context a latent meaning
opposed to the meaning of the scenario. The examples are of course any kind of
instructional picture or advertisement, propaganda.
So we must realise that the sense of the pictures is not only a question of
anchorage guiding us to the proper meaning, but that any (good) picture opens
up several meanings - and that this does not disturb the intended meaning (the
narration of the scenario), at the contrary it makes the brain work properly!
If you are in face of boring, badly made pictures you spend too much time on -
being bored and on seeking destructive pleasure. If the pictures in the
multimedia system are too redundant the brain does not work.
We must realise that pictures only give sense when we (the on lookers) accept
that the picture asks us questions, breaks the redundancy.
Space: The single picture can give you a partial description of space.
Sequential pictures can describe space adequately.
Time: The single picture can only implicitly give an impression of
situation or action/narration and thereby give an impression of time Sequential
pictures form a succession and thereby an impression of time.
Causality: If the single still picture refers to some kind of fixed
narration/scenario be it as a pattern or as a concrete story the picture is
able to present causality
Sequential pictures can via the succession more easily present causality - but
still via fixed narration/scenario.
Abstractions: Pictures are normally considered to be able to represent
the concrete, visible world - but not the abstractions. This is a rather
simplistic way of seeing. But it is true that pictures can not be visual
translations of linguistic abstractions.
Objects: Pictures can represent objects!!
Characters: Pictures can represent faculties/characters if they have a
visible expression (and then mostly a conventional expression): strength,
cruelty, stupidity etc.
Situations and actions: Situations can of course be represented as can
actions, but not if they are complex and not if there is not a kind of
conventional symbols used.
Some illustrative examples. Which is the most logical, precise non-ambiguous,
language or pictures, in the following examples?
Compare a map or a graphic representation of space ( a square with trees placed
in a specific pattern) with a linguistic description of the place of the trees.
Which of the two presentations gives you a clear information?
Take a fairy tale - any fairy tale. Make illustrations to this tale. There is
no limit to possible number of illustrations and the kinds of illustrations
possible. All the illustrations match with the same linguistic text - are
translations of the text. The text must be terrifically ambiguous!
Ask 25 people to draw this simple non-ambiguous sentence: "He stood in front of
the house". Even if we agree about who is "he" and what is "the house" there
are many drawings possible.
If we think that the visual expression has a pedagogical importance in multi
media systems the use of pictures has to be taken into account from the very
beginning. We cannot start by making specifications about linguistic content,
the interaction with the computer etc. if we do not imply the pictures.
Pictures are not servants - they must have equal rights!
If not this is what happens: You see plenty of language textbooks filled up
with pictures who tell you above all this simple fact: the artist was bored
when making the pictures, the pictures are bad icons (or worse), but they are
good indexes of the state of mind of the poor artist.
Fausing, Bent & Larsen, Peter: Visuel kommunikation bd. 1 - 2. Medusa
1980.
Fausing, Bent & Larsen, Peter: Billeder - analyse og historie.
Dansklærerforeningen 1982.
Fresnault-Deruelle, Pierre: L'éloquance des images. puf 1993.
Groupe My: Traité du signe visuel. Seuil 1990.
Johansen, Jørgen Dines & Larsen, Svend Erik: Tegn i brug. Amanda
1994.
Kjørup, Søren: Hvorfor smiler Monal Lisa? Gjellerup 1983.