Mathematics as multimodal semiosis

Mathematics, Substance and Surmise: Views on the Meaning and Ontology of Mathematics (2015)

Kay L. O’Halloran¹

(1)

Faculty of Humanities, School of Education, Curtin University, Level 4, Building 501, Kent Street, Bentley, Perth, WA, 6102, Australia

Kay L. O’Halloran

Email: Kay.Ohalloran@curtin.edu.au

Abstract

In this chapter, mathematics is considered to be a multimodal semiotic process involving the use of language, images, and mathematical symbolism, each with their own systems of meaning which integrate to create mathematical knowledge. The specific relations between linguistic, visual, and symbolic systems have led to semantic expansions in mathematics, resulting in a hierarchical knowledge structure which extends beyond the reach of other forms of human communication.

1 Introduction

Mathematics is a subject that is one of the finest, most profound intellectual creations of humans, a subject full of splendid architectures of thought. It is a subject that is also full of surprises and paradoxes. Mathematics is said to be nothing more than organized common sense, but the actuality is more complex. As I see it, mathematics and its applications live between common sense and the irrelevance of common sense, between what is possible and what is impossible, between what is intuitive and what is counterintuitive, between the obvious and the esoteric. The tension that exists between these pairs of opposites, between the elements of mathematics that are stable and those that are in flux, is a source of creative strength [11, p. viii].

Davis [11, p. xix] describes mathematics as “one of the greatest human intellectual accomplishments,” with inherent features which are not easy to characterize. Indeed, mathematics is a remarkable achievement and a major “source of creative strength” [11, p. viii] which has, if one considers technology and other scientific products derived from mathematical formulations, (literally) changed the nature of life on earth. But how is this possible? That is, what is the nature of mathematics that has resulted in new domains of knowledge, which are so usefully employed in the natural sciences? The aim of this chapter is to explore this question by conceptualizing mathematics as a multimodal semiotic process (i.e., involving sets of inter-related sign systems) which draws upon and integrates language, images, and symbolism to create meanings which extend beyond those possible with other forms of human communication. In this regard, mathematics is considered to be a specialized tool for thinking, specifically designed to move beyond our everyday experience of the world to an abstract semiotic realm for restructuring thought and reality. Natural language also functions to organize and structure human experience on an abstract semiotic plane (e.g., [17, 50]) but, unlike mathematics, it lacks the ‘meaning potential’ [23] to effectively model and predict events in the physical world. The purpose of this paper is to explore how mathematics achieves its unique functionality, as a designed system, which is the result of centuries of human effort.

In what follows, multimodal social semiotics, the theoretical approach to mathematics adopted in this paper, is presented, followed by a discussion of the functions of natural language, images, and symbolism, and their integration in mathematics texts. There are three key ideas. First, human sign systems are tools for structuring thought and reality. Second, by developing new integrated, written systems in mathematics which combined textual forms (linguistic, symbolic) with visual forms (graphs, diagrams, and figures), it was possible to construct new views of the physical world. Third, by its very nature, mathematics as a multimodal hierarchical knowledge system can be usefully employed to describe and predict events in the material world, but it has limitations in terms of modeling and predicting the human socio-cultural world. From this perspective, an important challenge today is to develop new semiotic tools for modeling the social world—a general “science of [human] meaning” [6]—to replicate the effectiveness of mathematics in the natural sciences.

2 Theoretical Foundations: Multimodal Social Semiosis

Semiotics is the study of sign systems and processes (e.g., [8, 39]), and ‘social semiotics’ is the branch of semiotics which studies human signifying processes as social practices [17, 30, 49]. Social semiotics is concerned with different sign systems (e.g., linguistic, visual, aural, and gestural), and their integration in texts and interactions, interpreted within the context of situation and culture. ‘Multimodal social semiotics’ is specifically concerned with the study of relations within and across semiotic resources, both as multimodal systems (i.e., the process) and as multimodal texts (i.e., the product) (e.g. [32]). That is, multimodal social semiotics is concerned with modeling semiotic resources as inter-related systems of meaning, and analyzing how choices from the different systems work together to create meaning in multimodal texts. In this case, the semiotic resources of interest are language, image, and mathematical symbolism, which are the fundamental resources through which mathematical knowledge is created. Other resources, such as three-dimensional objects, calculating devices and digital technology, are also briefly considered.

Following Halliday’s [17] and Halliday and Matthiessen [28] social semiotic theory, the basic premise is that language and other semiotic resources, such as images, symbolism, gestures, actions, and sounds, are sign systems which have evolved to fulfill particular functions: (a) to record experience as happenings and events; (b) to logically connect those happenings and events; (c) to create a stance toward the happenings and events while enacting social relations; and (d) to organize the message itself. These functions, called the experiential, logical, interpersonal, and textual ‘metafunctions,’ respectively [22, 23], are the four strands of meaning which are communicated in any social interaction—that is, humans construct, reason about, and evaluate happenings and events while negotiating social relations. In reality, these four strands of meaning combine, but they can also be considered individually for theoretical and analytical purposes.

In order to (differentially) fulfill the four metafunctions, language and other semiotic resources have each evolved a unique infrastructure (or architecture), formulated as ‘systems of meaning’ in systemic functional theory [21]. For example, language has a multidimensional architecture [22], which includes lexical and grammatical systems which operate at different levels (i.e., sound, word, clause, sentence, and paragraph). In this case, the term ‘grammar’ does not refer to a set of rules that govern language use; rather, it is a description of the potential of language to create certain meanings, from which selections are made in texts. In this respect, Halliday [27] views “language as system” and “language as text” as “two aspects of one single phenomenon” (p. ii). That is, the linguistic system has a meaning potential, modeled as sets of inter-related networks of options, and the text is the process and product of selecting from that potential. Linguistic patterns are built up culturally over time so that any instance of language use is conditioned by previous choices. That is, linguistic choices are routinely deployed so they form recognizable cultural configurations (e.g., casual conversation, news bulletin, academic paper, and so forth). In multimodal semiotics, Halliday’s formulations of metafunction, system and text are extended to other semiotic resources, which are also seen to have evolved their own infrastructure to fulfill particular functions, often in collaboration with language. In this regard, systemic functional theory forms a useful framework for viewing mathematics as a (multimodal) semiotic tool for restructuring thought and reality.

In what follows, language, image, and mathematical symbolism are viewed as semiotic resources which integrate to create mathematical knowledge. In this case, ‘the whole is other than the sum of the parts,’ as maintained by Gestalt psychologist Kurt Koffka [31], because the precise relationships between the three resources create a semantic space which extends beyond the potential of each individual resource. Indeed, the “creative tension” of mathematics [11] originates (at least in part) from the transitions between language, image, and symbolism, where ‘meta–metaphorical phenomena,’ referred to as semiotic metaphors (or inter-semiotic metaphors) are created (e.g., [40]). The semantic potential of semiotic metaphor extends beyond traditional conceptions of metaphor which are language-based; for example, conceptual metaphors are seen to map one domain of experience onto another (e.g., “life is a journey”) (e.g. [34]). Derivatives of conceptual metaphor have been applied to multimodal texts, resulting in the formulation of ‘visual metaphor’ and ‘multimodal metaphor’ [15]. However, semiotic metaphors, arising from the transition from one semiotic resource to another, are fundamentally different because, rather than mapping one experiential domain onto another, these metaphors involve a grammatical reconfiguration which changes the essence of how experience is organized, as explained below.

Before investigating the integration of natural language, images, and symbolism in mathematics and the metaphorical transformations that result, the three resources are considered individually in order to examine their semiotic functions. As the discussion is concerned with how mathematics restructures human experience, the focus is directed toward experiential meaning (i.e., happenings in the form of entities, processes, and circumstances) and logical meaning (i.e., the connections between those happenings). The nature of interpersonal meaning (i.e., social relations and truth-value) and textual meaning (i.e., the organization of the message) in mathematics is also briefly considered.

3 Linguistic Formulations of Human Experience

As Halliday [24, p. xvii] explains, “the grammar of every language contains a theory of human experience; it categorizes the element of our experience into basic phenomenal types, construing these into configurations of various kinds, and these configurations in turn into logical sequences.” Thus, by organizing human experience in such a manner, the words and grammar of everyday discourse are used to construe happenings in a variety of ways, using the available options in the language system. For example, William Golding [16] uses language to create the world views of two early human species in The Inheritors, a novel concerned with the extinction of one of the last Neanderthal tribes by homo sapiens. In investigating the linguistic patterns in the novel, Halliday [20] focuses on the grammar of transitivity (i.e., process types, participants and circumstance, and active and passive voice) to show how Golding uses these systems to characterize the relatively simple Neanderthal world—a world dominated by material actions which have little or no impact on the environment, unclassified objects which lack basic description, and an absence of reasoning about unfolding events. “Thus the picture is one in which people act, but they do not act on things; they move, but they move only themselves not other objects” [20, p. 108]. As Halliday observes, even verbs that typically involve an impact on some other object (e.g., “grab”) invoke a sense of hopelessness and futility in the Neanderthal world (e.g., “he threw himself forward and grabbed at the branches with hands and feet”). As Lukin [37, pp. 362–363] explains: “The ‘syntactic tension expresses this combination of activity and helplessness’, a world view in which ‘there is no cause and effect’, where ‘people do not bring about events in which anything other than themselves, or parts of their bodies, are implicated’, where ‘people do not act on the things around them; they act within the limitations imposed by the things’ [20, pp. 109, 113, 114].”

In the final section of Golding’s novel the world view shifts to the relatively more sophisticated world of early homo sapiens, where the horizons of early humans are broadened beyond the confines of the immediate context. “This transformation, as Halliday shows, [is] encoded lexicogrammatically; … transitive structures predominate, agency belongs to humans not inanimate things, humans intentionally act on external objects, their actions are more varied, and they produce results, and things in the world are increasingly taxonomized” [37, p. 363]. In this way, readers are able to recognize the beginnings of human social life: for example, religion, war, and control of the physical environment.

Natural language is also used to create a scientific view of the world; a world that differs dramatically from the early human worlds described by Golding. For example, in a study of the emergence of scientific writing from the time of Chaucer’s discussion of the astrolabe in the 14th century and the origins of modern science in the writings of Newton, Priestley, Dalton, Darwin and Maxwell in the 17–19th centuries. Halliday [26] demonstrates how a major semantic shift from a commonsense view of the world to an abstract scientific view took place. That is, everyday experience as a series of events or happenings was gradually replaced with a virtual world of cause and effect. Linguistically, the abstract scientific view was constructed by recoding processes (i.e., verbs) as metaphorical (virtual) entities (i.e., nouns), and logical connections (i.e., conjunctions) as causal processes (i.e., verbs), in order to relate the virtual entities to each other. For example, ‘a interacts with b, thus transforming z’ is recoded as ‘the interaction of a and b results in the transformation of z,’, so that: (a) the processes ‘interacts’ and ‘transforms’ become the virtual entities ‘interaction’ and ‘transformation’; and (b) the conjunction ‘thus’ is recoded as the causal processes ‘results in.’. As Halliday [26] explains, scientific writers from Newton onwards favored such metaphorical modes of expression “– one in which, instead of writing ‘this happened, so that happened’, they write ‘this event caused that event’” [26, p. 174] because their aim was to construct a logical argument:

Newton and his successors were creating a new variety of English for a new kind of knowledge: a kind in which experiments were carried out; general principles were derived by reasoning from these experiments, with the aid of mathematics; and these principles in turn tested by further experiments. The discourse had to proceed step by step … [and] the most effective way of doing this, in English grammar, is to construct the whole step as a single clause, with the two parts turned into nouns, one at the beginning and one at the end, and a verb in between saying how the second follows from the first [original emphasis] [26, p. 174].

Halliday [25] uses the term ‘grammatical metaphor’ to describe the semantic shift whereby processes, logical conjunctions, and other grammatical elements (e.g., qualities, circumstances) are reconstrued metaphorically to transform the commonsense world of actions and events into an abstract semiotic world specifically designed for thinking and reasoning: “Grammatical metaphor creates virtual phenomena – virtual entities, virtual processes – which exist solely on the semiotic plane; this makes them extremely powerful abstract tools for thinking with. Thus what grammatical metaphor does is to increase the power that a language has for theorizing” [original emphasis] [24, p. xvii]. Over time, increasing amounts of information were encoded into noun group structures to condense and propel logical thinking even further; for example, the grammatical metaphors in the titles of recent articles in Nature (Volume 521, Number 7552, May 2015) include: “counteraction of antibiotic production and degradation,” “the crystallography of correlated disorder,” and “spin–motion entanglement and state diagnosis with squeezed oscillator wavepackets,” which are typical of the extended noun structures found in scientific writing. Scientific language is the “product of conscious design,” where “grammatical metaphor reconstrued the human environment, transforming the commonsense picture of the world into one that imposed regularities on experience and brought the environment more within our power to control. … It is presented most clearly in the discourse of the natural sciences, which is where it evolved” [24, p. xvii]. While scientific language also introduced a large number of technical terms, the change was more profound than a new vocabulary; that is, the underlying grammatical change, whereby virtual entities are logically related to each other, transformed our view of the world.

Newton and his successors based their science on mathematics, however, the tool through which their results and findings were derived. Fundamentally, mathematical symbolism restructures the world in very different ways to scientific language, as discussed below.

4 The Symbolic Creation of a World View

Diophantus, looking over the clutter of words and numbers in algebraic problems, saw that an abstraction could be a great simplification. And so, Diophantus took the first step toward introducing symbolism into algebra [29, p. XIII].

Histories of mathematics reveal that mathematical symbolism evolved from rhetorical forms which were linguistic in nature, to syncopated forms which contained linguistic elements and some symbols, to the modern symbolic system found today (e.g., [5]). Although it would seem logical to have introduced symbols for arithmetic operations after symbols were created for entities and numbers in early mathematical systems, this step was not always taken due to the existence of mechanical calculating devices (e.g., counting rods, pebbles, counting boards, and so forth) (e.g. [9]). While most calculating devices were symbolic in nature to some degree (e.g., the abacus), they operated across different modes, including three-dimensional space, action, and movement, making it difficult to develop a unified symbolic system for mathematics. Eventually the Hindu-Arabic number system was fully adopted in 16th century Europe (see discussion of algorists versus abacists in [44, 45]), leading to written forms of arithmetic algorithms and the later development of modern mathematical notation. Historians note that mathematical notation was the subject of much debate and controversy, and the result was often a matter of politics rather than the usefulness of the symbol itself [7].

Even though the symbolism evolved from linguistic forms, the grammatical systems which evolved in modern mathematical notation differ from those found in natural language. There were several reasons for the departure from the grammatical systems of language, and the forms of scientific writing which later evolved. First, language operates in both written and spoken modes, and so possesses graphological and phonological systems for encoding meaning via words and sounds. However, modern mathematical notation developed in written form, so new visual-based grammatical strategies could be developed (e.g., use of spatial notation, lines, brackets, textual layout, and so forth). The invention of the printing press in the 15th century permitted these new grammatical systems to be standardized and widely circulated. Indeed, the printing press was a major factor in the scientific revolution because, as Eisenstein [14] points out, Newton (for example) taught himself mathematics from books he bought or borrowed: “At least in my view the changes wrought by printing provide the most plausible point of departure for explaining how confidence shifted from divine revelation to mathematical reasoning and man-made maps” [14, p. 701]. The printing press, and the subsequent shift to standardized forms of symbolic notation, paved the way for the development of mathematical symbolic notation as a semiotic tool developed for written (and today digital) modes of communication.

Second, language and mathematical symbolism, as designed semiotic systems, evolved to fulfill different purposes. Scientific writing was designed to develop a logical argument about scientific findings, hence the creation of virtual entities which could be used to reason about a world of cause and effect. Mathematical symbolism, however, provided the results upon which those arguments rested, and so the functions and subsequent nature of this semiotic tool are necessarily different to language. In fact, it can be argued that scientific language evolved in the manner described by Halliday [25] due to the intellectual advances and the remodeling of the world made possible through mathematics. This claim is explored below by examining mathematical notation as a grammatical system for encoding meaning, which may be compared to the linguistic system from which it evolved.

In language, a limited number of key entities are configured around a single process (often with circumstantial information), which in turn is logically related to a similar configuration, so that happenings unfold in sequence, one after the other. As discussed earlier, scientific writing partially overcomes the limitations of the serial encoding of happenings by creating virtual entities and processes so logical thinking can be intensified. However, despite the recoding of reality achieved through grammatical metaphor, the restrictions of the serial encoding of happenings remains in linguistic descriptions of experience. While mathematical symbolic statements also unfold in a linear (logical) fashion, happenings are encoded in each statement as multiple configurations of generalized entities and processes which interact different ways that are not necessarily serial in nature. That is, mathematics organizes experience in terms of the relations between multiple (generalized) entities interacting with each other via multiple (generalized) processes at any given instance. The configurations of entity and process interactions are encoded in an economical and unambiguous fashion, using deep levels of embedding and other grammatical strategies (e.g., positional notation, spatial notation, specialized symbols, rule of order for operations, ellipsis, use of brackets, and so forth), so that the symbolic configurations are easily rearranged to derive results.

For example, the equation ${x}^3+\frac{xy}{3}=\frac{y^2}{4}-{x}^2y+2\left({y}^2+3\right)$ involves configurations of entities (variables x and y and integers 2, 3, and 4) and processes (addition, subtraction, multiplication, and division), which are economically encoded using exponents, lines for division, brackets, and ellipsis of the multiplication sign. Using such grammatical resources, mathematical symbolism creates a dynamic representation of multiple relations in a format that is easy to comprehend and manipulate; that is, the symbolism is specifically designed for logical thinking. Scientific language is dense and static (given the amount of information encoded in the virtual entities and processes) compared to the fluidity and intricacy of mathematical symbolism, but unlike spoken language which Halliday [18] characterizes in similar terms, the encoding of multiple generalized entities and processes in the symbolism extends its meaning potential for capturing patterns and relations beyond the capabilities of language. As such, mathematics is designed to manage the complexity of the universe, as the ultimate tool for mapping and thinking about patterns and relations.

In summary, language imposes a certain way of experiencing reality, where key entities are identified and assigned roles within a semantic configuration which features a single process, so the world is constructed as a series of happenings which unfold in a linear sequence. Mathematical symbolism has no such restrictions with regard to ordering experience; rather, the world consists of many interacting entities and processes, forming relations which can be reconfigured in order to establish results. Moreover, mathematics is a ‘hierarchical’ (i.e., cumulative) knowledge structure [3, 4], where knowledge is built upon earlier results. In this world, the focus is the experiential and logical domains, aided by textual resources (e.g., spatial layout) that organize mathematical knowledge in very specific ways. In order to foreground the experiential and logical domains in mathematics, the interplay of social relations is curtailed by holding interpersonal meaning and truth-value (e.g., possibility, probability, obligation, and potentiality) within limited spheres of activity. That is, the interpersonal realm is subdued (or constrained) in mathematics, in order to focus on the experiential and logical domains. Indeed, unlike language, mathematics is not designed to map the human social world where interpersonal meaning is often the primary consideration, along with other domains of experience (e.g., sensing, feeling, behaving, and so forth).

Mathematical symbolism and language are designed for different purposes, and accordingly, they have different grammatical systems and strategies for managing complexity. One key aspect of mathematical symbolism is the direct link to mathematical images, which creates a semantic circuit between language, symbolism, and image—the defining feature of mathematics for reformulating reality. In what follows, mathematical images are considered, before examining mathematics as an integrated form of multimodal semiosis.

5 The Visual Creation of Reality

Seeing comes before words. The child looks and recognizes before it can speak. But there is also another sense in which seeing comes before words. It is seeing which establishes our place in the surrounding world; we explain that world with words, but words can never undo the fact that we are surrounded by it. The relation between what we see and what we know is never settled [2, front cover].

Images, along with natural language and other semiotic resources, function to structure thought and reality. That is, in much the same way that writers are able to use linguistic resources to create different world views, painters, photographers, illustrators, and others use visual resources to structure the world in alternative ways. For example, Berger [2] considers paintings of female nudes (also advertising images) in western culture and shows how the images differ in terms of the choices which the painters make with respect to facial expression, gaze, body posture, skin texture, clothes, use of objects (e.g., mirrors), the immediate environment, and so forth. Berger [2] demonstrates that, with the exception of a small number of grand masters who depict the woman as herself, the paintings perpetuate cultural perceptions about women as objects of male gaze and idealization. In doing so, Berger [2, p. 54] explains the difference between nakedness and nudity: “To be naked is to be oneself. To be nude is to be seen naked by others and yet not recognized for oneself. A naked body has to be seen as an object in order to become a nude”; that is, “nudity is a form of dress” [2, p. 54]. The difference between nudity and nakedness then is how the painter (or photographer, artist or image creator) chooses to portray the woman, selecting from a range of available options.

Images function to create the world around us, rather than reflect it, as Berger [2] demonstrates in his study of paintings and other visual genres. In this regard images are semiotic resources which are used to make experiential, logical, interpersonal, and textual meanings (e.g., [1, 33, 43]). As such, images may also be conceptualized as system and text, where the system is the meaning potential (e.g., participants, objects, visual processes and circumstances, and so forth) and the text is the actual image itself (e.g., the painting, with participants and their gaze, facial expression, and so forth), constructed through choices from systems, which constitute the visual grammar. As for language and other resources, visual systems can be organized according to a ranked scale; for example, figure (e.g., a woman) and associated components (her face, hands, and body), the episodes in the image (what she is doing, and what others are doing), and the entire work (the picture itself) [43].

Images have a unique meaning potential, which is related to the way in which we perceive world as a series of parts, all of which are related to the whole, following Gestalt theory [31]. In particular, mathematical images (e.g., graphs, diagrams, and other visualizations) are powerful resources for reasoning because they permit mathematical entities, processes, and circumstances to be viewed in relation to each other, offering a powerful lens to understand patterns and relations. Despite the potential of the visual image for understanding mathematical relations, this semiotic resource has traditionally occupied a secondary place compared to the symbolism which has been privileged as the most significant resource for deriving mathematical results [10]. In what follows, the pivotal contributions of mathematical images are considered, before turning to mathematics as a multimodal construction.

The history of the semiotics of mathematics discourse (e.g., [40]) reveals how the human participants and the context of situation and culture (e.g., everyday activities, machines, human figures) were gradually removed from mathematical images, so the focus became the lines, the curves, and the mathematical objects themselves. Descartes’ [13] analytic geometry (see Figure 1), where algebra and geometry became united, was a major turning point for the shift from the everyday world to the abstract world of mathematics so clearly observable in Newton’s writings. For the first time, algebraic equations could be viewed using a visual coordinate system and geometrical shapes could be described using algebra, opening up the potential of the visual for viewing patterns which were encoded symbolically and vice versa, as depicted in Figure 1 where the focus shifts back and forth between the diagram and symbolism constantly. Images play an important role in mathematics, because the parts are seen in relation to the entire mathematical construct, opening up new avenues of reasoning. Significantly, mathematical images also relate to our lived-in sensory experience, providing a bridge from perceptual understanding of the world to the abstract semiotic realm of mathematics, which in turn lead to further abstractions, given the close links between the images and the mathematical notation.

Figure 1

Descartes [13, pp. 234–235], with Facsimile of First Edition (left)

Given that the focus of the visual representations is mathematical symbolic entities, processes, and relations, the mathematical image is also a purpose-built semiotic tool for encoding meanings that are unambiguous and precise. As there are various ways of framing mathematical abstractions (e.g., scale, color, perspective, and so forth), the creative tension which Davis [11] mentions is also generated through visual representations: for example, Tufte [47, 48] shows how images are used to visualize information and quantitative data and to provide explanations using visual systems of reasoning. Today, dynamic and three-dimensional visualizations of patterns and relations across space and time offer new opportunities for exploring mathematical patterns and relations. However, another major source of creative tension in mathematics arises from shifts between textual (linguistic and symbolic) and visual forms of semiosis [35, 36, 40–42], as discussed in the following section.

6 Multimodal Semiotics: Language, Images, and Symbolism

As Hawking [29, p. XI] explains, revolutions in the way humans have perceived the world have always gone hand in hand with revolutions in mathematical thought: for example, “Isaac Newton could never have formulated his laws without the analytic geometry of Rene Descartes and Newton’s own invention of calculus.” Descartes [13] and Newton [38] derive their results in a running text format, constantly moving back and forth between language, geometrical diagrams, and symbolism as the need arises (e.g., see Figure 1). This format is the precursor to modern mathematics, where symbolic expressions and equations are set apart from the remainder of the text. Regardless of format, the intent is clear—to reason logically in the most effective way possible, drawing upon linguistic, visual, and symbolic resources as required.

In order to investigate how and why such transitions between the three resources take place, the semantic expansions which occur as a result are explored, based on the key concepts of meaning potential and semiotic metaphor. As explained earlier, meaning potential is the ability of a resource to create meaning arising from its architecture; that is, the grammatical systems from which selections are made. Semiotic metaphor is defined as a shift in the functional status of a choice (e.g., entity, process, and entity/process configuration) when that element is resemiotized into another semiotic form; for instance, when linguistic choices are visualized, or visual choices are symbolized (e.g., [41]). Semiotic metaphor also includes the introduction of new metaphorical entities and processes which occur in the transition from one semiotic resource to another. Semiotic metaphor is based on Halliday’s [25] concept of grammatical metaphor in language (see above), but in this case, the semantic realignment takes place across different semiotic resources. In examining such metaphorical constructions, it is apparent that inter-semiotic transitions give rise to semantic expansions that extend beyond those possible with language or any single semiotic resource.

For example, the general formula for the distance between any two given points P(x ₁, y ₁) and Q(x ₂, y ₂) is given by $PQ=\sqrt{{\left({x}_2-{x}_1\right)}^2+{\left({y}_2-{y}_1\right)}^2}$ in analytic geometry. The semantic expansions of meaning that take place across language, image, and symbolism in the derivation of this formula (e.g., Figure 2) are discussed below.

Figure 2

Distance in Cartesian Coordinate Plane [46, p. 91]

Initially, the first steps for deriving the distance formula are described linguistically, given the potential of language to organize everyday experience in terms of different process types; in this case, mental processes (“consider”) and material actions (“completing”) and possession (“have”). These linguistic instructions (with symbolic participants) result in a transition to the mathematical image, where “distance” is resemiotized as the line segment connecting P(x ₁, y ₁) and Q(x ₂, y ₂) and the (virtual) right-angled triangle PQR with coordinates of R (x ₂, y ₁) is drawn. In addition to accessing the meaning potential of the image, where the parts of the right-angled triangle are perceived in relation to the whole, the vertices and sides of the triangle are exactly describable using the xy coordinate system. That is, the problem is reconstrued visually as parts (i.e., vertices, sides) related to the whole (i.e., the right triangle) within a symbolic coordinate system. The length of the line segments PR and QR are resemiotized as configurations of mathematical entities and processes (x ₂ – x ₁) and (y ₂ – y₁). From there, there is a shift to the mathematical symbolism, where the formula $PQ=\sqrt{{\left({x}_2-{x}_1\right)}^2+{\left({y}_2-{y}_1\right)}^2}$ is derived using the Pythagorean theorem.

It is apparent that the shift from language to image to symbolism permits the meaning potentials of the three resources to be accessed, with a move from everyday reality to the abstract semiotic realm of mathematics. Beyond this, the transitions mean that linguistic elements (e.g., “distance”) are reconstrued visually (i.e., line segment in visual image). This in turn permits the introduction of a new entity (the right triangle), where the length of the sides and the relations between the length of the sides can be perceived and expressed using mathematical symbolism. The semiotic metaphors which occur include (a) the introduction of new visual entities, i.e., triangle PQR and its component parts (vertices and sides); and (b) the lengths PR and QR as configurations of mathematical entities and processes, i.e., (x ₂ – x ₁) and (y ₂ – y ₁); and (c) the relations between the lengths of the sides as a configuration of mathematical entities and processes, i.e., $P{Q}^2={\left({x}_2-{x}_1\right)}^2+{\left({y}_2-{y}_1\right)}^2$ . That is, the problem is resolved via the semantic expansions involving the introduction of new visual entities (the triangle and its associated parts) and the reconstrual of the visual elements (the sides of the triangle and their relations) as mathematical symbolic configurations. Indeed, mathematics is the result of shifts across linguistic, visual, and symbolic resources where experience is restructured by reconfiguring semiotic phenomena in new ways and introducing new semiotic phenomena that did not previously exist.

The study of the semiotic transitions between language, image, and symbolism in mathematics also sheds light on the functions and roles of the three resources and their resulting grammatical strategies for encoding meaning. For example, information is packed into the noun group structure in “the general formula for the distance between any two given points P(x ₁, y ₁) and Q(x ₂, y ₂).” This may be compared to the symbolic formula “ $PQ=\sqrt{{\left({x}_2-{x}_1\right)}^2+{\left({y}_2-{y}_1\right)}^2}$ ” which is a configuration of mathematical entities and processes. That is, scientific language has extended noun group structures, but the symbolism retains a dynamic formulation of happenings in order to derive results, as demonstrated in this example (also Figure 1). In other words, language provides a bridge from everyday experience to the abstract semiotic plane, but the image and symbolism permit thinking to extend beyond linguistic semiotic boundaries, basically to rewrite human experience of the universe.

7 Conclusions

“Can there be knowledge without words, without symbols [and images]?” [12, p. 44]

As Davis et al. [12, p. 44] point out, mathematics has been a human activity for thousands of years, and most cultures developed some form of mathematical activity for investigating quantity, space, and patterns. Hawking [29, p. XI] reflects how mathematics has been responsible for great insights into nature, “such as the realization that the earth is round, that the same force that causes an apple to fall here on earth is also responsible for motion of heavenly bodies, that space is finite and not eternal, that time and space are intertwined and warped by matter and energy, and that the future can only be determined probabilistically.” In order to make these advances, mathematics developed specific ‘tools of the trade’ [12], giving rise to semiotic metaphor as the ultimate form of multimodal metaphor for meaning expansion.

But mathematics is more than a tool and language for understanding the universe. As Hawking [29, pp. XII–XIII] and others point out, mathematics has affected our world view in its own right, in terms of introducing concepts such as infinity, continuous functions, logic as a system of processes, the power and limits of digital computing, and unproven laws and logical consistency of systems. Beyond reformulating the world and introducing new mathematical concepts, the types of metaphorical construal developed in scientific language through grammatical metaphor is found in most discourses of modern life, from politics and economics to social life. Indeed today, our everyday view of the world is largely a metaphorical construct, originating from the need to organize the material universe. Lastly, computers are having a major social impact on how we live and communicate: “From the point of view of social impact, the major mathematical breakthrough since the end of World War II is the digital computer in all its ramifications. This breakthrough, involved a combination of mathematics and electronic technology” [11, p. xxiii].

Despite the widespread impact of mathematics and science on human life today, a division still exists between sciences and the humanities, where scientists generally understand more about the humanities than vice versa. As Davis et al. [12, p. 54] explain, “part of the reason for this lies in the fact that the locus of the humanities is to be found in sound, vision, and common language. The language of science with its substantial sublanguage of mathematics poses a formidable barrier to the humanist.” However, the major challenges in our globally connected world of today are social, political, and economic in nature (e.g., poverty, healthcare, social, and cultural unrest). In order to address these issues, it would seem that the sciences, mathematics, and the humanities must necessarily unite to develop new semiotic tools which permit the human realm to be understood. This could be achieved by harnessing the potential of digital technology, the derivative of mathematics and classical science, combined with the insights from humanities and the wealth of knowledge about human life which exists today, as the product of digital technology (e.g., Wikipedia and other structured knowledge systems about human life). Indeed, it is difficult to imagine how big problems in the world today can be resolved, unless the focus on the experiential and logical domain is expanded to include the interpersonal domain, and this necessarily involves bringing together science, mathematics, and the humanities, aided by digital technology. From there, it may be possible to understand human and physical worlds as an integrated system.

As Halliday [19, p. 128] claims, “[i]f the human mind can achieve this remarkable combination of incisive penetration and positive indeterminacy, then we can hardly deny these same properties to human language, since language is the very system by which they are developed, stored and powered.” However, if we add multimodal mathematics and science to human language, and explore the potential of new digital semiotic tools, then we may be a step closer to understanding and preserving human life and the earth itself.

References

Bateman, J. (2014). Text and Image: A Critical Introduction to the Visual/Verbal Divide. London & New York: Routledge.

Berger, J. (1972). Ways of Seeing. London: BBC and Penguin Books.

Bernstein, B. (1999). Vertical and Horizontal Discourse: An Essay. British Journal of Sociology of Education, 20(2), 157-173.CrossRef

Bernstein, B. (2000). Pedagogy, Symbolic Control and Identity: Theory, Research and Critique (revised ed.). Lanham: Rowan & Littlefield.

Boyer, C. B., & Merzbach, U. C. (2011). A History of Mathematics (3rd ed.). New Jersey: John Wiley & Sons.

Butt, D. (2015). The ‘History of Ideas’ and Halliday’s Natural Science of Meaning. In J. Webster (Ed.), The Bloomsbury Companion to M. A. K. Halliday (pp. 17-61). London: Bloomsbury.

Cajori, F. (1993). A History of Mathematical Notations (Two Volumes). New York: Dover Publications [first published 1928 and 1929].

Cobley, P. (Ed.). (2010). The Routledge Companion to Semiotics. London & New York: Routledge.

Cooke, R. (2005). The History of Mathematics: A Brief Course. New Jersey: John Wiley & Sons.CrossRef

10.

Davis, P. J. (1974). Visual Geometry, Computer Graphics and Theorems of Perceived Type. Proceedings of Symposia in Applied Mathematics, 20, 113-127.

11.

Davis, P. J. (2006). Mathematics and Commonsense: A Case of Creative Tension. Wellesley MA: A. K. Peters Ltd.CrossRef

12.

Davis, P. J., Hersh, R., & Marchisotto, E. A. (2012). The Mathematical Experience, Study Edition. Boston: Birkhäuser Boston

13.

Descartes, R. (1954 [1637]). The Geometry of Rene Descartes (D. E. Smith & M. L. Latham, Trans. 1637 ed.). New York: Dover.

14.

Eisenstein, E. L. (1979). The Printing Press as an Agent of Change. Cambridge: Cambridge University Press.

15.

Forceville, C. J., & Urios-Aparisi, E. (Eds.). (2009). Multimodal Metaphor. Berlin & New York: Mouton de Gruyter.

16.

Golding, W. (1955). The Inheritors. London: Faber and Faber.

17.

Halliday, M. A. K. (1978). Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Edward Arnold.

18.

Halliday, M. A. K. (1985). Spoken and Written Language. Geelong, Victoria: Deakin University Press [Republished by Oxford University Press 1989].

19.

Halliday, M. A. K. (1993). Language and the Order of Nature. In M. A. K. Halliday & J. R. Martin (Eds.), Writing Science: Literacy and Discursive Power (pp. 106-123). London: The Falmer Press.

20.

Halliday, M. A. K. (2002 [1971]). Linguistic Function and Literary Style: An Inquiry into the Language of William Golding’s The Inheritors. In J. Webster (Ed.), Linguistic Studies of Text and Discourse: Collected Works of M. A. K. Halliday (Volume 2) (pp. 88-125). London: Bloomsbury.

21.

Halliday, M. A. K. (2003a). Introduction: On the “architecture” of Human Language On Language and Linguistics.: Volume 3 in the Collected Works of M.A.K. Halliday. (pp. 1-29). London and New York: Continuum.

22.

Halliday, M. A. K. (2003b). On the ‘Architecture’ of Human Language. In J. Webster (Ed.), On Language and Linguistics: The Collected Works of M.A.K. Halliday (Volume 3) (pp. 1-31). London and New York: Continuum.

23.

Halliday, M. A. K. (2003 [1985]). Systemic Background. In J. Webster (Ed.), On Language and Linguistics: The Collected Works of M. A. K. Halliday Volume 3 (pp. 185-198). London and New York: Continuum.

24.

Halliday, M. A. K. (2004). Introduction: How Big is a Language? On the Power of Language. In J. Webster (Ed.), The Language of Science: Collected Works of M. A. K. Halliday (Volume 5) (pp. xi-xxiv).

25.

Halliday, M. A. K. (2006). The Language of Science: Collected Works of M. A. K. Halliday (Volume 5). London & New York: Continuum.

26.

Halliday, M. A. K. (2006 [1993]). Some Grammatical Problems in Scientific English. In M. A. K. Halliday & J. R. Martin (Eds.), Writing science: Literacy and Discursive Power (pp. 159-180). London: Falmer Press.

27.

Halliday, M. A. K. (2008). Complementarities in Language. Beijing: Commercial Press.

28.

Halliday, M. A. K., & Matthiessen, C. M. I. M. (2014). Halliday’s Introduction to Functional Grammar. London & New York: Routledge.

29.

Hawking, S. (Ed.). (2005). God Created the Integers: The Mathematical Breakthroughs that Changed History. London: Running Press.

30.

Hodge, R., & Kress, G. (1988). Social Semiotics. Cambridge: Polity Press.

31.

Hothersall, D. (2004). History of Psychology. New York: McGraw-Hill.

32.

Jewitt, C. (Ed.). (2014). The Routledge Handbook of Multimodal Analysis (2nd ed.). London: Routledge.

33.

Kress, G., & van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd ed.). London: Routledge.

34.

Lakoff, G., & Johnson, M. (1980). Metaphors We Live By. Chicago: The University of Chicago Press.

35.

Lemke, J. L. (1998). Multiplying Meaning: Visual and Verbal Semiotics in Scientific Text. In J. R. Martin & R. Veel (Eds.), Reading Science: Critical and Functional Perspectives on Discourses of Science (pp. 87-113). London: Routledge.

36.

Lemke, J. L. (2003). Mathematics in the Middle: Measure, Picture, Gesture, Sign, and Word. In M. Anderson, A. Sáenz-Ludlow, S. Zellweger & V. V. Cifarelli (Eds.), Educational Perspectives on Mathematics as Semiosis: From Thinking to Interpreting to Knowing. (pp. 215-234). Ottawa: Legas.

37.

Lukin, A. (2015). A Linguistics of Style: Halliday on Literature. In J. Webster (Ed.), The Bloomsbury Companion to M. A. K. Halliday. London: Bloomsbury.

38.

Newton, I. (1729). The Mathematical Principles of Natural Philosophy (A. Motte, Trans.). London: Printed for Benjamin Motte.

39.

Nöth, W. (1990). Handbook of Semiotics. Indianapolis: Indiana University Press.

40.

O’Halloran, K. L. (2005). Mathematical Discourse: Language, Symbolism and Visual Images. London and New York: Continuum.

41.

O’Halloran, K. L. (2008). Inter-Semiotic Expansion of Experiential Meaning: Hierarchical Scales and Metaphor in Mathematics Discourse. In C. Jones & E. Ventola (Eds.), New Developments in the Study of Ideational Meaning: From Language to Multimodality (pp. 231-254). London: Equinox.

42.

O’Halloran, K. L. (2011). The Semantic Hyperspace: Accumulating Mathematical Knowledge across Semiotic Resources and Modes. In F. Christie & K. Maton (Eds.), Disciplinarity: Functional Linguistic and Sociological Perspectives (pp. 217-236). London & New York: Continuum.

43.

O’Toole, M. (2011). The Language of Displayed Art (2nd ed.). London & New York: Routledge.

44.

Stone, W. E. (1972). Abacists Versus Algorists. Journal of Accounting Research, 10(2), 345-350.CrossRef

45.

Swetz, F. J. (1987). Capitalism and Arithmetic. La Salle, Illinois: Open Court.

46.

Teh, K. S., & Looi, C. K. (2001). New Syllabus Mathematics 3. Singapore: Shinglee Publishers Pte Ltd.

47.

Tufte, E. R. (1997). Visual Explanations: Images and Quantities, Evidence and Narrative. Cheshire, Connecticut: Graphics Press.MATH

48.

Tufte, E. R. (2001). The Visual Display of Quantitative Information. Cheshire, Connecticut: Graphics Press.

49.

van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.

50.

Whorf, B. L. (1956). Language, Thought and Reality: Selected Writings of Benjamin Lee Whorf. MA: MIT Press.