The multimedia principles researched by educational psychologists today have direct links with the principles of film editing investigated by the semioticians of the 20th century, who saw film as a language. Both are interested in how people construct meaning. This session identifies important continuities and relates them to instructional design.
By mid-2020, most anthropology and other social sciences fieldwork involving live interaction was suspended. Video essays, visual anthropology, and digital ethnography were already well-known methods of inquiry and engagement, but with the forced pandemic pause in traditional forms of teaching and research, they acquired a renewed status. Michael Wesch, a cultural anthropologist interested in how media ecologies affected human interaction, as well as learning, had been producing videos for some time. However, when transitioning to 100% remote teaching, he discovered that his videos were getting much less traction with students than he had hoped for, and he turned to viewership analytics for clues about why that was. In a way, through trial and error, he discovered that better videos required what Instructional Designers already know as Mayer’s multimedia principles (Mayer 2016).
Richard E. Mayer and other cognitive researchers argue that people learn more deeply from words and pictures than from each modality alone (Mayer and Moreno 2005, Plas et al. 2010, Mayer and Colvin Clark 2016). By breaking down multimedia texts into learning units sorted by categories such as “decorative, representational, organizational, and explanative” educational psychologists assign value (in terms of cognition) to the different audiovisual components. The semioticians who studied film as a language in the 20th century attempted in a similar way to decipher the grammar of media and to assess the value (in terms of meaning) of audiovisual components. On a macro scale, they investigated how cultural meanings were encoded by media texts. With motion pictures and sound, the semioticians were specifically interested in how the juxtaposition, conflict, and reassembling of images conveyed specific ideas from author to viewer. They referred to this as the montage principle (Huttunen 2013, Lefebvre 2017).
Culture and Cognition belong to separate academic streams of inquiry, but they are inextricably linked in the everyday experience of instructional designers and educators. What the first group refers to as cognitive value, and the second as cultural meaning, is the essence of what both groups expect that viewers, or learners, will take away from the multimedia text. Both the multimedia principle and the montage principle are concerned with explaining human perception and the construction of meaning through media, and scholars have generated a large body of work within their respective disciplinary bounds.
The focus on how and what humans learn is part of longer processes of adaptation, which new technologies do not reset with each upgrade, but which they modulate in powerful ways. Acknowledging those common connections affords a more sophisticated understanding of the history of ideas, and enables building on top of, rather than rediscovering that which already exists, although rediscovery is itself a critical part of the learning process. By drawing on methodologies from cultural anthropology and enactive ethnography (Wacquant 2015), whereby the researcher is embedded into the social and symbolic world of instructional design, I make connections between the fields of multimedia learning, media semiotics, and instructional design, and present some recent work. With the multimedia/montage principles as a starting point, this presentation touches on common discoveries about attention that surface across disciplines, particularly in the construction of narratives through video and sound editing. The presentation demonstrates that besides the representational aspect of images and text, there is also a signification aspect to how they are presented, in what sequence, with what “beats,” which has direct implications for the construction of meaning, intrinsic motivation, subject interest, retention and transfer of attention, and cognitive load.
Goals:
Following this interactive discovery session, participants will be able to make interdisciplinary connections that they can apply to their own work in instructional design, and more broadly in their work with multimedia. They will be able to identify certain mechanics from our extensive visual culture vocabulary, which will assist them in shaping their own audiovisual narratives. This is early work, so the author invites further discussion about interdisciplinary work in education, media, and online learning.