By Frank Serafini, author, Beyond the Visual: An Introduction to Researching Multimodal Phenomena

Frank Serafini is professor of literacy education and children’s literature in the Mary Lou Fulton Teachers College, Arizona State University. He received the 2021 International Visual Literacy Association (IVLA) Education Award.

---

In 2010, I published an article in the educational journal Children’s Literature in Education that presented my original tripartite framework for multimodal research and analysis. This framework described three dimensions or analytical perspectives, namely the perceptual, structural, and ideological perspectives that were synthesized from a variety of theories and scholarship across the fields of semiotics, art criticism, sociology, literary criticism, and linguistics. Subsequently, this framework became the theoretical foundation for my book, Reading the Visual: An Introduction to Teaching Multimodal Literacy (Serafini, 2014).

I developed the tripartite framework based on a review of various literary, art-based, semiotic, linguistic, and sociocultural perspectives across several fields of study, including Barthes’ (1977) concepts of denotation, connotation, and myth, Panofsky’s (1955) distinctions among pre-iconographical, iconographical, and iconological perspectives, and Scholes’ three-tiered concepts of reading, interpretation, and criticism. In developing the original framework, I wanted to highlight the distinctions made between looking—a biological or perceptual process and seeing—an interpretive and social process (Berger) or the distinction between vision and visuality offered by Rose (2014).

Between the release of Reading the Visual and my new book Beyond the Visual: An Introduction to Researching Multimodal Phenomena (January 2022), I have spent time rethinking and revising my tripartite framework to reflect some changes in my analytical approaches and theoretical understandings. One important revision is a broadening of attention from the basic visual and design elements manifested in visual images and multimodal texts to the socio-cultural contexts of their production, distribution, and reception. Perception may be studied as primarily a biological process, however, as Gombrich has pointed out the innocent eye is blind, suggesting that looking is not the same as seeing. Studying the inherent qualities of a particular visual or multimodal phenomenon without also considering its structure or composition, the history of the phenomenon, and the political, social, and cultural contexts in which these phenomena are experienced seems theoretically myopic.

As stated in Reading the Visual: An Introduction to Teaching Multimodal Literacy (Serafini, 2014), the analytical dimensions or perspectives should be regarded as a set of three concentric circles, with each individual perspective (perceptual, structural, and ideological) nested within subsequent perspectives rather than conceived as a leveled hierarchy. Each perspective is not at a higher or lower level or relegated to a position of authority by its hierarchical order, rather each dimension or perspective is nested within the other dimensions to suggest an inter-animation of these dimensions rather than a developmental or hierarchical progression.

There are several additional revisions I have made to the original framework since its original publication that I believe are worthy of further consideration and explication. The first revision concerns my challenges with the word structural for the second dimension in my tripartite framework. There are connotations associated with the word structural and potential connections to structuralist theories that I did not intend when I selected this label for the second dimension. I was trying to call attention to the way things are arranged or composed, or what Kress and van Leeuwen have referred to as visual grammars rather than suggest a particular theoretical orientation.

Another revision to the original framework has to do with the ideological dimension. My goal was to reconsider this dimension not as a monolithic or homogenous dimension, rather as a way of including a range of different theoretical orientations or perspectives. For example, multimodal researchers and theorists have used critical theory, feminist theories, critical social theory, and post-humanist theories as ideological framing devices from which to conduct research on contemporary picture books. I have begun to see the three dimensions of my tripartite framework as framing devices or analytical perspectives that support particular research designs.

At this point an example might help clarify my need for changing the various dimensions or framing devices that are part of my revised tripartite analytical framework. A research project focusing on picture books from a perceptual frame might analyze a series of selected picture books to understand the role of panels and gutters in the design aspects of picture book. Drawing on narratology as an ideological or theoretical frame, the study might look at how panels and gutters help with narrative flow and structure. A research project that draws upon critical theories might look at how certain design elements are used to foreground or background issues of race, gender, or sexual orientation. From an art history frame, research on the same series of picture books might focus on how certain artistic styles and techniques are used to enhance and illustrate fictional narratives. These different framing devices can be used to call attention to or illuminate various aspects of visual and multimodal phenomena across research designs. The revisions I have outlined here are a small part of the continuing evolution of approaches to understanding visual and multimodal phenomena and the analytical frameworks used across the social sciences and divergent fields of inquiry. As my research continues to move forward, I am sure this framework will continue to evolve as well.

The revisions made to the original tripartite analytical framework demonstrates the evolution of my thinking and the changes I have made in the research designs I utilize to better understand various visual and multimodal phenomena. It is the continued renewal and questioning of which analytical approaches are most appropriate for our research studies purposes that makes our search for better ways to understand visual and multimodal phenomena so exciting and challenging.

---

Photo by Mitchell Luo from Pexels