Beyond the Visual

Beyond the Visual is a survey of contemporary approaches to researching a wide range of visual and multimodal phenomena. Building on his earlier book, Reading the Visual, Serafini shares resources for conducting multimodal research across the social sciences. Beginning with a comprehensive overview of the theoretical foundations that support the analytical frameworks, the text is organized into two parts—texts and objects, events and spaces—with corresponding analytical approaches. Examples and outlines are provided to help novice and experienced researchers conduct their own studies. Vignettes by some of the most renowned scholars in the field of multimodality research take the reader behind the scenes of various projects to experience the thoughts and decisions that go into conceptualizing and applying the analytical frameworks presented in the book. This resource will enable both students and experienced scholars to acquire new research skills and designs resulting in more rigorous, high-quality research.

Book Features:

• Assists researchers and educators to make better connections among theoretical orientations, analytical frameworks, and research designs.
• Showcases 16 models for conducting research on visual and multimodal phenomena across a variety of social, virtual, and physical contexts.
• Provides examples of how eminent researchers conceive, design, initiate, and conduct their studies.
• Explores the research methods cited in the author’s previous book, Reading the Visual.

Frank Serafini is professor of literacy education and children’s literature in the Mary Lou Fulton Teachers College, Arizona State University. His books include Reading the Visual: An Introduction to Teaching Multimodal Literacy and Remixing Multiliteracies: Theory and Practice from New London to New Times. He received the 2021 International Visual Literacy Association (IVLA) Education Award.

“This book opens up the field and shows the next generation of multimodal researchers how much there already is, but also how much there is still to discover, and how much scope for innovation and creativity the field still offers.”
—From the Foreword by Theo van Leeuwen, professor, University of Southern Denmark

“Frank Serafini’s latest book is a particularly welcome addition to the growing body of textbooks on visual and multimodal communication. Through an original combination of conceptual definitions, analytical frameworks, methodological insights, and research vignettes by established researchers, Beyond the Visual offers an unusually thorough overview of social semiotic approaches to researching visual and multimodal communication. Using both accessible and precise language, this volume makes an important intervention in that it provides both students and educators with insider knowledge about the terminologies, questions, instruments, procedures, epistemologies, and problems that set apart contemporary research on multimodality.”
—Giorgia Aiello, University of Leeds and University of Bologna

“The sheer breadth of approaches to multimodality that have emerged over the past 20 years constitutes a significant challenge, particularly for newcomers to the field. Serafini's book meets this challenge head-on, presenting introductory but never simplistic descriptions of a variety of accounts, accompanied by illustrative vignettes of actual research projects written by those active in the field. Both components deserve a central place in any course introducing multimodality and multimodality research, regardless of discipline. This is precisely what the next generation of multimodality researchers now urgently requires, moving not only beyond the visual, but beyond the confinements of individual theoretical perspectives.”
—John A. Bateman, linguistics, University of Bremen

Contents

Foreword Theo van Leeuwen xi
Acknowledgments xiii
Introduction 1

PART I: FOUNDATIONS

1. Conceptualizing Visual and Multimodal Phenomena 17

2. Theoretical Foundations of Multimodal Research 32

PART II: ANALYTICAL FRAMEWORKS: TEXTS AND OBJECTS

3. Iconographical Analysis 51

4. Visual Discourse Analysis 58

5. Visual Rhetorical Analysis 65

6. Multimodal Framing Analysis 71

7. Multimodal Content Analysis 78

8. Systemic Functional Multimodal Discourse Analysis 86

9. Multimodal Genre Analysis 93

Research Vignettes: Texts and Objects 99

Research Vignette 1: Young Children’s Multimodal Compositions 99
Lindsey Moses
Research Vignette 2: Multimodal Content of Magazine Covers 102
Stephanie F. Reid, Danielle Kachorsky, and Kathryn P. Chapman

Research Vignette 3: Critical Visual Discourse Analysis of 3-D Sculpture 107
Peggy Albers

Research Vignette 4: Multimodal Content Analysis of Wine Labels 110
Frank Serafini

Research Vignette 5: Examining Wordless Picture Books 115
Evelyn Arizpe and Julie E. McAdam

Research Vignette 6: A Critical Multimodal Comparison of Animation Software 118
Emilia Djonov

Research Vignette 7: Multimodality and Orientation-to- Action in Video Games 122
Jeffrey B. Holmes, Earl Aguilera, and Kelly M. Tran

Research Vignette 8: Animated Movie Adaptations of Literary Picture Books 125
Len Unsworth

PART III: ANALYTICAL FRAMEWORKS: EVENTS AND SPACES

10. Social Semiotic Multimodal Analysis 133

11. Critical Multimodal Analysis 140

12. Mediated Discourse (Interactional) Analysis 147

13. Multimodal Ethnographic Analysis 153

14. Spatial Discourse Analysis 160

15. Multimodal Cultural Analysis 166

16. Digitally Based Multimodal Analysis 173

Research Vignettes: Events and Spaces 180

Research Vignette 9: Mapping (Visual) Identities During COVID-19 180
Jennifer Rowsell

Research Vignette 10: Critical Multimodal Analysis of Voting Spaces 183
Marva Cappello

Research Vignette 11: Multimodal Interaction Analysis of Social Positioning in Young Children at School 187
Katie Bernstein

Research Vignette 12: Understanding Spatial Pedagogy 190
Fei Victor Lim

Research Vignette 13: Collective Multimodal Research of Social Interaction in the COVID-19 Pandemic 194
Elisabetta Adami

Research Vignette 14: Analyzing Children’s Virtual Realities 197
Kathy A. Mills and Lesley Friend

Research Vignette 15: A Multimodal Analysis of Children’s Play 201
Kate Cowan and John Potter

Research Vignette 16: Spatial Discourse Analysis of Informal Outdoor Learning Spaces 204
Louise Ravelli

Epilogue: Looking Back, Moving Forward 209

References 215
Index 235
About the Author and the Contributors 245