Challenge of Applying Language Models to Visual Genres

This week’s reading is about linguistic theories and the potential of applying language models to other nonlinguistic semiotic systems. As you all know, I did research on building film cognition models based on Parallel Architecture. I re-read the materials again for this week and in this post I want to think through the challenge of applying linguistic model to visual genres. One big challenge laying in front of me is that after realizing the parallel cognitive model of visual images or films: How can we go further on top of this realization to do more valuable research?

In film studies, there’s a branch that focuses on film narration structure since 1960s. The mainstream film narrative analysis is to look film as a structured language. Based on Chomsky’s generative grammar, they build multiple theories based on the methodology called film semiotics. Within this approach, scholars try to synthesize possible narrative structures of film generally. Figure 1 is a famous film narrative model called grande syntagmatique, built by Christian Metz. He believes that based on these film specific syntax, we generate stories and meanings, which echoes Chomsky’s generative grammar theory that the sentence and meaning/semantics is produced on the based of grammar/syntax.


Figure 1

But after this week’s reading, we know that meanings are in networks (Figure 2.) Syntax, however, is not the presumption of meaning. Same for film meaning construction. The process of encoding and decoding of meaning in film is embedded in larger cultural context and the syntax (narrative structure), semantics, sound and other elements work together at the same time.


Figure 2

However, I think the big challenge for me so far is that after realizing the model of film meaning construction, it is hard to go further. There are indeed so much potential to start with things I talked above. But how to frame further possible research? I do get lost in this sense.