Posted by Cristóbal Pagán Cánovas (Daedalus Director, Ramón y Cajal English Professor at the University of Murcia, Alexander von Humboldt Fellow at the University of Tübingen), Daniel Alcaraz Carrión (Séneca Foundation Postdoctoral Researcher at the Cognitive Development & Communication Lab, University of Wisconsin-Madison), and Javier Valenzuela (Daedalus Director, English Professor at the University of Murcia)
The CREATIME team of the Daedalus Lab are studying gestural patterns across large datasets of videos where people utter the same time phrases.
Time expressions are a classic case study for research into mental mappings, conceptual integration, and concept formation in general. People build temporal meanings, usually in intricate but nonetheless widely conventional ways, by reusing spatial notions, such as motion or relative position (“Saturday is approaching,” “My youth is now behind me”), interactions with objects or substances (“Give me two minutes,” “We are running out of time”), complex notions of agency (“Time is no healer: the patient is no longer here”—T.S. Eliot), and more. Interestingly, some of these patterns recur cross-culturally with practically identical structure, while there is also considerable variation in multiple aspects of the patterns, not only across cultures but also in ontology: e.g. children will need to reach a certain stage of cognitive development to make sense of clock time, time metaphors, or films flashbacks, all of which seem straightforward to young adults.
Gestures can offer very relevant information about the grounding of complex temporal meanings on relatively basic spatial notions. However, they are hard to study, because the low frequency of most linguistic expressions makes it impossible to obtain multiple examples of people uttering exactly the same time-phrase in elicited responses or controlled recordings of behavior. “Yesterday,” “earlier than,” “from start to finish…” any time expression that you can think about only occurs a few times per million words in conversational speech, and most of them have a frequency below once in a million. This means that, to match the size and variety of the current CREATIME datasets (which we expect to expand dramatically in the next months), you would need at least tens of thousands of hours of lab recordings involving thousands of different subjects.
Thanks to the new tools developed by the Red Hen Lab for the NewsScape Library of Television News, the CREATIME team of the Daedalus Lab have been building and analyzing large datasets containing multiple videos of the same time expression. We extract thousands of videos where people are acting in a rich, authentic context, uttering phrases such as “back then,” “later than,” “from beginning to end,” and so forth.
These methods are allowing us to make interesting new discoveries. For example, the use of gestures that clearly relate a spatial relation or an imaginary trajectory to a temporal meaning is the standard behavior, but it varies greatly depending on the semantics of the phrase that co-occurs with the gesture. Also, and beyond these differences, the informativity of the expression, that is, how unusual it might be in discourse and to what extent it might be adding relevant new information to what is being said, is a good predictor of the presence of gesture.
Studying a dataset of almost 2,000 videos of utterances of around 20 different time expressions in English, we found that the frequency of the phrase – a reasonable proxy for its informativity – had an inversely proportional relation to the probability of a relevant gesture being performed: less frequent expressions had an increasingly higher likelihood of a related gesture co-occurring with them.
This shows that the communicative system maximizes the use of the gestural modality in its efforts to produce satisfactory messages. When an expression is more frequent and hence more likely to be expected, the pressure to include a relevant gesture decreases significantly. Therefore, humans are not only building intricate webs integrating spatial and temporal notions: they are also adjusting their enactment during real-time performance, striving to achieve communicative success without unnecessary effort.
This dynamicity becomes even more apparent when we look at the details of time gestures. For example, gestural patterns associated to the mental timeline, which has been shown to be relevant for the spatialization of time across cultures, are enacted in very rich ways alongside conventional time expressions. In all cases, the rhetorical goals and the stylistic factors related to context, background, and individual play a role in shaping the way in which the timeline gestures are performed.
Relevant publications
-
Alcaraz-Carrión, D. & Valenzuela, J. Forthcoming. Time as space vs time as quantity in Spanish: a co-speech gesture study. Language and cognition.
-
Alcaraz Carrión, D. & Valenzuela, J. Forthcoming. Coming and going in time: a multimodal study. Language and Communication.
Alcaraz Carrion, D., & Valenzuela, J. 2021. Duration as Length Vs Amount in English and Spanish: A Corpus Study. Metaphor and Symbol 36(2), 74–84.
-
Pagán Cánovas, C., Valenzuela, J., Alcaraz Carrión, D., Olza, I., Ramscar, M. 2020. Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions. PLoS ONE 15(6): e0233892. Video