Children were also sampled, based on teacher recommendation in communication in schools. None of the participants had any known neurological or behavior condition, and none wore glasses. The study was approved by the ethics committee of Allahabad University. Natural images were selected representing humans in two types of actions: a those depicted through transitive verbs. There were thirty images in total and ten of each type. These were photographs taken specifically for the occasion with actors representing several actions Appendix1.
These images were presented to the participants for a period of 10 seconds each. Eye movements were recorded as participants visually saw the images. Images were captured using a Sony cyber-shot 7. The outdoor images were clicked primarily for the transitive verb category. For these, the researcher went to the original settings where the actions normally took place: example, a pan shop for a photograph depicting ek admi paan laga raha hai "One man is preparing the betel leaf". All photographs were clicked from a front-on perspective; the mode of the camera was kept constant on "intensive sensitivity".
Figure 1 shows examples of each type of images and the AOIs, considered for analysis. Participants were part of the experiment in an individual way. The experimental room was dimly lit and was sound proofing. At the beginning of the experiment, after placing the head-mounted ear-phone-and-microphone on their head, they were instructed on the following issues.
First, they were instructed about basic eye-tracking, issues that they needed to know in order to participate for example, they were told to avoid moving their head out of the eye-tracker once the calibration has taken place; also, they were requested to not make too many eye blinks, and so forth. A chin rest was used to stabilize the head movements. Second, they were asked to view the photograph on display and start describing it in a single sentence as soon as possible.
Third, the sentence must necessarily be produced in Hindi, and should be the one that best describes the image that is on display. Fourth, participants were asked to keep viewing the photograph while they described it, and not shift their gaze out of the visual display. Finally, participants were instructed on practice tests, and how to take breaks between the experiment, when necessary.
Eye movements were recorded as participants viewed the images. The photographs were presented on a inch LCD screen. All sentences produced were recorded using the Goldwave software version 5. Participants generated sentences as soon as the picture was on the screen and the display went off with the sentence being recorded. The next test began after a delay of msec. There were thirty trials consisting of thirty pictures in total for each participants.
The order of presentation was randomized for each participant and was counterbalanced for adults and children. Each photograph was divided into two areas of interests, for the purpose of measuring eye movements. A verb region contained zones of action or instruments used in performing the actions themselves. The subject region contained the face and body regions of the human actors.
For the photographs containing the intransitive actions, the subject regions and verb regions contained the body and the hands, which denoted actions respectively. For the transitive photographs, the human body was the subject region, while hands and objects together denoted the verb region. For two agents with an object acting out an action with each other, bodies served as subject region and the instrument was the verb region.
In this case eye movements made to both the regions were averaged for the purpose of this analysis. We measured both fixational and saccadic eye movements on these regions during the act of speaking. Dependent measures were the total number of fixations, the total number of saccades, the average of duration of fixations, as well as the total gaze durations. These dependent measures were selected as they indicate different aspects of visual attention deployed on photographs during sentence generation.
Measurement of eye movements was made using BGaze software. Proportion of fixations on two areas of interest, verb and subject region, for all three sentence types, was measured. Time window from the onset of image until its offset was considered, spanning the production duration. To be precise, the total time of recording i. Figure 2 shows the fixation proportion of each individual plot, showing the fixations towards an AOI verb or subject for children and adults. The upper panels show proportion of fixations to the verb regions for three different types of images for children and adults.
The time course plot begins from the onset of the picture on the computer screen until the sentence has been spoken. We calculated the proportion of fixations for each bin measuring 50 ms for the entire duration. The x-axis shows the time in milliseconds from this onset for ms. For statistical analysis, voice onset latencies i. VOL for children was ms approx. We compared the average fixation proportion on each AOI during the Voice onset latency window for children, as well as for adults, in order to do a group comparison.
Independent sample t-test was conducted in order to see whether at VOL window, any difference between adults and children existed, or to establish whether modulation of visual attention differs significantly for particular AOI's in children and adults, during the conceptualization phase. We assumed the VOL time period as the conceptualization period. We also analyzed the total number of fixations, total number of saccades, and the total gaze duration, as overall measures of visual attention. Mean comparisons for the number of saccades came out to be insignificant, as there was no interaction effect on the age with AOI'S or sentence types.
Comparison of means for the total dwell time also shows the interaction of age with sentence type, as well as AOI's. Post hoc analysis was done to compare individual mean differences within and between groups. For image type 1, dwell time for children and adults on the verb region was found out to be insignificant. The eye movement patterns of children and adults during sentence generation, while viewing a static photograph, revealed a strong effect on age and also on the type of picture. Children in general deployed higher visual attention to the subject and object regions of the pictures for all three picture types.
Thus, it seems that children needed more time to look at the corresponding portion of the pictures, to retrieve the conceptual knowledge and also to transform that information for the ongoing activity of sentence generation. Thus children made consistently higher number of saccades to both the subject and object regions while producing the sentences, and their overall gaze duration was higher. This could mean that sentence generation with simultaneous viewing of a complex scene requires consistent allocation of visual attention to the relevant regions for extracting syntactic structures.
Further, most previous studies have revealed a tight coupling between visual attention and name retrieval and a sequence of visual and linguistic information as naming is in progress. However, none of these studies have investigated what is the pattern of visual attention and its alignment with conceptualization during scene viewing and speaking.
- Eye Movements and the Visual World, 1st Edition;
- Art Models 3: Life Nude Photos for the Visual Arts;
- Missa Carminum: Kyrie.
Therefore, allocation of higher visual attention in our study by children could not be attributed to the fact that this time was used for lemma and phonological form retrieval, but for sentence construction: since sentence generation is not just a sequential compilation of phonological information from several entities. Since syntactic structure generation would include both an update of form and content, in a holistic way.
Adults, on the other hand, did not require longer deployment of visual attention, probably because of their more proficient strategies and experience. However, interesting difference appeared depending on the number of actors and actions involved. What is theoretically interesting in these results is to note how visual attention is directed towards the verb regions, representing the action zones and subject regions. As noted earlier, most contemporary linguistic theories believe that verbs are important structures in a sentence whose transitivity determines how many possible arguments can accompany it in a sentence.
Therefore, for sentence generation, the transitivity of a verb could have important influences on the structural arrangements of the other arguments. Assuming that one produces uttarances in a sentence in a linear fashion, information from verbs must be derived first for further realisation of it's arguments i.
However, world's languages differ from one another in terms of the word order that they manifest. Thus word order i. Thus, we had hypothesized that in Hindi, being a verb final language, in its canonical sense, speakers must devote maximum visual attention to this region early on during the conceptualization process, so that they can figure out the other arguments and their position in the sentence.
Interestingly, the proportion of fixations to subject and verb regions in our pictures show that adults deployed consistently higher visual attention to the verb region, than to the subject region in all cases, compared to children. This is interesting from a developmental and linguistic point of view. However, this does not mean that children produced wrong sentences or were unable to produce structures, since they were gazing at the subject region.
The explanation could lie with a more efficient adult system of language processing where information from verb could be used quickly to compute online the sentence. Children, on the other hand, might have followed a less canonical pattern of sentence construction i. However, at this point of time, without further controlled studies, this approach remains as a hypothesis. However, the noticeable difference between children and adults in their looking behavior towards the verb and subject regions during conceptualization, does suggest a basic difference in planning strategy.
From a vision-language interaction perspective the differences between children and adults in their deployment of visual attention is important. This means, the overall experience with the visual context and rapid generation of syntactic structures will determine attentional mechanisms.
Thus, when encoding the visual material in the scene, with the goal to articulate a sentence, subjects have to look at those locations more. For example in our case, subjects are mostly going to look at the faces and bodies of the agents and the actions they are engaged in, rather than to other objects that are in their environment. Thus, visual attention and its deployment as linguistic encoding in progress is controlled in a top down manner by the goal of the subject.
However, during speaking, an object based attention is constrained by the type of linguistic material being processed.
The Interface of Language, Vision, and Action: Eye Movements and the Visual World / Edition 1
Thus, children and adults differ in terms of eye movement behavior. For example even between the subject and verb regions we see a very different type of fixational eye movements for different type of pictures. Interestingly, fixations to the subject and verb regions for both children and adults were more or less consistently deployed, though variably throughout the act of speaking. Thus, our results show a very subtle and systematic difference between children and adults in their allocation of visual attention to subject and verb regions of pictures in a sentence generation task.
This difference may have a developmental cause, but more so it tells about the systematic development of multimodal interaction i. At most, the findings of this study should be considered as preliminary, since the study has its own methodological limitations and could not answer many important questions. The results obtained show that children require to pay higher attention during sentence production compared to adults. However, the analysis could not reveal how visual attention was used for formulation of different sentential constituents.
This is because it is nearly impossible to control sentence production during free scene viewing, unless once uses few line drawings and rigidly controlled the order of production of some noun phrases, as has been done in previous studies.
Further, for an SOV language, since the verb comes at the end, it is very difficult to pin point in time when during viewing verb was conceptualized. Since, many psycholinguistic studies have indicated that speakers first generate verb information and then attach relevant arguments to construct sentences. Future studies in this direction should use more refined design and control above factors to draw any meaningful conclusion. From this study's results, we can safely conclude that children and adults differ from one another in terms of their viewing strategy, when they are asked to process pictures and produce sentences.
Allopenna, P. Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of memory and language, 38 4 , Altmann, G. The realtime mediation of visual attention by language a world knowledge: Linking anticipatory and other eye movements to linguistic processing. Journal of Memory and Language, 57 4 , Single and multiple object naming in healthy ageing.
Language and Cognitive Processes, 22, Bock, K. Sentence production: From mind to mouth. Eimas Eds. Vol Speech, language, and communication pp. Orlando, FL: Academic Press. Minding the clock. Journal of Memory and Language , 48, Brown-Schmidt, S. Cho, S. What goes wrong during passive sentence production in agrammatic aphasia: An eyetracking study. Aphasiology, 24 12 , Coco, M. Cognitive Science. Griffin, Z. Gaze durations during speech reflect word selection and phonological encoding.
Cognition, 82 1 , B1-B Why Look? Reasons for eye movements related to language production. New York: Psychology Press. What the eyes say about speaking. Psychological science, 11 4 , A technical introduction to using speakers eye movements to study language. The Mental Lexicon, 6 1 , Speakers gaze at objects while preparing intentionally inaccurate labels for them. Human gaze control during real-world scene perception.
technical summary | Gerry Altmann
Trends in cognitive sciences, 7 11 , Huettig, F. Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm.
Cognition, 96 1 , Mechanisms and representations of language-mediated visual attention. Frontiers in Psychology, 2, Irwin, D. Ferreira Eds. New York: Psychology Press, xiv. Johansson, R. Journal of Vision, 14 14 : 9. Olejarczyk, J. Incidental memory for parts of scenes from eye movements. Visual Cognition, 22 7 , Eye movements.
Reisberg Ed. The Oxford Handbook of Cognitive Psychology. New York: Oxford University Press. Co-registration of eye movements and event-related potentials in connected-text paragraph reading. Frontiers in Systems Neuroscience, 7 28 , Predicting cognitive state from eye movements.
Eye movement control during scene viewing: Immediate effects of scene luminance on fixation durations. Eye movement control in scene viewing and reading: Evidence from the stimulus onset delay paradigm. Temporal oculomotor inhibition of return and spatial facilitation of return in a visual encoding task. Frontiers in Psychology, 4 , Smith, T. Change blindness in a dynamic scene due to endogenous override of exogenous attentional cues. Perception, 42, Apel, J. Targeting regressions: Do readers pay attention to the left? Demiral, S. ERP correlates of spatially incongruent object identification during scene viewing: Contextual expectancy versus simultaneous processing.
Neuropsychologia, 50 , Introduction to computational approaches to reading and scene perception. Visual Cognition, 20, Oculomotor inhibition of return in normal and mindless reading. Nuthmann, A. Using CRISP to model global characteristics of fixation durations in scene viewing and reading with a common mechanism. The penny drops: Change blindness at fixation.
Perception, 41, Do the eyes really have it? Dynamic allocation of attention when viewing moving faces. Journal of Vision, 12 13 : 3, Acunzo, D. Emotion, 5, Andersson, R. Acta Psychologica, , Eye movements and scene perception. Liversedge, I.
Gilchrist and S. Everling Eds.
Oxford: Oxford University Press. Functions of parahippocampal place area and retrosplenial cortex in real-world scene analysis: An fMRI study. Visual Cognition, 19, Matsukura, M. Oculomotor capture during real-world scene viewing depends on cognitive load. Vision Research, 51, Mital, P. Clustering of gaze during dynamic scene viewing is predicted by motion. Cognitive Computation, 3 , Does oculomotor inhibition of return influence fixation probability during scene search? Looking back at Waldo: Oculomotor inhibition of return does not prevent return fixations.
Journal of Vision, 11 1 :3, Object-scene inconsistencies do not capture gaze: Evidence from the flash-preview moving-window paradigm. Psychological Review, , Object based attentional selection in scene viewing. Journal of Vision, 10 8 : 20 , The time course of initial scene processing for eye movement guidance in natural scene search.
Journal of Vision , 10 3 , Malcolm, G. Combining top-down processes to guide eye movements during real-world scene search. Journal of Vision, 10 2 :4 , Lamont, P. Where science and magic meet: the illusion of a 'science of magic'. Review of General Psychology , 14 , Castelhano, M. Viewing task influences eye movement control during active scene perception.
Journal of Vision, 9 3 :6, The influence of clutter on real-world scene search: Evidence from search efficiency and eye movements. Journal of Vision, 9 1 , Searching in the dark: Cognitive relevance drives attention in real-world scenes. How are eye fixation durations controlled during scene viewing? Further evidence from a scene onset delay paradigm.
Visual Cognition, 17, More attention and greater awareness in the scientific study of magic. Nature Reviews Neuroscience, 10 , doi The effects of target template specificity on visual search in real-world scenes: Evidence from eye movements. Journal of Vision, 9 11 :8 , Overt attentional prioritization of new objects and feature changes during real-world scene viewing. Rayner, K. Eye movements and visual encoding during scene perception. Psychological Science, 20 , Facilitation of return during scene viewing. Does gravity matter? Effects of semantic and syntactic inconsistencies on the allocation of attention during scene perception.
Journal of Vision, 9 3 , Williams, C. Age differences in what is viewed and remembered in complex conjuction search. Quarterly Journal of Experimental Psychology, 62, Brockmole, J. Prioritizing new objects for eye fixation in real-world scenes: Effects of object-scene consistency. Visual Cognition, 16 , The role of meaning in contextual cueing: Evidence from chess expertise. Quarterly Journal of Experimental Psychology, 61 , The influence of color on the activation of scene gist. Stable individual differences across images in human saccadic eye movements.
Canadian Journal of Experimental Psychology , 62, Ferreira, F. Taking a new look at looking at nothing. Trends in Cognitive Sciences, 12 , Eye movements and scene memory. Luck and A. Hollingworth Eds , Visual memory pp. Eye movements during scene viewing: Evidence for mixed control of fixation durations. Differential detection of global luminance and contrast changes across saccades and flickers during active scene perception.
Vision Research, 48 , Full scenes produce more activation than close-up scenes and scene-diagnostic objects in parahippocampal and retrosplenial cortex: An fMRI study. Brain and Cognition, 66 , Edit Blindness: The relationship between attention and global change blindness in dynamic scenes. Journal of Eye Movement Research, 2 2 :6, Initial scene representations facilitate eye movement guidance in visual search. I see what you see: Eye movements in real-world scenes are affected by perceived direction of gaze.
Paletta and E. Rome Eds. Berlin: Springer. Regarding scenes. Current Directions in Psychological Science, 16 , Cortical activation to indoor versus outdoor scenes: An fMRI study. Experimental Brain Research , , Visual saliency does not account for eye movements during visual search in real-world scenes. Fischer, W. Hill Eds. Oxford: Elsevier. The face inversion effect is not a consequence of aberrant eye movements. Nigg, J. Concepts of inhibition and developmental psychopathology.
Gorfein Eds. Contextual cueing in naturalistic scenes: Global and local contexts. Using real-world scenes as contextual cues for search. Visual Cognition, 13, Recognition and attention guidance during contextual cueing in real-world scenes: Evidence from eye movements. Quarterly Journal of Experimental Psychology, 59 , Senior, T.
Gazzaniga Eds , Methods in Mind pp. Torralba, A. Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Carr, L. Attentional versus motor inhibition in adults with Attention Deficit Hyperactivity Disorder. Neuropsychology, 20 , Prioritization of new objects in real-world scenes: Evidence from eye movements. Object appearance, disappearance, and attention prioritization in real-world scenes. Incidental visual memory for objects in scenes.
Gajewski, D. The role of saccade targeting in the transsaccadic integration of object types and tokens. Human gaze control in real world search. Paletta, J.
The Interface of Language, Vision, and Action : Eye Movements and the Visual World
Tsotsos, E. Humphreys Eds. Heidelberg: Springer-Verlag. Minimal use of working memory in a scene comparison task. Eye movements are functional during face learning. Eye movements and visual memory for scenes. Underwood Ed. Introduction to real-world scene perception. Incidental visual memory for targets and distractors in visual search. Journal of Abnormal Psychology, , Introduction to the interface of vision, language, and action. Henderson and F. Ferreira Eds.
Related The Interface of Language, Vision, and Action: Eye Movements and the Visual World
Copyright 2019 - All Right Reserved