Modelling the limits of peripheral vision

My latest publication about the limits of peripheral vision is now fully referenced and available. The journal has given me permission to host a PDF version for folks to download for a period of time, so get it while it’s hot! (Be sure also to download the Supplemental Info!) The aim of this blog post is to describe the main points of the paper.

Preamble

Although we experience a highly detailed visual world, only a small portion of our vision – central vision – is high resolution. In the visual periphery, objects appear a bit blurry, and worse, they are prone to visual crowding. Visual crowding refers to the inability to recognise an object when other objects surround it. I’ve written about crowding before, here and here, but for simplicity, check out the picture below. When fixating the central spot, the letter A on the left should be easily visible, whereas the same letter on the right is almost impossible to identify. (This demo was inspired by a similar one in a great review by Pelli and Tillman1.)

crowding demonstration

The obvious difference between the two sides of the above figure is that there are extra distracting lines on the right. These lines interfere with your ability to recognise the A. This is crowding. It happens with all kinds of objects, and happens throughout the entire visual field. At any given moment, there are probably many objects in your vision that are crowded beyond recognition. Right now, the speed at which you read this text is determined by your level of crowding2. We tend not to be aware of such limited recognition, however, because we can effortlessly make eye movements to use our high-resolution central vision to break crowding.

Our contribution

In my paper, we discuss why crowding occurs. We describe a new method for measuring crowding, and we provide a computational model that simulates specific neural processes that may cause crowding.

Our new method involves using a certain method of reporting the appearance of a stimulus in peripheral vision. Studies of crowding typically involve showing an observer a target, such as a letter, and then asking the observer to report the target identity. In these experiments, the observer’s response can only be correct or incorrect. However, we know even from introspection when viewing demonstrations like the figure above, that when an object is crowded, we can still see something. In fact, being able to see something but not being able to correctly identify it is a defining feature of crowding3. It may be informative, therefore, to quantify crowded perception in more ways than as simply correct or incorrect. We thus used a measure that allowed us to describe crowded perception as a loss of perception along a continuum.

The task

We used a target stimulus like the one on the left below, known as a Landolt C. An observer saw this target in peripheral vision. Importantly, from trial to trial, it was rotated randomly, so that the gap section was orientated toward any direction. We then presented the observer with a second Landolt C, this time in their central vision, that they could rotate by pressing buttons. They had to rotate the central C so that it matched the target orientation as closely as possible. On each trial, therefore, the observer’s response was not classed as wrong or right, but instead we found their perceptual error – the difference in rotation between the actual target and the observer’s report.

Landolt floating gaps

This method can be particularly useful for modelling, because over dozens of trials, the observer’s errors conform very nicely to known mathematical functions. People with typical vision are generally pretty good at this – their errors cluster nicely around the actual target orientation. A recent study has related these sorts of perceptual errors to the noisy encoding of the stimulus in primary visual cortex4, the first cortical sight of visual processing (though in that study stimuli were presented at fixation, not in the periphery).

We can quantify crowding by seeing how these patterns of data – and their corresponding mathematical functions – are affected in the case of a crowded target, like that shown on the right of the figure above. In this example, the target is surrounded by a larger ring with a gap in it. When fixating on the blue spot, you may be able to discern that there is a gap in each ring, but the exact position of the gaps may appear “fuzzy”. You may even experience the impression that gaps are sort of joined in some strange way, and that their orientations are very difficult to identify precisely. At the very least, it’s unlikely the target gap will appear as clearly as it does on the left of the figure, despite the physical properties of the targets being the same.

Our human data

When we examined the errors that observers made, we noted that we could see in our data patterns that corresponded to separate, previously competing explanations of crowding. First, we found that, when the orientation of the target and distractor were similar, observers tended to report an orientation close to the average of the orientations. Such feature averaging has been reported widely5, and strongly supports the idea that the visual system simplifies perception in the periphery by finding higher-level statistical associations between objects6. Second, we found that, when the orientation of the target and distractor were dissimilar, observers reported either the target or distractor, and their reports in these cases were quite precise. These sorts of errors are known as confusions or substitutions7, because the observer can apparently see both gap elements of the stimulus, but they are apparently unsure about which gap belongs to which ring. You may be able to experience both of these during different periods of viewing the target stimuli above. Note that these effects disappear in our data when the distractor ring is large and very far away from the target, showing that these phenomena are linked to processes that depend on the spatial positions of items. (Note that there is nothing particularly new about these data – these patterns merely replicate the same effects that have already been widely reported across studies.)

The model

We next generated a computational model that simulates a basic visual process – the coding of orientation information. Because the human data in the uncrowded condition of our experiment conformed well to a mathematical function (a circular normal distribution), we simply asserted that there is some neural process that results in an output of data with the same probabilities as given by that function. That is, for any given uncrowded target, we make our model make the same sorts of errors humans make. We then make the model spatial, by asserting that it’ll respond most strongly to objects centred right on the target; the strength of the model response decreases gradually with the distance of an object from the centre of the target. In the figure below, I’ve tried to give an intuition for how the model works. On the left are three example stimuli, the target is coloured magenta and the distractors are coloured green and blue. The yellow lines can be thought of us our model trying to detect whether there is a gap section at any of those orientations.

model example

On the right side of the above figure are illustrative model responses to the stimuli presented on the left. For any given gap orientation the model will indicate the probability that various orientations of the gap are present. The colours of each of the distributions correspond to the stimuli on the left. The magenta target distribution is centred on 0° (the target orientation in polar coordinates), and is strongest because the target is closest to the centre of the model. You can see that the peaks of the distractor distributions are shifted away from zero, and are weaker than the target.

The model works…

In our study, we simply summed the model response to a target with the model’s response to a distractor, and tested how well those summed probabilities predicted observers’ perceptual reports. We essentially had our model perform the exact same trials our observers performed. This worked very well: the model predicted the average errors made by observers over a range of distractor conditions. More importantly, this model predicted the mixed pattern of data described above, in which some trials indicate that the observer reported the average of the target and distractor orientations, and in other trials they confuse which gap corresponds to which object.

The model works… sort of.

In a second experiment, we used a novel behavioural paradigm to quantify the level of detail an observer could identify. This experiment requires much more explanation, so I’ll save it for another post. However, it’s important to note that, although we found it preferable than models proposed by others, our model had to be altered slightly to be able to account for our observations in this second experiment. This issue – how well our model can be used to describe other datasets – is important and remains to be tested. So, although our model describes the data using our specific task, how well it represents what the brain is doing in crowded scenes requires much further testing. Based on another great model that is conceptually similar{vandenBerg:2010bj}, I’m optimistic these sorts of models can go far.

And finally…

As with all of science, our work builds upon a lot of great stuff of work that came before. While the content of the paper is new, similar methods8-10 and models11,12 have already been published. I’ll save a discussion of the differences between our work and this other work in another post.

My paper’s citation:

Harrison, W. J., & Bex, P. J. (2015). A Unifying Model of Orientation Crowding in Peripheral Vision. Current Biology, 25(24), 3213–3219. http://doi.org/10.1016/j.cub.2015.10.052

References

  1. Pelli, D. G. & Tillman, K. A. The uncrowded window of object recognition. Nature Neuroscience 11, 1129–1135 (2008).
  2. Kwon, M., Legge, G. E. & Dubbels, B. R. Developmental changes in the visual span for reading. Vision Research 47, 2889–2900 (2007).
  3. Pelli, D. G., Palomares, M. & Majaj, N. J. Crowding is unlike ordinary masking: distinguishing feature integration from detection. Journal of Vision 4, 1136–1169 (2004).
  4. van Bergen, R. S., Ji Ma, W., Pratte, M. S. & Jehee, J. F. M. Sensory uncertainty decoded from visual cortex predicts behavior. Nature Neuroscience (2015). doi:10.1038/nn.4150
  5. Greenwood, J. A., Bex, P. J. & Dakin, S. C. Positional averaging explains crowding with letter-like stimuli. Proceedings of the National Academy of Sciences 106, 13130–13135 (2009).
  6. Freeman, J. & Simoncelli, E. P. Metamers of the ventral stream. Nature Neuroscience 14, 1195–1201 (2011).
  7. Strasburger, H. & Malania, M. Source confusion is a major cause of crowding. Journal of Vision 13, (2013).
  8. Ester, E. F., Klee, D. & Awh, E. Visual crowding cannot be wholly explained by feature pooling. Journal of Experimental Psychology: Human Perception and Performance 40, 1022–1033 (2014).
  9. Ester, E. F., Zilber, E. & Serences, J. T. Substitution and pooling in visual crowding induced by similar and dissimilar distractors. Journal of Vision 15, 1–12 (2015).
  10. Tamber-Rosenau, B. J., Fintzi, A. R. & Marois, R. Crowding in Visual Working Memory Reveals Its Spatial Resolution and the Nature of Its Representations. Psychological Science (2015). doi:10.1177/0956797615592394
  11. van den Berg, R., Roerdink, J. B. T. M. & Cornelissen, F. W. A neurophysiologically plausible population code model for feature integration explains visual crowding. PLoS Computational Biology 6, e1000646 (2010).
  12. Dakin, S. C., Cass, J., Greenwood, J. A. & Bex, P. J. Probabilistic, positional averaging predicts object-level crowding effects with letter-like stimuli. Journal of Vision 10, 14 (2010).

 

New paper out: “A unifying model of orientation crowding in peripheral vision”

My latest paper is now available online at Current Biology. Please email me for a copy of the PDF if you don’t have access. In the paper, we summarise a new method, data and model that help to quantify the limits of peripheral vision.

I’ll write a longer blog post soon with some more detailed information, but for the time being, this fantastic press release written by Craig Brierley at University of Cambridge gives a great overview of the key findings:

“At the edge of vision: Struggling to make sense of our cluttered world.”

New paper: Visual crowding is anisotropic along the horizontal meridian during smooth pursuit

My latest paper testing the interaction between eye movements and object recognition has been published at Journal of Vision. You can read the whole thing for free, and download the PDF, via the journal’s website. I co-authored this paper with PhD advisors, Roger Remington and Jason Mattingley, and this happily means all chapters of my thesis have been published in journals.

In this paper, we measured how a particular kind of eye movement affects peripheral vision. We typically move our eyes in one of two ways: 1) we make fast “saccadic” eye movements — reading this text your eyes will be rapidly jumping from word to word; and 2) we make “smooth pursuit” eye movements to track moving objects — imagine watching a bird fly through the sky.

In this movie, when you watch the dot move across the screen, you’ll be using smooth pursuit eye movements:  smooth_pursuit_demo.mov

The quality of motion in that movie isn’t great, but hopefully when you track the dot in the next movie you’ll feel your eyes are doing something quite different than before – this time you’ll be making saccades:  saccade_demo.mov

We were interested in the former type of eye movement, smooth pursuit, and how these movements affect “visual crowding”, an interesting case in which visual perception is highly limited. In the image below, stare at the dot in the centre.

crowding demo

While staring at the dot (and keeping your eyes still!), you’ll probably find it pretty easy to identify the letter “A” in the right circle, whereas the left circle looks like it’s filled with a bunch of random white lines. If you move your eyes around, you’ll see that the same letter “A” appears in both circles, and is as easy to see in the left circle as the right when you move your eyes around. This simple demonstration shows us that our ability to recognise objects (e.g. a letter) in peripheral vision depends on what other information is surrounding the object. The difficulty identifying an object when it’s surrounded by distracting information is called “crowding”.

Here’s another example. There are four concentric circles on each side of the display, and the three inner circles all have gaps in one side. While still looking at the dot in the centre, can you see where the gap is in the inner most circle on the right side of the picture? How about the left side?

concentric rings crowding demo

Something interesting you may notice when staring at the centre dot in the above image is that, not only is it difficult to identify where are the gaps in the rings on the left, but it’s difficult to tell which ring is which colour. You probably can see that there are gaps somewhere, and it’s quite obvious that there are blue and pink rings, but it’s really hard to tell which rings are pink or blue. This demonstration therefore shows us that we are not completely blind to crowded objects – we get a good impression of detail, but the detail gets “mixed up”.

So we tested whether visual crowding is altered during smooth pursuit eye movements. There are some interesting reasons why we expected crowding may be different during pursuit, but I won’t go into these — read the paper’s introduction if you’re interested. In particular, we were interested in whether crowding is different for objects positions ahead of the pursuit target versus behind the pursuit target. Image the dot in the picture below is moving rightward – observers would have to pursue the dot with their eyes, and then (on separate trials) try to identify the crowded letter behind the dot (to the left in this picture) or in front of the dot (to the right in this picture).

In short, we found that there is more crowding for objects opposite to the direction of pursuit, than for objects in the same direction as pursuit. We know that crowding got worse opposite to the direction or pursuit, not that crowding was released (ie. the target was easier to see) in the same direction as pursuit, because we included conditions in which participants did not move their eyes at all. When participants kept their eyes still, crowding was the same as crowding in the same direction as pursuit; crowding opposite to the direction of pursuit was worse than when no eye movements were made.

Why is crowding worse opposite to the direction of an eye movement? This is an open question (assuming our results can be independently verified). My hypothesis, that we put forward in the published paper, is that objects opposite to the direction of pursuit could distract you from making the eye movement, and so the visual system sort of “degrades” how visible they are. We have some evidence for this too: the change in crowding only applied to objects quite close to the pursuit target; when we moved the target farther into peripheral vision, there was no directional change in crowding. Put differently, the objects close to the thing you’re trying to pursue with your eyes, but that are in a position irrelevant to the eye movement, are not as important to you as other objects. What do you think?

And if you’re curious how saccades, the other type of eye movements, affects visual crowding, we’ve published that study too.

The full reference to my Journal of Vision paper is:

Harrison, W.J., Remington, R.W. & Mattingley, J.B. (2014). Visual crowding is anisotropic along the horizontal meridian during smooth pursuit. Journal of Vision 14(1):21, 1-16. http://www.journalofvision.org/content/14/1/21.fulldate

Visual crowding at a distance during predictive remapping

My latest paper is now available online ahead of print via the Current Biology website. Click here to view the page. Please email me (willjharri AT gmail.com) to request a copy if you are unable to download via the site.

You can read my summary of the article here.

Harrison, W. J., Retell, J. D., Remington, R. W., and Mattingley, J. B. (2013). Visual crowding at a distance during predictive remapping. Current Biology 23, 1–6.
Available at: http://www.sciencedirect.com/science/article/pii/S0960982213003527.

Remapped crowding

I’m happy to write that I have just received notice that my most recent manuscript submission has been accepted and is now in press:

Harrison, W. J., Retell, J. D., Remington, R. W., and Mattingley, J. B. (2013). Visual crowding at a distance during predictive remapping. Current Biology.

In my last post, I had a little demonstration of “visual crowding”, the phenomenon where an object in peripheral vision becomes extremely difficult to identify when it is surrounded by visual clutter. You can also experience crowding by fixating the blue dot in part (a) of the figure below. Can you identify the Y buried amongst the Es? You might find that the parts of the letters appeared all jumbled. This is crowding. In part (b), fixate inside the blue dotted circle, and you’ll find it simple to identify the Y on the right side of the figure because it’s not crowded. In my previous paper, my co-authors and I showed that the deleterious effects of crowding (e.g. panel (a)) are reduced in the brief moments just prior to a “saccadic” eye movement toward the crowded object. That is, objects at the goal of the eye movement become easier to identify even before the eyes begin to move.

HarrisonRetellRemingtonMattingley-Figure01-v01

The main finding of this new Current Biology paper is that, under specific conditions, we get the opposite result: just prior to a saccade, an object in peripheral vision that is free from visual clutter, and therefore easy to identify, can be crowded by visual clutter quite distant from the object. This “crowding from a distance” occurs because of predictive remapping. Predictive remapping is a theory which suggests, just before an eye movement is made, the visual system predicts where visual objects will fall on the retina when the eye movement is complete. Similar predictions are made within our motor systems: when you plan to grab a cup of coffee on your desk, you plan to move your hand, and you have an expectation – a prediction – about the outcome of the plan (presumably to bring your hand to the cup). You can then compare the actual outcome of the movement with the prediction made before the movement to check if the movement achieved its goal. The same goes for movements of the eyes, but the outcome of the eye movement is new visual input.

In our study, we had people make a saccade to a specific location so that we knew where their eyes would move, and we thus had a good idea about the internal prediction regarding where things would fall on the retina following the eye movement. For example, in part (b) of the figure above, the observer would start by fixating within the blue dotted circle in the centre of the display. As shown by the orange arrow, they would then have to make a saccade (a fast eye movement) to the green dot at the far right. After they moved their eyes, observers had to report the identity of a letter presented briefly just before the eye movement began. In the case of the example, the letter Y would appear and disappear before the observer moved their eyes. The presentation of this letter probe was timed so that it appeared during the period of predictive remapping.

Because we knew the direction of the eye movement and the position of the probe, we also knew the predicted position of the probe. The predicted position of the probe letter is represented by the red arrow in the figure. That is, before an observer’s eyes moved, we knew where their visual system predicted the probe would appear on the retina following the eye movement. At this location, the probe’s “remapped location”, we presented visual clutter, the letter Es in this case. The positioning of the stimuli resulted in the probe becoming difficult to identify during, and only during, the period of predictive remapping, hence the title of the paper: “Visual crowding at a distance during the period of predictive remapping.”

The spatial arrangement of stimuli and timing of their presentation needs to be controlled tightly relative to a viewer’s eye movement, so it’s not feasible for me to create a demonstration of the effect. In the coming days (weeks?) I’ll update this post with a figure from the paper that schematises the logic and layout of stimuli. UPDATE: figure added.

More details and a link to the paper to come when it goes online.

If you’re interested in reading more about remapping, the following articles are good places to start:

Duhamel, J. R., Colby, C. L., and Goldberg, M. E. (1992). The updating of the representation of visual space in parietal cortex by intended eye movements. Science 255, 90–92.

Merriam, E. P., Genovese, C., and Colby, C. L. (2003). Spatial updating in human parietal cortex. Neuron 39, 361–373.

Rolfs, M., Jonikaitis, D., Deubel, H., and Cavanagh, P. (2011). Predictive remapping of attention across eye movements. Nature Neuroscience 14, 252–256.

If you’re interested in reading more about visual crowding, try:

Bouma, H. (1970). Interaction effects in parafoveal letter recognition. Nature 226, 177–178.

Pelli, D. G., and Tillman, K. A. (2008). The uncrowded window of object recognition. Nature Neuroscience 11, 1129–1135.

Eye movement targets are released from visual crowding.

Here’s a link to my recent Journal of Neuroscience article, coauthored by my PhD advisors Jason Mattingley and Roger Remington. We tested whether a target closely surrounded by distractors becomes easier to identify just before an eye movement is executed toward the target.

To explain the main findings in the paper briefly, consider the following. Look at the top cross on the left, and, without moving your eyes, see if you can identify the second letter in the string of letters at the far right. Now try identifying the letter in the second row while looking at the bottom cross.

crowdDemo

You’ll probably find it very difficult (impossible?) to identify the second letter in the top row, but it’ll be much easier to identify the same letter in the bottom row. The difference in difficulty can’t simply be due to the distance of each letter from the fixation cross, because the distance of each letter is the same. So, in the top row, there is some sort of interaction between the letters making it difficult to distinguish one from another. The difficulty recognising something in peripheral vision when it’s surrounded by clutter is referred to as “crowding”.

Back to my study. I had participants try to identify a crowded object. In half the trials, they kept their eyes still on a specific place, like the cross above, and they had great difficulty identifying the target as we expected. In the other half of the trials, the participants were required to make a saccadic eye movement toward the crowded target. Using an eye tracker, I was able to switch off the target prior to the start of each eye movement, so that the participants never saw the target after they moved their eyes. I found that, when participants had to identify the crowded target while also preparing an eye movement towards it, their ability to identify the target improved significantly. What’s most important about my findings is that this improved target recognition began before their eyes moved! So, preparing to make an eye movement resulted in better identification of a crowded target, effectively giving the participants a “sneak peak” of what’s in their peripheral vision even before their eyes moved. Pretty neat.

Harrison, W. J., Mattingley, J. B., & Remington, R. W. (2013). Eye movement targets are released from visual crowding. Journal of Neuroscience, 33(7), 2927–2933. doi:10.1523/JNEUROSCI.4172-12.2013