Columnar processing of border ownership in primate visual cortex

  1. Tom P Franken
  2. John H Reynolds

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This paper is a milestone towards understanding the formation and representation of visual object structure in the brain. It shows that in the pivotal area V4, border ownership selectivity emerges in the deep layers earlier than in the granular layers which receive the input from V1/V2, indicating that border ownership is not inherited from the input, but computed by deep-layer neurons using visual context information possibly provided through horizontal connections, cortico-cortical feedback or thalamic input. They further report that the preferred side of border ownership across layers is similar, i.e. it is organized in a columnar fashion. The study is elegantly done, with the outstanding questions clearly laid out and the results presented in a clear and informative fashion.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 and Reviewer #3 agreed to share their names with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

To understand a visual scene, the brain segregates figures from background by assigning borders to foreground objects. Neurons in primate visual cortex encode which object owns a border (border ownership), but the underlying circuitry is not understood. Here, we used multielectrode probes to record from border ownership-selective units in different layers in macaque visual area V4 to study the laminar organization and timing of border ownership selectivity. We find that border ownership selectivity occurs first in deep layer units, in contrast to spike latency for small stimuli in the classical receptive field. Units on the same penetration typically share the preferred side of border ownership, also across layers, similar to orientation preference. Units are often border ownership-selective for a range of border orientations, where the preferred sides of border ownership are systematically organized in visual space. Together our data reveal a columnar organization of border ownership in V4 where the earliest border ownership signals are not simply inherited from upstream areas, but computed by neurons in deep layers, and may thus be part of signals fed back to upstream cortical areas or the oculomotor system early after stimulus onset. The finding that preferred border ownership is clustered and can cover a wide range of spatially contiguous locations suggests that the asymmetric context integrated by these neurons is provided in a systematically clustered manner, possibly through corticocortical feedback and horizontal connections.

Article activity feed

  1. Version published to 10.7554/elife.72573 on eLife
  2. eLife

    Author Response:

    Reviewer #1:

    The study is elegantly done, with the outstanding questions clearly laid out and the results presented in a clear and informative fashion. I have only a few suggestions to strengthen some of the results.

    1. Determination of layers: The CSD based method used to determine the layers seems a bit ad hoc, although other studies have often used a similar approach. Some histological evidence would be great. If that is not possible, the authors should provide some more details to determine the layer specificity. For example, where were the supragranular-granular and granular-infragranular borders for different penetrations (i.e., which electrode(s) marked these boundaries)? These could be expressed as fractions of the shaft length, and from that, we would approximately know the depth. Also, were these results affected by how the CSDs were smoothed?

    We thank the Reviewer for the suggestions. Using CSD from individual penetrations to define the position of laminar compartments is a strategy has been used by several different laboratories, not only in primary visual cortex (e.g. Mitzdorf, 1985; Poort et al., 2016; Bijanzadeh et al., 2018), but also in extrastriate areas, such as other studies in V4 (Nandy et al., 2017; Lu et al., 2018, Pettine et al., 2019; Ferro et al., 2021), and in other cortical areas, for example in medial temporal cortex (Takeuchi et al., 2011). We include a new figure that shows the similarities between the CSD profiles from different studies (Figure 1-figure supplement 3). Figure 1-figure supplement 3A shows the population average of the CSD in our data. The similarity between this panel and the individual examples shown in Figure 1 and Figure 1-figure supplements 1 and 2 further highlights the fairly consistent sink-source patterns observed across individual penetrations in our data. Figure 1-figure supplement 3B shows that this pattern is also consistent with that found in macaque V4 in another laboratory (Pettine et al., 2019). Figure 1-figure supplement 3C shows that this pattern is not specific for V4, but also occurs in other areas, such as the medial temporal cortex (Takeuchi et al., 2011).

    Importantly, in the latter study Takeuchi et al. were able to verify histologically, after applying electrolytic marks, that the prominent current sink with short latency (white star in Figure 1-figure supplement 3C) corresponds to the granular layer, thus consistent with CSD analysis in V1 (Mitzdorf, 1985). Thus although we are not able to perform such histological verification in our study (one animal has been euthanized, and the other animal is destined to take part in another project), the very similar sink-source patterns between these different studies (Figure 1-figure supplement 3A-C), including with work from others that has been verified histologically, gives us confidence that we can meaningfully use them to assign electrode contacts to granular, superficial and deep layers, as other laboratories have done (e.g. Lu et al., 2018). In addition to the new Figure 1-figure supplement 3, we added language in the corresponding paragraph in Results to explain this better. We clarified in Results that interpretable CSD maps were found in 81 out of 88 penetrations and that only those were used in the laminar analysis (which was already indicated in Methods in the previous version of the manuscript).

    As suggested by the Reviewer, we have added the positions of the electrode contacts to the CSD maps in Figure 1 and Figure 1-figure supplement 2 (labels along ordinate on the right of the panels). The electrode contacts on the probe covered 3.1 mm (32 contacts with 0.1 mm distance between adjacent contacts), thus the full depth of the cortex was covered even though the vertical position of the probe varied between penetrations (also because a layer of granulation tissue develops over time between the artificial dura and the pial surface). Therefore, to aid in estimating the depth of the individual penetrations, we indicate the position of the most superficial contact on which multiunit activity was recorded (solid black triangles in Figure 1; Figure 1-figure supplements 1,2; for all cases where this contact could be identified, i.e. if the most proximal contact on the probe did not show multiunit activity). The average position of this contact is shown on the population CSD map (solid black triangle on Figure 1-figure supplement 3). The advantage of a method that assigns layers for a penetration based on data from the same penetration, as we have used here, as opposed to a method that assigns compartments based on depth derived from a population average, is that the former helps to avoid errors due to variations in probe position, and due to a variable degree of tissue compression that may occur for different penetrations.

    To test whether the results were affected by how the CSDs were smoothed, we performed the layer assignment separately using different methods of smoothing. To ensure that we did not bias the results we blinded ourselves to the original layer assignment when applying each method. We find that the laminar position of >97% of well-isolated units was identical as that obtained when using the standard procedure of smoothing the CSDs. This resulted in robust latency differences of border ownership signals between layers, irrespective of which smoothing method was used. These results are presented in a new supplementary figure (Figure 2-figure supplement 3).

    1. Another important factor is the orthogonality of the penetrations. This can also be better quantified based on the variation of the RF centers with depth.

    We followed the Reviewer’s suggestion and evaluated orthogonality of the penetrations by computing the distance D between receptive field centers along the probe (Methods). We show this metric for the vertical positions of receptive field contours shown for the penetrations in Figure 1H,I and Figure 1-figure supplement 1, and describe the population data in the first paragraph of Results (median (IQR) 0.83 o/mm (1.00 o/mm), for all 81 penetrations that were included in the laminar analyses). This indicates that the variation is small relative to the average diameter of the receptive fields (7.36 o), and that the deviation from orthogonality is limited.

  3. eLife

    Evaluation Summary:

    This paper is a milestone towards understanding the formation and representation of visual object structure in the brain. It shows that in the pivotal area V4, border ownership selectivity emerges in the deep layers earlier than in the granular layers which receive the input from V1/V2, indicating that border ownership is not inherited from the input, but computed by deep-layer neurons using visual context information possibly provided through horizontal connections, cortico-cortical feedback or thalamic input. They further report that the preferred side of border ownership across layers is similar, i.e. it is organized in a columnar fashion. The study is elegantly done, with the outstanding questions clearly laid out and the results presented in a clear and informative fashion.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 and Reviewer #3 agreed to share their names with the authors.)

  4. eLife

    Reviewer #1 (Public Review):

    The study is elegantly done, with the outstanding questions clearly laid out and the results presented in a clear and informative fashion. I have only a few suggestions to strengthen some of the results.

    1. Determination of layers: The CSD based method used to determine the layers seems a bit ad hoc, although other studies have often used a similar approach. Some histological evidence would be great. If that is not possible, the authors should provide some more details to determine the layer specificity. For example, where were the supragranular-granular and granular-infragranular borders for different penetrations (i.e., which electrode(s) marked these boundaries)? These could be expressed as fractions of the shaft length, and from that, we would approximately know the depth. Also, were these results affected by how the CSDs were smoothed?

    2. Another important factor is the orthogonality of the penetrations. This can also be better quantified based on the variation of the RF centers with depth.

  5. eLife

    Reviewer #2 (Public Review):

    Identifying and representing the object structure of a scene is a fundamental visual function that needs to be clarified. It is this function that allows the brain to detect object structure, to link the emerging visual properties of objects (like shape, color and movement) to the individual detected structures, to establish object representations that are stable across eye movements, and to focus selective attention on objects. Previous studies have shown that border ownership selectivity, that is, object-based coding, emerges in visual cortex shortly after the onset of feature signals. The unsolved puzzle is where the necessary context information comes from and how the cortex computes object structure from it so fast. Previous studies have ruled out feed-forward mechanisms and intra-cortical horizontal connections of V1/V2, the former because they do not have the context, the latter because they are too slow.

    The present study, showing that border ownership selectivity emerges in the deep layers of area V4 earlier than in the granular layers is significant in two ways. First it rules out the possibility that V4 simply inherits border ownership selectivity from V1/V2. Second, it points to the deep cortical layers as the origin of border ownership computation. This points to several possible sources of the context input, including cortico-cortical connections and thalamic input.

    There is one previous study (Bushnell et al., J Neurosci, 2011) that also concluded, based on the latency of the effect, that border ownership selectivity in V4 cannot be simply inherited from upstream areas. They tested curvature selective V4 cells and found that the responses to the optimal shape were suppressed when the critical curved contour was not intrinsic to the shape but owned by an occluding object. They found latencies of suppression as short as 46 ms, shorter than the latencies of V2 cells reported in Zhou et al. 2000. The present study should mention this finding.

    The present study also demonstrates that the cell's spatial tuning for direction of border ownership is somewhat independent of their orientation tuning. That is, their border ownership preference is not simply for one or the other side of their preferred edge stimulus; and even cells without orientation tuning can be direction-of-object selective. This is further indication that object location is computed by an independent mechanism.

    It would be very interesting to see a similar laminar analysis applied to area V2. Perhaps the computation of border ownership there occurs also in the deep layers, using similar sources of context information. Alternatively, the V2 border ownership selectivity could be the result of back projection from V4. Because of the importance of representing object structure for many visual tasks, as pointed out above, I would not be surprised if similar fast border ownership computation would be generally found in the deep cortical layers, in V1 and V2 and in areas of the ventral stream beyond V4.

    The new findings support the idea of the grouping cell model, namely that the context information is provided by an external grouping signal that modulates the activity of the feature neurons in the visual cortex. The grouping signal is supposed to represent a 'proto-object', a computational structure that (1) links the visual feature signals, (2) is being remapped across eye movements, and (3) serves object-selective attention. Compared to the large number of studies of feature selectivity in visual cortex, the question of the representation of object structure has prompted relatively few studies, despite its theoretical importance, and the big question of where and how grouping signals are generated still awaits an answer.

  6. eLife

    Reviewer #3 (Public Review):

    The authors ask a simple but important question related to our ability to assign borders to objects, namely whether these are computed at early stages of cortical processing (and inherited at mid-level), whether they are organized in columnar manners, and whether they match to orientation preference. They report that border-ownership in V4 is not inherited from upstream areas. They speculate that it is computed de novo by infragranular neurons, but there is no proof for that. It could also be is due to feedback from higher areas. Border ownership is organized in columns, and while it often aligns with preferred orientations, there is often a surprizing mismatch. The experiments are performed to high standard and convincing, and answer a relevant question.

  7. Version published to 10.1101/2021.08.06.455427v1 on bioRxiv