Top-down bottom-up Neuroscience

Research suggests that perception and imagination engage neuronal representations in the same visual areas. However, the underlying mechanisms that differentiate sensory perception from imagination remain unclear. Here, we examine the directed coupling [effective connectivity] between fronto-parietal and visual areas during perception and imagery. We found an increase in bottom-up coupling during perception relative to baseline and an increase in top-down coupling during both perception and imagery, with a much stronger increase during imagery. Modulation of the coupling from frontal to early visual areas was common to both perception and imagery. Furthermore, we show that the experienced vividness during imagery was selectively associated with increases in top-down connectivity to early visual cortex. These results highlight the importance of top-down processing in internally as well as externally driven visual experience.

Visual experience can be caused by external events in the outside world, like the appearance of an object, or by internal signals generating visual images in our minds eye. Localisation of the neural structures that represent the content of visual imagery is an important step in the process of understanding the underlying mechanisms that generate visual images1. In 1980, Kosslyn proposed that imagery uses the same visual buffer as perception to represent visual content. In line with this idea, neuroimaging has shown that visual areas have similar neural representations of imagined and perceived objects, with higher overlap in late visual areas2, 3. The overlap in early visual areas depends on the exact imagery task4, 5 and the experienced vividness of the imagery6, 7.

Developing a detailed understanding of the mechanisms by which our brains generate visual experience calls for the elucidation of dynamic top-down and bottom-up connectivity within and between the neural structures involved8, 9. Whereas during perception, activation of visual representations is ultimately caused by bottom-up influences from the retina, these exogenous influences are absent during visual imagery. How visual areas are activated in the absence of stimulus bound, bottom-up input, remains an open question. Recent work using measures of effective [directed] connectivity during imagery suggests that top-down projections from fronto-parietal areas to visual areas are involved in visual imagery10, 11.

There is a large body of research showing that top-down influences also play an important role in perception12,13,14. The predictive coding account of perception proposes that visual experience is a product of the reciprocal exchange of bottom-up and top-down influences throughout the neuronal hierarchy15, 16. From this perspective, the question arises to what extent recurrent exchange differs during perception and imagery. Here, we investigated this aspect of distributed neuronal processing by examining how effective connectivity changes during these two forms of visual experience.

We hypothesized distinct context sensitive patterns of top-down and bottom-up influences during imagery compared to perception. We used dynamic causal modelling [DCM] to characterise the effective connectivity that best explains the BOLD [Blood Oxygen-Level Dependent] response during visual perception and imagery. Based on hierarchical predictive coding, we hypothesized an increase in bottom-up coupling, relative to baseline, during perception but not imagination and an increase in top-down coupling during visual experience; i.e., both perception and imagery.

Subjects

Twenty-nine healthy adult volunteers with normal or corrected to normal vision gave written informed consent and participated in the experiment. An initial analysis of these data is already published in Dijkstra, Bosch, and van Gerven7. Three participants were excluded; two due to insufficient data caused by scanner problems and one due to not completing the task. Twenty-six participants [mean age = 24.31, SD = 3.05, 18 female] were included in the reported analyses. The study was approved by and in accordance with the guidelines of the local ethics committee [CMO Arnhem-Nijmegen].

Procedure and experimental design

The experimental paradigm is depicted in Fig.1a. We adapted a retro-cue working memory paradigm17. In each trial, participants were shown two successive images, followed by a cue indicating which of the two they should subsequently imagine. The stimulus set consisted of six images obtained from the World Wide Web: two faces [Barack Obama and Emma Watson], two letters [D and I] and two kinds of fruit [banana and apple]. During imagery, a frame was presented within which subjects were asked to imagine the cued stimulus as vividly as possible, while maintaining fixation on a cross presented in the centre of the screen. It has been shown that neural activity during visual imagery is influenced by the experienced vividness of visual imagery2, 6, 18. Therefore, we asked participants to indicate their experienced vividness of visual imagery on each trial on a scale from one to four, where one was low vividness and four was high vividness. Previous research has shown that such a subjective imagery rating shows high test-retest reliability and correlates with objective measures of imagery vividness18, 19.

The experiment comprised nine blocks, each consisting of twenty trials. Hence, each stimulus was perceived 60 times and imagined 30 times over the course of the whole experiment, resulting in a total scanning time of approximately 1.5 hours per participant.

General linear model

The fMRI acquisition and pre-processing details are described in the Supplementary Material. All analyses were performed using SPM12. To identify our regions of interest, prior to the connectivity analyses, we inverted a general linear model [GLM], with data and regressors concatenated over runs. As our focus was on establishing domain general mechanisms of imagery and perception, we collapsed over stimulus categories, thereby simplifying the subsequent DCM analysis. Our regressors of interest modelled the perception events, the imagery events, and the parametric modulation of imagery vividness. Inspection of the design orthogonality revealed an absolute cosine angle of 0.13 between the perception and imagery regressor, indicating that the experimental design enabled the imagery response to be disentangled from the perception response. Analysis of behavioural data [vividness ratings during scanning] demonstrated an effect of stimulus category on vividness, with letters being experienced as more vivid than fruit and fruit as more vivid than faces7. This indicated that any effect of vividness could be explained by stimulus category. To control for contributions of stimulus category, we regressed out the category effect by mean-centring the vividness scores per category. Finally, the visual cues, the presentation of the vividness instruction screen and the button presses were modelled with separate regressors, along with the six movement [nuisance] regressors.

Selection of regions of interest

We selected regions of interest [ROI] for our connectivity model based on the results from our GLM analysis and prior knowledge from the literature3, 20, 21. To allow for variation between participants in the exact location of the effect, we selected the ROIs per participant as an 8 mm sphere centred on the subject-specific maximum within a 16 mm sphere centred on the group maximum, in space. This approach ensures that the subject- specific ROIs were close to the average group activity, while allowing for slight variation in functional anatomy between participants.

In each subject, time series were extracted from every voxel in each of the ROIs. The data were subsequently adjusted based on an F-contrast that retained the experimental effects of interest [i.e., perception, imagery, vividness, cue, button press, instruction text] and regressed out task-unrelated variance caused by sources that were not of interest, for example, head movement. The first principal component [eigenvariate] of each regions adjusted data was then used as the time series for subsequent DCM analysis.

Dynamic causal modelling

In order to quantify the effects of perception, imagery and vividness on effective connectivity between the ROIs, we used dynamic causal modelling [DCM]22. DCM uses the following bilinear state equation to infer effective connectivity parameters:

where the dot notation denotes the time derivative. The variable z describes neuronal activity resulting from the interplay of three different influences. First, the A matrix represents endogenous or fixed connectivity during baseline, in the absence of external stimulation. Second, the elements in B  j represent the changes in connectivity due to experimental influences u j . Finally, the matrix C denotes the direct influence of each experimental input u j . The neuronal model is coupled to a biophysically plausible model of neurovascular coupling and the BOLD response, which together generate the predicted BOLD time series. The slice timing model within DCM was set to its default value of 0.5 TR23. The model is then inverted to find the neural and haemodynamic parameters which offer the best trade-off between model fit and complexity [i.e. maximises the negative free energy, which is an approximation to the log model evidence log p[y|m]]24.

A parameter-estimate Ai,j of a connection from area i to area j, reflects the rate of change [in Hz] in the neuronal activity of area j that is caused by a change in the firing rate of area i. In this context, a positive parameter refers to an increase in the firing rate of area j caused by area i, which can be interpreted as an excitatory influence. A negative parameter refers to a decrease in the firing rate of area j caused by area i, which can be interpreted as an inhibitory influence. Note that this does not relate to specific excitatory or inhibitory neurons or their axonal projections; rather it is the net influence of one brain region on another. Similarly, a parameter-estimate Bi,j,k of an experimental influence k on the connection from area i to j, reflects the increase [in Hz] in the coupling from area i to area j caused by the experimental manipulation k.

Connectivity parameter estimation

Typically, in DCM, a few models are specified, each of which represents a hypothesis about the connectivity architecture of the system being studied. These models differ in the presence or absence of an influence of experimental manipulations on certain connections. However, in the current context we expected almost all connections to be influenced by perception and imagery, but that the strength of influence would be different. In other words, our experimental questions were about the relative strength of coupling under different experimental manipulations, instead of the presence or absence of context sensitive changes in coupling. In other words, our aim was to test quantitative hypotheses about model parameters rather than qualitative aspects of model structure. To test hypotheses about changes in connectivity at the group level, we used hierarchical Bayesian modelling and averaging. This entails evaluating a large number of connectivity models and taking a weighted average of the connectivity parameters, weighted by the evidence of each model25. This approach, called Bayesian Model Averaging [BMA], accommodates uncertainty about the underlying model structure on the parameter estimates.

Estimating the parameters of all plausible DCMs can be a time consuming process and novel ways have been developed to make BMA more computationally efficient. Instead of estimating every possible model separately, it is possible to only estimate a full or parent model, containing all the parameters of interest and use the posterior estimates of this full or parent model to derive the posterior estimates [and model evidence] of reduced models, in which one or more parameters are systematically removed. This is called Bayesian model reduction [BMR] and gives similar results to the standard approach, but is computationally much more efficient25.

The BMA scheme described above could be used with classical inference methods such as t-tests to evaluate whether a parameter is non-zero at the group level. However, this ignores uncertainty at the within subject level [and assumes that every subject contributes an equally good estimate of the parameters]. To circumvent this issue, we used a parametric empirical Bayesian [PEB] approach with DCM26, 27. In short, the PEB scheme uses BMR to invert a hierarchical [Bayesian] model of between subject effects on within-subject parameters. This involves specifying a general linear model of between-subject effects. Here, we simply tested for nontrivial group means of condition-specific changes in connectivity [i.e., input specific B  j parameters above]. Further details on how we used PEB in the current analysis can be found in the Supplementary Material.

Modelling driving inputs and changes in connectivity

In DCM, the activity in the network is driven by direct experimental inputs u j , here modelled as boxcar regressors [Fig.1b,c]. We assume that during perception, sensory input enters the cortical network via the early occipital cortex [OCC]. We expressed this in the DCM by modelling the perception regressor [Fig.1b] as driving the dynamics via region OCC. Imagery, in contrast, is assumed to be internally driven, via top-down coupling from higher areas. However, because we did not want to bias the model towards a top-down account of imagery [and because we did not have sufficiently strong prior beliefs about whether imagery should arise in parietal or frontal areas], we used BMR over all possible driving locations for imagery, prior to estimating condition-specific changes in connectivity [see Driving input during imagery].

The main aim of this study was to investigate changes in top-down and bottom-up connectivity between visual perception and imagery. Research on connectivity during visual imagery has been surprisingly scarce. Functional connectivity studies of working memory maintenance, which is assumed to show a substantial overlap with the neural mechanisms of visual imagery6, 28, suggests strong connectivity among all our ROIs29. Thus, we have no a priori reasons to constrain our model. Therefore, we modelled both imagery and perception as potentially influencing all connections. We also had no reason to constrain which connections could be influenced by imagery vividness: more vivid imagery could be caused by an increase in top-down processing from either parietal or frontal areas to early or late visual areas, or by a decrease in bottom-up processing in the same pathways. Furthermore, since most effects of vividness have been found in early visual areas2, 6, 30, this could also be due to an increase in excitability within this area as a result of decreased self-inhibition31. Therefore, we modelled vividness on all connections and on the self-connection of OCC. We used the analysis scheme described above with PEB and BMR to estimate differential [group average] changes in directed connectivity within the DCM.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Imagery vividness ratings

The mean vividness rating over trials, averaged over participants, was 2.97 [SD = 0.52]. All participants showed variation in their vividness ratings over the course of the experiment [mean SD = 0.79, SD = 0.17]. On average, 8.79% of trials wererated as vividness = 1, 20.85% as vividness = 2, 35.10% as vividness = 3 and 35.28% as vividness = 4.

Activated brain areas during perception and imagery

The standard SPM [GLM whole brain] analysis showed that both perception and imagery activated early visual, ventral stream and parietal areas [Fig.2]. Furthermore, imagery was associated with a large increase in activity in inferior frontal gyrus [IFG] and supplementary motor area [SMA]. SMA responses were possibly due to motor preparation for the vividness rating; therefore, we did not include this area in further analysis. We focused on the right hemisphere to reduce model complexity and because it has been shown that imagery is associated with stronger activation of the right hemisphere32.