The auditory system converts the physical properties of a sound waveform to neural activities and processes them for recognition. During the process, the tuning to amplitude modulation (AM) is successively transformed by a cascade of brain regions. To test the functional significance of the AM tuning, we conducted single-unit recording in a deep neural network (DNN) trained for natural sound recognition. We calculated the AM representation in the DNN and quantitatively compared it with those reported in previous neurophysiological studies. We found that an auditory-system-like AM tuning emerges in the optimized DNN. Better-recognizing models showed greater similarity to the auditory system. We isolated the factors forming the AM representation in the different brain regions. Because the model was not designed to reproduce any anatomical or physiological properties of the auditory system other than the cascading architecture, the observed similarity suggests that the AM tuning in the auditory system might also be an emergent property for natural sound recognition during evolution and development.SIGNIFICANCE STATEMENT This study suggests that neural tuning to amplitude modulation may be a consequence of the auditory system evolving for natural sound recognition. We modeled the function of the entire auditory system; that is, recognizing sounds from raw waveforms with as few anatomical or physiological assumptions as possible. We analyzed the model using single-unit recording, which enabled a fair comparison with neurophysiological data with as few methodological biases as possible. Interestingly, our results imply that frequency decomposition in the inner ear might not be necessary for processing amplitude modulation. This implication could not have been obtained if we had used a model that assumes frequency decomposition. For exaa) . The AM a) b (Houtgaa) (b) . Psychopa) (b) .The auditory system converts the physical properties of a sound stimulus to neural activities and processes them through a cascade of brain regions . Classification accuracy is defined as the average correct classification rate for each category, namely the number of time frames correctly estimated as a particular category divided by the total number of time frames in the category.Two datasets were used to train the DNNs. The first consisted of nonhuman natural sounds and is a subset of ESC-50 . The oriThe second dataset consisted of speech sounds . Each enOur DNN consisted of a stack of dilated convolutional layers  Fig. 2)Fig. 2), We used DNNs with 13 layers, each containing 128 units, for nonhuman sound and DNNs with 12 layers, each containing 64 units, for speech. The number of layers and the number of units in each layer were determined based on a pilot study and fixed throughout the study. In the pilot study, DNNs with various numbers of layers and units were trained using a random portion of the training set. The filter length was 2 and the dilation length was 2 to the power of the layer number . The numWe tested multiple architectures with random filter sizes and dilation lengths in each convolutional layer and selected the DNN that achieved the best classification accuracy on the novel dataset . The filThe DNNs were trained on the training set and the classification accuracy was calculated for the validation set. The initial filter weights were randomly sampled and the biases were set at 0 in accordance with a previous study . The filFor a physiological analysis of a DNN, a sound stimulus was fed to the DNN and the values of each unit were recorded. The root mean square (RMS) of the input sound was adjusted approximately to the RMS of the training set. Before the analysis, 1 was added to the values of all the units because the minimum possible value of the activation function is −1 .The stimulus was 8 s of sinusoidally amplitude-modulated white noise. In physiological studies, tuning to an AM rate is measured using sinusoidally amplitude-modulated tones with carriers at the neurons' best acoustic frequency , sinusoidally amplitude-modulated white noises, or click trains. We did not use tones as carriers because many units showed multiple troughs in the AF tuning curves or nonmonotonic responses to the input amplitude, making it difficult to define the best AFs.t is an index of the time frame, a(t) is the unit activation, fs is the sampling rate, and fm is the stimulus AM rate. The average activity was defined as the temporal average of the values, which could be considered as the DNN version of an average spike rate. The synchrony and the average activity were averaged for 16 instances of the carrier white noise to reduce the effect of stimulus variability. A temporal modulation transfer function (tMTF) and a rate MTF (rMTF) were defined as the synchrony and average activity as functions of the AM rate, respectively. In physiology, an MTF is usually defined only at the AM rates at which the unit shows a statistically significant synchrony or spike rate. Because a statistical test on the results of a deterministic model such as our DNN makes no sense, we considered the synchrony or average activities below a certain threshold as “nonsignificant” and excluded them from the following analysis. The threshold was arbitrarily set at 0.01 for the synchrony and at 0.01 above the average activity in response to unmodulated white noise for the average activity.The synchrony to the stimulus and the average activity was calculated from the activations of each unit. The synchrony to the stimulus was quantified in terms of vector strength . When deAn MTF was classified as one of the following four types: low-pass, high-pass, band-pass, or flat. A low-pass type MTF was defined as one that had no values smaller than 80% of its maximum for AM rates smaller than the peak rate. A high-pass type MTF was defined as one that had no values smaller than 80% of its maximum for rates larger than the peak rate. A flat MTF was defined as one that had no values smaller than 80% of its maximum or one with a peak to peak range of <0.1. The band-pass MTF was defined as being other than the above.a and the other for quantitative comparisons with specific physiological data for neurons found in the literature. The UCF for visualization was defined as the rate at which the MTF crosses 80% of its maximum value. If an MTF had multiple such rates, the geometric mean of the rates was used. The threshold of the UCF used for a quantitative comparison with the auditory system varied according to the reference physiology study. The thresholds were 50% (The best modulation frequencies (BMFs) were calculated from the band-pass type MTFs and the upper cutoff frequencies (UCFs) were calculated from the low-pass and band-pass type MTFs. The BMFs of low-pass, high-pass, or flat MTFs and the UCFs of high-pass or flat MTFs were considered to be impossible to define. The BMF was defined as the modulation rate at the peak of the MTF. If there were multiple peaks with the same height, the geometric mean of the rates was taken. The UCF was calculated in two different ways: one for qualitative visualization as shown in Figure 7were 50% , 80% , we derived the final layer–region pairwise similarity matrix. Because no distribution of tBMF has been reported in auditory nerves (ANs); no distribution of rBMF has been reported in ANs, cochlear nucleus (CN), or superior olivary complex (SOC), and no distribution of rUCF has been reported in AN or CN, their similarities were set at 1 if there was no unit with a definable BMF or UCF and set at 0 otherwise. For regions other than these, the similarity was set at 0 if there was no unit with a definable BMF or UCF.We extracted the BMF and UCF distributions reported in previous physiological studies by digitizing the figures. If multiple figures were available, then we chose the clearest figure or that with the most neurons. The extracted values were used for a qualitative visualization in Figure 7i, j) was proportional to the following:Ni and Nj are the number of brain regions (7) and the number of DNN layers, respectively. The weight was scaled so that the squared mean of the weight matrix was 1. The weight was maximal on the diagonal line and minimal in the top left and bottom right corners. The layerwise similarity was defined as the mean obtained in each layer.The similarity of the entire cascade and that of each layer were calculated from a pairwise similarity matrix. We wanted to evaluate the pairwise similarity matrix in a way in which high scores are obtained by a DNN with its lower, middle, and upper layers being similar to the peripheral, middle, and central brain regions, respectively. To realize this evaluation concept, we defined the cascade similarity as the weighted mean of the pairwise similarity matrix. The weight at the cell  denote the AM rate and tMTF, respectively. Integration was calculated with the trapezoidal rule. Q factors were calculated only in units with a band-pass tMTF.The Q factors of tMTFs were calculated as in a previous physiological study  as folloSpearman's rank correlation coefficients were calculated with the recognition performance and the cascade similarity. The sample size was 100 when comparing them in relation to optimization progress and 39 when comparing them across different model architectures.https://github.com/cycentum/cascaded-am-tuning-for-sound-recognition. Trained models and recorded activities are available at https://doi.org/10.6084/m9.figshare.7914611.The datasets used for the training and validation of the model are available from the cited studies . The modThe DNN was trained to classify raw sound data  of nonhuman natural sounds consisting of animal vocalizations and environmental sounds. Therefore, the model covers the entire range of auditory processes from the stage in the ear to the final recognition . This maa,b). A single unit responded differently to stimuli with different AM rates . We characterized the tuning of the units' activities to AM rate in terms of temporal and rate coding with tMTF and rMTF .To enable a direct comparison of our DNN and the auditory system, we simulated the experimental approaches of typical neurophysiological studies. Specifically, we conducted “single-unit recording” for each unit in the DNN while presenting a sinusoidally amplitude-modulated sound stimulus a,b. A siand rMTF , namely a shows MTFs of representative units in the first, fifth, ninth, and 13th layers. As in typical physiological experiments, we classified the MTFs into low-pass, band-pass, high-pass, or flat types according to certain criteria. Most units exhibited low-pass, band-pass, or flat MTFs (b). All of the MTFs in the first layer were flat, indicating that the first layer did not tune to AM rates. In the fifth layer, units with low-pass or band-pass tMTFs appeared and a very small number of units with low-pass rMTFs were observed. In the ninth and higher layers, the tMTF magnitude generally increased and the number of units with low-pass or band-pass rMTFs also increased. Heat maps of all tMTFs normalized by their peaks reveal a downward shift of the distribution of the preferred AM rates from the fifth layer to the highest layer and distinct tuning in the rMTFs appearing in the ninth layer and above (c).Figure 6lat MTFs b. All ofnd above c.a,b). In the third and fourth layers, some units exhibited definable tBMFs and tUCFs, but no rBMFs or rUCFs were definable. In the fifth layer, the tBMFs and tUCFs appeared to be high and a small number of units exhibited definable rBMFs and rUCFs. When ascending the layer cascade from the fifth layer, the mode tBMF/tUCF decreased and the number of units with definable rBMFs/rUCFs increased. In summary, the distributions of the tBMFs and tUCFs shifted toward lower AM rates when ascending from the middle to the high layers  and the units that code AM rate by their average activities appear only in the higher layers .As in typical neurophysiological studies, the MTF of a unit was characterized by its BMF, the AM rate at which a neuron exhibits the largest synchrony or average activity, and its UCF, the AM rate at which the synchrony or average activity starts to decrease. The BMF and UCF of temporal and rate coding are denoted tBMF/tUCF and rBMF/rUCF, respectively. In the first and second layers, no BMFs or UCFs were definable because all MTFs were flat a,b. In th layers a, left ac depicts the distributions of BMFs and UCFs in the auditory system by combining previously reported distributions for each of the seven brain regions: ANs  and averaged them to yield the layer–region pairwise similarity (b). Pairs consisting of a DNN layer and a brain region with a large similarity appeared in the diagonal direction, indicating that lower, middle, and higher DNN layers are similar to the peripheral, middle, and central brain regions, respectively. This similarity of the entire cascade is more clearly observed if we normalize the pairwise similarity by the maximum value in each brain region (c).Next, we compared those distributions quantitatively. For each tBMF, tUCF, rBMF, and rUCF, we calculated the similarity between the distribution in each layer of the DNN and the distribution in each region in the auditory system a and avemilarity b. Pairs n region c.a, left).The similarity of the entire cascade could be due to the convolutional architecture inherent to the DNN  or to opFig. 9b, top). Auditory-system-like AM tuning gradually emerged in parallel (a). We evaluated the similarity over all the cascades by measuring the degree of diagonality of the pairwise similarity matrix  and correlated very well with the classification accuracy . The results indicate that the AM representation in the DNN emerged during the optimization.As the optimization progressed, the classification accuracy increased as expected b, top. Aparallel a. We evay matrix  and we ry matrix b, bottoma) and the cascade similarity correlated with the classification accuracy . The results indicate that AM representation in better-recognizing DNNs have a greater similarity to that in the auditory system. The similarity to the auditory system correlated with the classification accuracy across both different model parameters and different architectures, which suggests that the auditory AM representation is strongly related to task optimization but not to the convolutional operation alone.Because the classification accuracy of a DNN generally depends on its architecture , so coulFig. 11a). This pattern was more clearly seen when we calculated the similarity to the auditory system in each layer, which we refer to as layerwise similarity . When the DNN was trained on shuffled labels, very few units in the middle layers appeared to exhibit AM tuning . When the DNN was trained on shuffled waveforms, units in the middle layers appeared to exhibit some AM rate tuning, but they synchronized with a much higher AM rate than neurons in the auditory system, resulting in the upper layers being similar to the middle brain regions . The results indicate that a midlevel AM representation requires a natural data structure, although low-level and high-level representations could emerge even by optimization with unnatural data.The first two control experiments tested the effect of the data structure. It has been shown that a DNN is capable of learning the input–output correspondence even when trained on data with random category labels or data without natural statistics . Under tFig. 12c). Animals do not usually follow a stimulus amplitude waveform precisely and the task is also trivial in the sense of signal processing. The AM representations in the middle to upper layers were to some degree similar to those of the middle brain regions, but no layers exhibited an AM representation similar to that of the central brain regions . In the upper layers, only small numbers of rMTFs exhibited clear tuning and the tBMFs and tUCFs were higher than those of the central brain regions . The result indicates that the emergence of auditory-system-like AM tuning in the upper layers requires natural objectives and the waveform-following task did not induce such a representation even if the input data consisted of natural sounds.The third control experiment examined the effect of the optimization objective. A DNN can be optimized for behaviorally irrelevant objectives such as a waveform following task c. AnimalTogether, the modification of the category labels, the sound statistics, and the optimization objective caused the auditory-system-like AM representations in the middle layers and above to deteriorate. Lower layers never exhibited AM tuning consistently across all conditions, probably because of the nature of the cascading architecture. The middle layers exhibited auditory-system-like AM tuning when trained on natural input sounds and the proper sound–category correspondence. The upper layers exhibited auditory-system-like AM tuning when optimized for the categorization task but not for the waveform following task .a).As a DNN trained on one dataset recognizes another dataset very well with only a slight modification , it may Fig. 13b). The lower, middle, and upper layers were similar to the peripheral, middle, and central regions, respectively. The similarity emerged during the optimization  and was weak under control conditions . The similarities in the DNNs with various architectures correlated with the classification accuracy . The results indicate that auditory-system-like AM tuning emerged robustly across different datasets.The speech dataset provided essentially the same conclusions as those obtained with animal and environmental sounds. The layer–region pairwise similarity matrix exhibited a diagonal pattern b. The loThus far, we have focused on the BMF and UCF for characterizing the MTF. Another aspect often considered is the sharpness of a tMTF, which is represented by quality factors (Q factors). Here, we calculated the Q factors of the unit tMTFs as in a previous physiological study . The disa). The responses generally increased as the input amplitude increased, but some units in the upper layers exhibited nonmonotonic responses to the input amplitude. As in the neurophysiological studies, a unit was characterized by an AF tuning curve, namely the minimum stimulus amplitude that provides a response larger above a certain threshold . Tuning curves from the first to third layers exhibited many troughs . Those around the fifth layer exhibited a small number of major troughs and many minor troughs. The major trough of a unit can be interpreted as exhibiting a band-pass property. The center frequencies of the major troughs of the unit population spanned a wide AF range , which may be interpreted as a band-pass filter bank. The tuning curves in higher layers were more complex without clear band-pass-like tunings. The overall results were in contrast to those for the auditory system, where neurons across many regions usually exhibit AF tuning with a relatively sharp single peak. This property is likely to originate from AF decomposition occurring in the cochlea. We did not explicitly implement the spectral decomposition of the input sound but directly fed raw waveforms to the DNN. The results suggest that AF decomposition in the cochlea may be essential for auditory-system-like AF tuning, but not for auditory-system-like AM tuning.We also examined tuning to the AF, which is the most frequently measured characteristic in auditory science . We presFig. 15We found that a DNN optimized for natural sound recognition exhibits an AM tuning similar to that of the auditory system throughout the entire cascade of layers. Because our DNN was not designed or trained to reproduce any physiological or anatomical properties of the auditory system, the results should reflect only the nature of the task and the data. Therefore, AM tuning in the auditory system might also emerge during evolution and development via optimization to sound recognition in the real world.Although DNNs have explained sensory representation in several modalities , to the From the perspective of machine learning, our results suggest the effectiveness of analyzing DNNs with physiological methods. To date, various methods have been proposed for analyzing representation in a DNN . Most ofThe result showing that there was little tuning in the peripheral region may be due to the architecture in which simple operations are cascaded. Computing the AM rate requires at least envelope extraction and the frequency decomposition of the envelope. A small number of peripheral brain regions are probably incapable of such nontrivial computation.Midlevel neural processing may be a necessary step if the brain is to form a proper stimulus representation for further processes in various tasks. Because lower layers do not tune to an AM rate, the middle layers are effectively the first layers that process AM signals. It is reasonable to think that the first stage of the data process is affected by the data structure and is critical for later recognition.A higher representation may be directly used for the final recognition process. In other words, whatever the stimulus representation is, the role of the central auditory regions is to derive appropriate outputs for the specific task.Both of the prominent characteristics of auditory AM coding, namely a decrease in synchronizing AM rate and time-to-rate conversion, involve a reduction in a signal's temporal resolution. Why is such a scheme beneficial for sound recognition? With our model, as in a typical recognition task with a DNN, the final output at each time frame is the category label assigned to the unit with the maximum activation at the time in the classification layer (the layer above the 13th layer). If the units synchronize with a fast AM, then the output category will be temporally unstable. Conversely, if the activation of a classification unit is large all the time, then the DNN will output a constant category over time. The latter case is preferable for the classification of sounds of a reasonable duration.d), at least we can conclude that proper setting for optimization is necessary for down-sampling properties to emerge in the DNN. Although we avoided making explicit assumptions on DNN parameters by learning them for sound recognition, the architecture of the DNNs with cascaded nonlinearity could induce some significant biases regarding the temporal properties of the extracted signals. Future experiments without half-wave rectification or convolution may reveal the effect of these operations.Strictly speaking, by avoiding the pooling operation that is often included in typical DNNs, the layers in our DNN do not necessarily down-sample the input. However, even without pooling, it is still possible that successive integration by convolution and half-wave rectification could bias the DNN toward implicitly extracting relatively clean temporal properties such as envelopes. From the result of the waveform following experiment d, at leaA in Fig. 8c). Also, although unitwise Q factors may not be directly comparable to those of psychophysical tMTFs, most Q values in our DNN were <1 and thus fall in the range suggested by psychophysical studies (In neurophysiological and psychophysical studies, tMTFs have also been characterized by their sharpness. One study has shown that most of the neurons recorded in the central nucleus of the IC exhibit Q factors between 0.5 and 1.5 (see supplemental figure S3 studies . These rUnlike neurons in the auditory system, our DNN did not exhibit sharp single troughs in the AF tuning curves, although some other studies have reported auditory-like AF tuning emerging in a DNN with a different architecture from ours . In the Several other computational studies have tried to explain auditory AM coding by using models with anatomical and physiological assumptions, including AF decomposition in a cochlea . In contIn this study, we analogized the auditory system with the optimized DNN. It is difficult to clearly identify the biological counterparts of the DNN optimization process. They probably include a mixture of the effects of short-term plasticity and long-time evolution over generations. Although it may be impractical to experimentally manipulate the long-time evolution in humans, studies with various AM detection or discrimination tasks in humans suggest that the responses of central auditory neurons to AM cues are plastic and that practice may modify the AM processing circuitry . It may


By combining the advantages of 360-degree field of view cameras and the high resolution of conventional cameras, the hybrid stereo vision system could be widely used in surveillance. As the relative position of the two cameras is not constant over time, its automatic rectification is highly desirable when adopting a hybrid stereo vision system for practical use. In this work, we provide a method for rectifying the dynamic hybrid stereo vision system automatically. A perspective projection model is proposed to reduce the computation complexity of the hybrid stereoscopic 3D reconstruction. The rectification transformation is calculated by solving a nonlinear constrained optimization problem for a given set of corresponding point pairs. The experimental results demonstrate the accuracy and effectiveness of the proposed method. Stereo vision systems have been widely used in tasks such as surveillance , search Generally, there are two types of methods to solve the aforementioned problem. One is to perform external calibration using the specific relation between the camera pairs or a large amount of known corresponding points. Under a local planar assumption, a non-linear approach for registering images in a hybrid vision system without requiring the calibration of cameras is proposed in . When thAnother method is to use pan-tilt-zoom (PTZ) cameras instead of conventional ones in the hybrid vision system, where it is not necessary to calculate the extrinsic parameters explicitly. Spatial mapping can be easily computed between an omnidirectional camera and a PTZ camera. Specifically, the pan-tilt angle of the PTZ camera is acquired by the corresponding points in the omnidirectional camera when the system is operating. This approach assumes that the pan-tilt angles are extremely correlated with the corresponding points. In some studies ,22, 3D rIn practical applications, for example, a hybrid vision system can be used for target tracking and observation tasks. The conventional camera will move as the position of the target changes to ensure that the target is within the common field of view of the omnidirectional camera and conventional camera. When the hybrid vision system is used in surveillance scenarios, the conventional camera will move as the monitored area changes. Therefore, automatic stereo rectification is highly desirable, while the accuracy is also needed. A perspective projection model is proposed for the omnidirectional image, which significantly reduces the computational complexity of 3D formulation for mixed-view pairs.A method based on a novel, well-defined cost function for optimizing the normalization matrix is employed, which can calculate the rectification transformation more accurately.To evaluate the performance of the proposed automatic rectification method and to provide a direct application, a target tracking and odometry hybrid vision system is established based on an automatic rectification approach.In our study, an automatic stereo rectification approach for one omnidirectional camera and one conventional camera is proposed. Compared with state-of-the-art methods ,27, the The remainder of this paper is organized as follows: The proposed automatic rectification approach for hybrid stereo vision system is presented in The block diagram of automatic stereo rectification for hybrid vision is shown in In  (ASIFT)  was used (ASIFT) .The main processing modules will be described in detail in the following sections.M can be determined uniquely by The configuration of the hybrid vision system in this paper is shown in To avoid the complex geometric relationship between the omnidirectional image and the conventional image, a novel perspective projection model for the omnidirectional image is proposed in this section. Unlike the conventional image, the generation of a virtual perspective image from an omnidirectional image is not one-to-one linear mapping—in this case, a simple perspective projection model is desirable.As described in , a centrAs shown in Supposing that point Equation (6) establishes the one-to-one correspondence between After obtaining the virtual perspective image, stereo rectification can be achieved by mapping the virtual and conventional images into the common plane and aligning the columns using rotation and translation matrices. These matrices can be obtained by decomposing the fundamental matrix. How to estimate the fundamental matrix robustly remains a challenging issue. Several methods for estimating the fundamental matrix have been proposed and can be classified into iterative and linear methods. Iterative methods ,32 are mIn our hybrid vision system, there is an affine transformation between the virtual image and the conventional image due to the different shooting angles of the omnidirectional camera and the conventional camera. Thus, ASIFT  was usedAs shown in rding to , the relThe relation of the essential matrix and the fundamental matrix is: offline . AccordiEquation (11) establishes the relationship between corresponding points in the image pair and the rotation and translation matrices. Supposing that corresponding points are known, the translation matrix In order to improve the stability of the 8-point algorithm, raw data was transformed into isotropic data using normalization matrices . HoweverIn our proposed method, we used a cost function to find the optimal normalization matrix and minimize the horizontal distance between corresponding pairs. The main processing algorithm is described in the following.by ASIFT . H and HThus, the error of epipolar geometry between two images can be expressed as:In addition to the epipolar geometry constraint, the horizontal distance between two images is another significant factor. The horizontal distance between pair correspondences is:According to (14) and (15), we define the objective function of the optimization problem as:In order to minimize ion (EM)  is adoptFrom (16), we can obtain the normalization matrices posed in .We first evaluated the performance of the proposed stereo rectification approach using 15 image pairs, where three of the original image pairs are shown in To represent rectification error quantitatively, we selected four stereo image pairs from the indoor environment that were rectified using the rectification method from  and our Y-axis of each point was used as a ground truth. The mean errors of distance in different orientations between cameras are shown in Stereo vision odometry is based on the parallax of two images. The accuracy of odometry represents the accuracy of the rectification method. Thus, we analyzed odometry accuracy in a simulated environment. As shown in ethod in , respectTo illustrate the performance of the proposed rectification method and provide a direct application, we used the hybrid vision system to track a target with a size of 1.7 × 0.6 × 0.3 m and perform odometry. The tracking algorithm for the omnidirectional camera in  was adopFive frames were randomly selected from the processing results, and are shown in We used our proposed method to rectify image pairs, the results of which are shown in In summary, we demonstrated an automatic rectification approach for the hybrid vision system. The geometric relationship between omnidirectional and conventional images was simplified by generating a virtual perspective image from the omnidirectional image. Image pairs were rectified using the 8-point algorithm with an optimized normalization matrix. We showed that the row coordinate parallax of rectified image pairs was within 2 pixels. The mean errors of the odometry based on triangulation were less than 12%. Based on this, we consider that the error of rectification is acceptable when compared with other methods in literature. Deng et al.  also useIt is noteworthy that the number of corresponding points strongly affects the rectification performance, because the fundamental matrix is estimated based on their position. The experiment in no salient feature scene was performed. The error of the fundamental matrix was very large since almost no corresponding points were detected.In the future, we would like to use our approach for different camera combinations, including omnidirectional-fisheye and fisheye-conventional. Additionally, we plan to develop an improved method which is not limited by various features in the scene.In this paper, an active hybrid vision system consisting of an omnidirectional camera and a conventional camera was presented. We provided the key techniques to rectify image pairs automatically. The virtual perspective image from an omnidirectional image was obtained using the proposed perspective projection model. ASIFT and the 8-point algorithm with an optimized normalization matrix were applied to rectify the image pair automatically. Our investigation in the simulated and real environment has demonstrated that the proposed approach not only overcomes the shortcomings of high computational complexity in the hybrid vision system, but is superior to other state-of-the-art methods  in regar


We developed a temporal population receptive field model to differentiate the neural and hemodynamic response functions (HRF) in the human lateral geniculate nucleus (LGN). The HRF in the human LGN is dominated by the richly vascularized hilum, a structure that serves as a point of entry for blood vessels entering the LGN and supplying the substrates of central vision. The location of the hilum along the ventral surface of the LGN and the resulting gradient in the amplitude of the HRF across the extent of the LGN have made it difficult to segment the human LGN into its more interesting magnocellular and parvocellular regions that represent two distinct visual processing streams. Here, we show that an intrinsic clustering of the LGN responses to a variety of visual inputs reveals the hilum, and further, that this clustering is dominated by the amplitude of the HRF. We introduced a temporal population receptive field model that includes separate sustained and transient temporal impulse response functions that vary on a much short timescale than the HRF. When we account for the HRF amplitude, we demonstrate that this temporal response model is able to functionally segregate the residual responses according to their temporal properties. Over the past 15 years, researchers have characterized the retinotopic organization of the human LGN in vivo ,3,4,5. ARecent attempts have been made to segment the LGN into its M and P subdivisions ,9,10 usiHowever, to properly account for the neural function in the LGN using fMRI, we must first account for potential differences in the hemodynamics throughout the LGN. Oxygenated blood from the internal carotid artery reaches the LGN by way of the anterior choroid and posterior cerebral arteries. These vessels ramify and enter the LGN at the hilum on its ventral surface about halfway along its medial-to-lateral extent. The hilum is richly vascularized, with the tissue of the LGN surrounding the hilum receiving approximately twice the blood supply as the rest of the LGN . The hilThe goal of this study was to develop a spatiotemporal population receptive field (pRF) model to capture the functional divergence between the M and P subpopulations of the LGN. By adding a temporal parameter to the spatial pRF model, it is possible to model and account for variations in neural function at timescales much faster than are typically observed in the slow hemodynamic response. Using a stimulus that varied in visuotopic location and flicker frequency over time, we sought to characterize each voxel both in terms of its spatial and temporal response and in doing so functionally segment the LGN. The parameter maps we measured indicate that this simple modification to the spatial pRF model is able to capture functional differences among the M and P layers of the LGN. In follow up experiments, we compared these model-based functional segmentations of the LGN to model-free, data-driven segmentations. We designed stimuli with no spatial component and attempted to differentiate the subpopulations of the LGN on the basis of the pattern of their activity in response to random changes in flicker frequency, spatial frequency, and isoluminance in a full-field visual stimulus. We compared these cluster labels to our spatiotemporal pRF model estimates to ascertain the effectiveness of functionally segmenting the layers of the LGN on the basis of temporal frequency tuning. The results suggest that (1) a spatiotemporal pRF model can capture functional differences across the M and P layers of the LGN, and (2) that such a model needs to include a scaling term in order to account for the difference in the hemodynamics across the extent of the LGN.Three subjects participated in the study . One subject was left-handed. The subjects were in good health with no history of psychiatric or neurological disorders, had normal or corrected-to-normal vision, gave their informed written consent, and were compensated for their participation. The study was approved by the York University Human Participants Review Committee, Certificate #2014-300. All of the subjects participated in 10 scanning sessions, including one anatomical and nine functional.The stimuli were generated on a Mac Pro i7 computer  with a NVIDIA Quadro 4000 video card, using Matlab software  and Psychophysics Toolbox 3 functions ,13. StimEach subject participated in three separate functional imaging experiments . The stiThe stimulus for the second experiment consisted of a checkerboard texture spanning the entire extent of the display B. The luThe stimulus for the third experiment also consisted of a full-field stimulus C, but hek-space to reduce acquisition time. The posterior edge of the acquisition volume was aligned in the midsagittal plane with the posterior edge of the superior colliculus. The subjects’ heads were surrounded by foam padding to reduce head movements. The distortions commonly found in T2* GE-EPI were significantly attenuated due to the deep location of the structures of interest, the short TE, the small voxels, and the phase-encoding direction.Data were acquired in the York University Neuroimaging Laboratory with a 3 T Siemens Trio MRI scanner  using a 32-channel head coil. Functional series were acquired with 14 coronal slices (1.5 mm isotropic voxels with 0.5 mm gap between slices) and a gradient echo, echo planar sequence with a 128 square matrix leading to an in-plane resolution of 1.5 × 1.5 mm . A partial Fourier factor of 7/8 was used to acquire an asymmetric fraction of 1-weighted MPRAGE  and 40 coronal proton-density weighted volumes  were acquired. Each proton-density image was motion-corrected to a single proton-density image, from which a mean proton-density image was computed and up-sampled by a factor of 2. These images were aligned to the MPRAGE and used for drawing ROI.In addition to the functional scanning sessions, each subject participated in an anatomical scanning session. During the anatomical session, a high-resolution TThe first four volumes of each run were discarded. To compensate for subject head movement, the remaining volumes were registered to a single volume obtained during the same scanning session. In addition to motion-correcting the functional imaging data, we also employed a volume-censoring procedure . The frat0 with a dispersion We modeled the BOLD signal of each voxel in terms of a spatiotemporal population receptive field. The spatial response of each voxel was characterized as a two-dimensional Gaussian in visuotopic coordinates ,17. It desponses ,19,20, wcaque V1  and tempcaque V1 .r2 > 0.1) in a yoked fashion, so that while the 6 spatiotemporal pRF model parameters In addition to these model parameters, we also included other model parameters for capturing the delay of the HRF and dispersion of the temporal RF across the LGN within a single subject. We estimated the double-gamma HRF delay  paramete the HRF C to apprThe goodness of fit between the model prediction and the measured BOLD signal was assessed via the residual sum of squared error (RSS). The time-series of each voxel was fitted in a two-phase procedure. The first phase consisted of a sparse and coarse global grid-search that adaptively constrained the search boundaries. The effective stimulus was down-sampled to a resolution of 5% of the original, using a two-dimensional bilinear interpolation. An adaptive brute-force search strategy was used to sparsely sample the model parameter space and the best fit of this was used as the seed point for a fine-tuned gradient descent error-minimization using the non-resampled stimulus. The gradient-descent procedure used was a downhill simplex algorithm  to fine-d models .To assess the reliability of our pRF model estimates, we used a bootstrapping procedure where we repeatedly fit the pRF model to the voxel-wise mean time-series computed from a variety of functional run sample sizes. For every voxel in each subject’s LGN, we randomly sampled without replacement a number of runs (ranging from 2–29), estimated the pRF model, and computed an error score by computing the difference between each bootstrapped pRF model fit against the full 30-run pRF model estimate. We repeated this 200 times for each voxel and sample size. The error between the resampled pRF estimate and the target pRF estimate was compared across sample sizes and subjects.The spatiotemporal pRF model parameter lgorithm , specifyp-values are reported for all voxel parameter regressions. FMRI Voxels are typically not independent; however, the BOLD point spread function (PSF) in the LGN is much smaller than in the cortex—the best estimate is 1.6 mm  showed that the eccentricity  and amplitude  best explained the variation in the model-free labeling of the LGN. Regression of the pRF estimates against the cluster labels derived from the full-field isoluminance stimulus  revealed that eccentricity , receptive field size , transient-sustained weight mixture , and amplitude  all significantly covaried with the cluster labeling in the LGN. Finally, we regressed the pRF estimates against the full-field flicker and isoluminance stimuli concatenated in time . We found that the amplitude  and baseline  both significantly covaried with the cluster labeling in the LGN. Interestingly, there was no significant correlation between our weight and amplitude estimates , indicating that these parameters are unrelated.We performed a logistic regression to determine which of the pRF model parameters best explained the cluster labels derived from the full-field datasets . A regreThe pRF model has been used to explore and describe the visual responses measured in a number of subcortical  and cortWe developed a spatiotemporal pRF model that includes both spatial and temporal receptive fields to capture differences in the response properties of neurons in the M and P layers of the LGN. While difficult to measure because of its small size and deep location in the brain, the unique functional and structural organization of the LGN offers a rich experimental arena in which to develop and test models of information flow and dysfunction throughout the visual system. Among their distinguishing characteristics, the M and P layers of the LGN are dissociable based on their response profiles to various stimuli. In particular, we sought to exploit the differences in the temporal frequency tuning and discharge patterns of M and P neurons. We extended the spatial pRF model to include a pair of temporal receptive fields, where the ideal P response was modeled as a Gaussian in time and the M response was modeled its first derivative. These impulse response functions were deliberately chosen to be simple and were not tuned to optimize the discrimination of the M and P layers of the LGN. The aggregate BOLD response was taken to be the weighted sum of the transient and sustained responses, in combination with the spatial and hemodynamic components. The functional form of the temporal RF helps to link the neural response properties of temporal frequency tuning and the discharge pattern. A transient discharge pattern—the rapid cycling of neural activity—acts as an edge detector in time and so confers the property of high temporal frequency tuning. A sustained discharge pattern—tonic activation during the presentation of a stimulus—acts as a low-pass temporal filter and confers the property of low temporal frequency tuning. When a stimulus with varied flicker frequencies is passed through the temporal RFs, the resulting response will demonstrate a frequency-driven amplitude modulation over time. Thus, the spatiotemporal pRF model estimation affords information about how a given voxel is tuned in space and time.Although temporal frequencies could be analyzed using a traditional GLM, the advantage of the pRF model is that we can apply the model to analyze the response to any stimulus, rather than having to craft a particular stimulus for each feature we would like to analyze. In addition, we could readily add other response features to the model, such as color or eye of origin. Each feature could be measured independently using a GLM, but if the stimulus features are separated in the stimulus, confounding factors such as attention can interfere. Additional theoretical questions could be addressed by the pRF but not GLM models such as questions of spatial pooling and differential center-surround temporal modulations.The retinotopic organization of our LGN maps is consistent with our previous work. All LGN showed strong contralateral representation of the visual field, with the lower vertical meridian representation along the superior-medial LGN, the upper vertical meridian representation along the inferior-lateral LGN, and the contralateral horizontal meridian bisecting the two about approximately 45°. The eccentricity and receptive field size maps showed a strong positive relationship such that voxels at the posterior pole of the LGN had small, foveal receptive fields and voxels at the anterior pole of the LGN had large, peripheral receptive fields. These findings demonstrate that the spatial component of the spatiotemporal model can be faithfully estimated with a stimulus that encodes multiple features.In addition to the spatial tuning, we were also able to recover the topographic organization of the temporal tuning across the LGN. Among our three subjects, we found a smooth gradation of the weight parameter estimates along the lateral-superior to medial-inferior axis of the LGN. Specifically, we found that voxels along the lateral-superior portion of the LGN contained voxels with sustained-rich weight estimates and voxels along the medial-inferior portion of the LGN contained voxels with transient-rich weight estimates. The configuration and orientation of these weight parameter maps is generally congruent with the known structural and functional organization of the LGN.β1−β2)/(β1+β2) instead of analyzing the absolute difference β1−β2. The pRF modeling approach we propose explicitly includes both the transient-sustained components of the BOLD signal as well as the amplitude of the BOLD signal for each voxel. That we found no correlation between the weight and amplitude parameters suggests that the weight parameter is sensitive to the temporal response properties across the extent of the LGN, independent of the changes in amplitude.Previous functional imaging experiments aimed at dissociating the M and P layers of the human LGN found a similar organization of the regression coefficients computed from a GLM using a battery of M-preferred and P-preferred stimuli ,9,10 incOther recent work has focused on characterizing the effects of achromatic flicker frequency on the BOLD response in human primary visual cortex (V1) and the LGN . In thisTo determine how much of the variation in the activity of the LGN the spatiotemporal pRF model captures, we performed two additional experiments. Here, we removed any spatial component from the stimulus and presented subjects with full-field checkerboard stimuli whose flicker frequency, spatial frequency, and red-green isoluminance were varied so as to maximally differentiate the underlying M and P response in the LGN. We then used model-free clustering and decomposition techniques in order to determine if we could replicate the topographies of the pRF model parameter estimate maps obtained in Experiment 1. The results of our hierarchical clustering revealed a functional organization of the cluster labeling that roughly agreed with our amplitude estimate maps. We performed several regressions to determine the extent to which the parameter estimates of the spatiotemporal pRF model explain the variation in the cluster labels derived from full-field flickering and isoluminance data, and the concatenation of these two datasets. While we found small variations in the cluster label organization across the different cluster label experiments, we did find that there was a consistent significant relationship between the cluster labels and the amplitude pRF model estimate. We interpret the amplitude to represent the hilum region of enriched vasculature in the center of the medioventral surface of the LGN . The ampr2 > 0.1. This could potentially place irreconcilable time constraints on clinicians scanning patients, although modifications could be made to the stimulus regime to sacrifice spatial precision for temporal sensitivity. Furthermore, only approximately one-third of the voxels in the anatomically defined LGN survived the statistical threshold. This represented about 25% reduction compared to a previous study using pRF modeling in the LGN [Part of the motivation for developing methods to parse the layers of the LGN was to provide a framework to test the magnocellular theory of dyslexia. However, the substantial amount of data collected for these experiments may limit the viability of using such an approach in a clinical setting. To address this issue, we conducted a bootstrapping procedure where we estimated the stability of the spatial and temporal pRF model parameters as sample sizes increases. We found that the stability of the spatial parameters quickly saturated after approximately 10 resamples, and the temporal parameter reached 90% stability after approximately 20 scanning runs. In other words, 75–100 min of functional scan time is required to detect the transient-sustained functional gradient of the human LGN using a statistical threshold of We have developed a spatiotemporal pRF model to account for the responses of the human LGN. We found that the response amplitude is dominated by the rich vascular hilum region, which drives unsupervised clustering attempts. Once this amplitude was parameterized, we were able to measure a gradient of temporal sustained to transient responses that can be attributed to the parvocellular and magnocellular subdivisions of the LGN. The temporal pRF model is flexible and simultaneously accounts for multiple parameters for a given stimulus, and additional parameters accounting for other stimulus parameters or neural features can be added.


Indoor navigation and location-based services increasingly show promising marketing prospects. Indoor positioning based on Wi-Fi radio signal has been studied for more than a decade because Wi-Fi, a signal of opportunity without extra cost, is extensively deployed for internet connections. Bayesian fingerprinting positioning, a classical Wi-Fi-based indoor positioning method, consists of two phases: radio map learning and position inference. Thus far, the application of Bayesian fingerprinting positioning is limited due to its poor usability; radio map learning requires an adequate number of received signal strength indication (RSSI) observables at each reference point, long-term fieldwork, and high development and maintenance costs. In this paper, based on a statistical analysis of actual RSSI observables, a Weibull–Bayesian density model is proposed to represent the probability density of Wi-Fi RSSI observables. The Weibull model, which is parameterized with three parameters that can be calculated with fewer samples, can calculate the probability density with a higher accuracy than the traditional histogram method. Furthermore, the parameterized Weibull model can simplify the radio map by storing only three parameters that can restore the whole probability density, i.e., it is not necessary to store the probability distribution based on traditionally separated RSSI bins. Bayesian positioning inference is performed in the positioning phase using probability density rather than the traditional probability distribution of predefined RSSI bins. The proposed method was implemented on an Android smartphone, and the performance was evaluated in different indoor environments. Results revealed that the proposed method enhanced the usability of Wi-Fi Bayesian fingerprinting positioning by requiring fewer RSSI observables and improved the positioning accuracy by 19–32% in different building environments compared with the classic histogram-based method, even when more samples were used. Because of their social and commercial value, indoor location-based services (ILBS), which are predicted to be worth US$10 billion by 2020 and US$58 billion by 2023 , RFID 5); opticaAccordingly, many studies have developed different Wi-Fi-based indoor positioning approaches ,12, suchIn contrast, fingerprinting positioning exploits the signature of environmental variations in RSSI observables as effective measurements and therefore does not need to know the locations of the APs ,19. HencBased on the above discussion, this study proposes a Bayesian density model to represent the probability density of Wi-Fi RSSI samples of a specific location. The proposed Bayesian density model is based on the Weibull function, which contains three parameters. With the proposed model, the Bayesian fingerprinting approach is enhanced in three aspects. First, the three parameters of the Weibull function can be estimated with a limited number of RSSI samples, and the resulting Weibull–Bayesian density model can represent the probability density with a much higher accuracy than can be achieved by classic Bayesian fingerprinting positioning methods, such as the histogram  or kerneLocation fingerprinting is mainly based on a fingerprint database of target features for their identification, and the positioning process can be divided into two phases: radio map learning and position inference ,34,35.The first phase is the offline radio map learning phase, the main purpose of which is to establish a number of reference points in the target area. Following this, a smartphone can be used to collect the received signal characteristic parameter data  from multiple APs at each reference point. These parameters, in addition to the position coordinates of the point, form a set of data stored in the database , and thei-th reference point. Here, i-th reference point. The entire fingerprint database is expressed by the following equation:i-th reference point, the probability of an RSSI measurement falling within bin mA can be expressed as follows:n is the number of bins; In the conventional probabilistic approach, the RSSI probabilities of all APs received at each reference point are stored in the Bayesian fingerprinting positioning database ,41. In turements . In the The second phase is the online position inference phase. Using the smartphone that receives the same signal parameters stored at the positioning point, we use the corresponding matching algorithm to determine which fingerprint data in the fingerprint database are the most similar. Then, we use the most similar sets or groups of fingerprint data corresponding to the matching algorithm, and those data can be used to estimate the actual user location .Two classic types of methods, namely, the deterministic approach and the probabilistic approach, are currently utilized . The detHowever, because Wi-Fi signals are susceptible to interference within indoor environments, the signal space composed of Wi-Fi RSSI measurements does not exhibit a one-to-one mapping relationship with the physical location . TherefoThe algorithm proposed in this paper is based on the probabilistic Bayesian position estimation method, and some improvements to existing problems in this method are proposed herein to enhance the accuracy and usability of the algorithm. In this paper, we optimize both the radio map learning phase and the position inference phase of the Bayesian probabilistic algorithm.Bayesian inference in models employed to generate a density estimation is usually described using a mixture of the Dirichlet process . These mUnderstanding the statistical characteristics of the RSSI probability distribution is fundamental ,50 to imIn this study, a sampling test was performed over a period of 12 h in an indoor environment, namely, in the official building of the Central Creative Building covered with Wi-Fi signals, and more than 17,000 RSSI samples were obtained. An interesting distribution can be observed in Accordingly, the cumulative distribution function (CDF) can be expressed as follows:arameter ,42.The parameters of the Weibull signal model can be estimated with a limited number of sampled RSSI measurements. The model parameters k=.5≤k≤2.5 ,47.Therefore, the distribution probability of each possible RSSI measurement in the fingerprint database can be expressed as follows:w is the width of the bin, and x is the RSSI value at the left boundary value of bin.For a fingerprint database measured with the Weibull signal model, we can calculate the probability of occurrence of any RSSI measurement. As the RSSI measurements are integers, the probability for each bin in the fingerprint database can be generated as follows:Am and a smartphone at a reference point The fingerprinting method using the radio map based on the Weibull–Bayesian density model can be represented by a set of Weibull signal models that simulate the distribution of RSSI measurements. Each Weibull signal model contains three parameters –(10) were used with all of the RSSI samples to calculate the parameters of the Weibull signal mode derived from the entire set of samples as follows: shape During the position inference phase, the bins were dynamically divided based on the Weibull signal model according to the RSSI measurement of each AP received in real time. Therefore, with the same fingerprinting method, we could compare the probabilities generated from the same data set using three different algorithms: the conventional Bayesian fingerprinting histogram algorithm with static bin; the Weibull bin fingerprinting algorithm based on the Weibull–Bayesian density model with static bin; and the Weibull PDF fingerprinting algorithm based on the Weibull–Bayesian density model with dynamic bin.x when the RSSI measurement is y; y also reaches a maximum. In other words, the best match with the anchor point can be used as the positioning result output. Therefore, the Bayesian probability algorithm is used to find the maximum value of The fingerprinting positioning method employed in this paper relied on Bayesian theory and the histogram maximum likelihood algorithm . The prij-th AP received by the smartphone at the anchor point; x is a reference point in the fingerprint database. Therefore, the conditional probability product of all APs at each reference point can be calculated, the maximum probability can be found according to the histogram maximum likelihood algorithm, and the corresponding reference point is the positioning result.To obtain the maximum value of In this paper, the conventional static bin method stored the probability distribution of the RSSI measurements of each AP received at each reference point in a well-defined bin as a radio map. In contrast, the dynamic bin method proposed in this paper added ±B dBm to the RSSI measurements of each AP received during the positioning phase, thereby incorporating a dynamic bin to account for the probability in Equation (5). Accordingly, we calculated a large number of Wi-Fi APs RSSI measurements and found that most of them had a standard deviation between 2 and 3, and 95% of the measurements errors were within twice the standard deviation if the errors followed the Gaussian distribution, according to the theory of the statistics. Thus, the parameter B was given the value of twice the standard deviation. Considering the differences between different APs, an experiment was carried out to compare the positioning accuracy with ±4 dBm, ±5 dBm, and ±6 dBm to identify which would be the best as the value of B. We used the proposed algorithm to calculate the average error and root-mean-square (RMS) error of each RSSI range and plotted the error distribution graphs with ±4 dBm, ±5 dBm, and ±6 dBm in addition to the probability cumulative distribution function of the positioning error. More details about the experiment will be introduced in mA received by the smartphone at reference point Am is Am and a smartphone at a reference point The three parameters required by the Weibull–Bayesian density model are stored in the radio map to first model the probability distribution of the RSSI measurements between an AP In this positioning experiment, the average error and root-mean-square error in the positioning results of the three algorithms were compared in different actual scenarios. The objective of the experiment was to compare the positioning performance of the two Weibull-based algorithms (Weibull bin and Weibull PDF) with that of the conventional histogram algorithm.In this experiment, two floor plans were selected to verify the broad applicability of the proposed algorithm in various environments. The two experiments were carried out in public areas on the second and fourth floors, both of which are 53 m by 73 m in size, within the office building of our laboratory. The second floor is mainly composed of a lobby, students’ computer labs, and multiple corridors  with 30 RSSI samples, while the Weibull-based solution was derived from Equation (12) with 20 out of 30 RSSI samples. The experimental results are presented in Examining the cumulative probability distributions in the error graph, the positioning results for the data from the second and fourth floors using the Weibull PDF algorithm  of signal observables and improved the positioning accuracy by 19–32% in different building environments compared with the classical histogram-based method. In general, the new method proposed in this paper exhibits good prospects.However, the presented results are only for Wi-Fi fingerprinting positioning, and the proposed method has not been integrated with pedestrian dead reckoning (PDR) or other localization sources; hence, the positioning accuracy is not fully up-to-date. In the future, we will conduct additional studies on multisource integrated positioning. For instance, the fusion of Wi-Fi fingerprinting with PDR and maps will result in a better positioning accuracy.


The authors report an unusual complication of acute infective endocarditis found in a 70-year-old man with a previous history of two cardiac surgery procedures. During median sternotomy, a massive bleeding occurred. The bleeding was contained within the mediastinum and originated from the 10-year-old anastomosis of the saphenous vein to a composite graft, which was completely detached due to infective endocarditis of the aortic graft. Infective endocarditis (IE) is an important cause of cardiovascular pathology that, despite diagnostic and therapeutic advances, continues to be a serious disease that leads to a high morbidity and mortality.We report the case of a 70-year-old man with an episode of IE affecting a composite aortic root and mitral prosthesis, who had an extremely high surgical risk, in whom a previous saphenous vein graft was found to be detached from the aortic graft because of endocarditic involvement, forming a huge collection of blood in the anterior mediastinum.A 70-year-old man was emergently referred to the Cardiac Surgery Department due to a confirmed diagnosis of acute IE. The patient had a previous clinical history of arterial hypertension and atrial fibrillation. He had two previous cardiac surgery procedures. He underwent, 10 years ago, a full aortic root replacement with a composite graft  associated with triple coronary bypass (IMA to anterior descending artery and vein grafts to intermediate branch and posterior descending artery).Staphylococcus epidermidis). He was admitted in cardiogenic shock with confirmed endocarditic involvement of the mitral valve. The critical preoperative status did not permit a preoperative angiogram. No signs of myocardial ischemia were present, so the patient underwent an emergent mitral valve replacement by a mechanical prosthesis performed through a right thoracotomy approach, to avoid possible complications related to the previous coronary grafts. The postoperative course was uneventful, and the patient was discharged home after completing 6 weeks of intravenous antibiotic treatment with daptomycin and rifampicin.The second procedure was performed 6 months before the current episode, when the patient suffered from a native mitral valve IE episode  scan was performed, which revealed a huge collection of unknown origin in the anterior mediastinal space (100 × 55 × 75 mm), adjacent to the Dacron composite graft, in its anterior aspect, and in close contact with the thoracic wall . This wBefore opening the sternum, because of the suspected high risk of rupture, arterial cannulation was performed in the axillary artery and venous cannulation in the femoral vein. Cardiopulmonary bypass and hypothermia were established. During the sternal opening, a massive acute bleeding occurred. The bleeding was partially controlled by occlusion of the bleeding point by digital pressure through the partially opened sternum. After completing the sternal opening, the site of bleeding was identified. It was caused by the complete detachment from the aorta of the previous vein graft anastomosis, due to endocarditic involvement. The institution of cardiopulmonary bypass before sternal opening allowed the medical team to maintain a stable hemodynamic situation, and the bleeding site was controlled.All the affected tissues were excised, including debridement of all infected and necrotic regions. The patient underwent a mitral valve replacement and a full root replacement with a mechanical composite graft (Medtronic Inc.). It was not possible to mobilize the native coronary ostia because of the firm adhesions due to the previous surgeries and the severe IE, so the new Bentall-De Bono procedure was performed with the Cabrol modification,Klebsiella pneumoniae(Carbapenemases producer) that triggered multiorgan failure with acute renal failure, coagulopathy due to low flow hepatic failure, intestinal ischemia, and respiratory involvement. He died due to an acute multifactorial etiology shock refractory to medical treatment.During the first 24 hours after the intervention, it was possible to reduce the dose of inotropic support because of the stable hemodynamic state of the patient. Unfortunately, on the second postoperative day, the patient had fever and worsening of infectious parameters. The patient suffered from an episode of septic shock due toThe presence of IE perivalvular complications increases the complexity associated with IE surgery.We report a case with an extremely high surgical risk, which was a second episode of IE, with an estimated 30-day mortality by EuroSCORE I and II of 81.6 and 42.6%, respectively. The case was discussed in the multidisciplinary endocarditis team, with the collaboration of the cardiologist, microbiologist, and cardiac surgeon, and despite the extreme surgical risk, the decision to proceed with surgery was accepted, as it was the only chance for survival. Surgery appeared to be the only possible strategy based on the clinical management guides.Moreover, we found a very infrequent complication of IE: the total detachment of a previous vein graft anastomosis ought to an endocarditic involvement of the ascending aortic graft, with the formation of a large collection of blood contained by the adhesions of the previous surgeries. Prosthesis detachment due to IE affectionOwing to the absence of similar reports in the previously published literature, we consider that this case is a good example of an extreme IE affectation. Moreover, our findings may help the planning of the surgical strategy in the unusual case of a mediastinal collection in an IE patient affecting a previous full-root graft and saphenous vein grafts. Cannulation for cardiopulmonary bypass before opening the sternum allowed the surgical team to control the potentially fatal complication of massive bleeding. Peripheral cannulation prior to sternal opening could be considered mandatory in advanced IE cases, when the destruction of outer heart structures is suspected.


This paper presents LocSpeck, a collaborative and distributed indoor positioning system for dynamic nodes connected using an ad-hoc network, based on inter-node relative range measurements and Wi-Fi fingerprinting. The proposed system operates using peer-to-peer range measurements and does not need ultra-wideband (UWB) fixed anchor, nor it needs a predefined network topology. The nodes could be asymmetric in terms of the available sensors onboard, the computational resources, and the power capacity. This asymmetry adversely affects the positioning performance of the weaker nodes. Collaboration between different nodes is achieved through a distributed estimator without the need of a single centralized computing element. The ranging measurement component of the system is based on the DW1000 UWB transceiver chip from Decawave, which is attached to a set of smartphones equipped with asymmetric sensors. The distributed positioning filter fuses, locally on each node, the relative range measurements, the reading from the internal sensors, and the Wi-Fi received signal strength indicator (RSSI) readings to obtain an estimate of the position of each node. The described system does not depend on fixed UWB anchors and supports online addition and removal of nodes and dynamic node role assignment, either as an anchor or as a rover. The performance of the system is evaluated by real-world test scenarios using a set of four smartphones navigating an indoor environment on foot. The performance is compared to that of a commercial UWB-based system. The results presented in this paper show that weak mobile nodes, in terms of available positioning sensors, can benefit from collaboration with other nearby nodes. Indoor positioning and localization systems constitute a broad research area that spans different sensing techniques and a multitude of position estimation techniques. The field has been experiencing continuous growth in the last few years, from both the research perspective and equally from the commercial perspective, with the proliferation of positioning and localization systems in indoor environments and the rapid adoption of location-based services (LBS) and real-time location systems (RTLS) in commercial, industrial, emergency response, and military settings . In someThe objective of this paper is to present LocSpeck, a collaborative and distributed positioning system targeting smartphones and handheld applications, which uses UWB-based relative range measurements, along with Wi-Fi fingerprinting and inertial sensors. The system is evaluated experimentally, and the results are compared to the performance of Pozyx, a commercial UWB-based positioning system. The rest of this section will provide a brief overview of different ranging and positioning techniques used in UWB-based systems, and then it will discuss the different network architectures used for range-based positioning applications.Ultra-wideband positioning and localization systems have been used in different scenarios that require centimeter ranging accuracy with constraints on the cost and the power of the ranging devices. The applications of UWB-based localization systems include first responders in emergency situations, assets tracking and monitoring, medical and wellness applications, security and access control, locating nodes in wireless sensor networks, and for military applications ,7,8,9.Ranging using ultra-wideband radios can be performed using different techniques: angle-of-arrival (AOA), received signal strength (RSS), time-difference-of-arrival (TDOA), or time-of-arrival (TOA)/time-of-flight (TOF) . The angThe Cramér–Rao bound for the time-of-arrival ranging accuracy using the IEEE 802.15.4a, the predecessor to the IEEE 802.15.4-2011, under single-path additive white Gaussian noise (AWGN) channel model, can be expressed as:se ratio ,11. Sett σR=2 cm .The RSS of the UWB signal is less susceptible to small scale fading compared to narrow-band signals  as a resg effect . This cog effect ,15. The g effect , which rThanks to the high accuracy of UWB time-based range measurements, positioning with a decimeter-level of accuracy is usually achievable by solving a multilateration problem in line-of-sight (LOS) working conditions , i.e., wSince non-line-of-sight (NLOS) measurements are quite frequent in indoor environments, several recent works consider the problem of identifying NLOS measurements  or dynamThe integration with the information provided by other sensors can be considered to mitigate the UWB measurement error and regularize the estimated trajectory, e.g., pedestrian dead reckoning based on the inertial sensor measurements . FurtherIn contrast to the static network architecture case, this paper considers a network that can dynamically self-adapt to the number of available UWB devices. The main advantage of this approach is that it does not require any a priori knowledge of the number of nodes in the network, hence perfectly adapting to the potentially real scenario where at each time instant users can enter or exit the considered area with tracking devices equipped with UWB transceivers. This work aims at investigating the positioning performance in such case, where the use of a fixed UWB architecture is not required, hence leading to a significant system cost reduction, and the use of a collaborative and approach is expected to support the positioning performance of weak nodes, e.g., nodes provided with fewer sensors/less informative measurements.The IEEE 802.15.4-2011 standard supports two network architectures for UWB devices in the context of the personal area network (PAN), as shown in Another aspect of a UWB network architecture is the design of medium access control (MAC) protocols, which has been studied extensively in the context of wireless sensor networks (WSNs). The design of the MAC protocol for sensor networks is guided by the operational goals of the network, and it is usually an attempt to balance between two conflicting goals: achieving high-throughput and maintaining energy-efficient operation. Recent survey articles describe the taxonomy of MAC protocols and outline the development of different MAC protocols ,35,36,37The rest of this paper describes the architecture of LocSpeck, a distributed and ad-hoc UWB-based positioning system, along with the hardware and software components of the nodes. This section describes the hardware and the software aspects of the proposed ad-hoc UWB-based positioning system. The system supports ad-hoc network architecture, run-time inclusion and removal of nodes, and dynamic role assignment to nodes. Furthermore, the proposed system does not depend on fixed anchors nor requires time synchronization between the collaborating nodes. The section starts by describing the supported network structure, then follows by describing the hardware and software underpinnings of the system. The details of the ranging messages and the medium access protocol are discussed later in this section.Flat network topology: the network is composed of symmetric nodes in terms of its communication capability, which means that each node can initiate a ranging request or respond to such requests from other nodes. In addition, there are no coordinating nodes as opposed to the peer-to-peer network architecture described in the IEEE 802.15.4-2011 standard ,40. HoweSingle-hop network: the nodes are only interested in exchanging ranging messages with their neighboring nodes.Energy conservation: after either a failed or a successful ranging exchange attempt, the radio chip goes to sleep for a predefined period of time before it can engage in a new ranging sequence.Flexibility: nodes can enter and exit the network in real-time, with no need to reconfigure or notify the existing nodes.An ad-hoc network is a network between independent nodes that can change its connectivity dynamically without the need for fixed infrastructure or central control units . The nodThese characteristics emphasize the main objective of the network: ranging and positioning. The nodes are identical, in terms of communication capabilities, and could be spread over a large physical space. The differences between the ad-hoc network topology and the fixed-role network topology for ranging and positioning applications can be seen in The dynamic nodes are the building blocks of the proposed ad-hoc UWB positioning system. Each node is composed of the ranging device and an associated smartphone. An overview of the architecture of the system is shown in The host smartphone connects to the ranging device through a Bluetooth Low Energy (BLE) interface—the smartphone also runs the sensor logging application, as shown in The ability of UWB systems to provide accurate ranging is the motivation to use them for indoor positioning and localization. The relative range measurements between two UWB transceivers can be achieved using delay or time-based methods, angle-of-arrival methods, or received signal strength methods . As showBefore proceeding to describe the range measurement algorithm implemented by the LocSpeck node, two time-based ranging methods will be briefly discussed: the time-difference-of-arrival (TDOA) and the time-of-flight (TOF) methods. In TDOA-based systems, the tag or the mobile node sends a periodic message which is received by the surrounding anchors. The internal clocks of the anchors must be synchronized, so the anchors can compare the arrival time of the tag message using the same time reference , and finDevice A begins the ranging exchange by sending a blink message to any of the surrounding nodes. The purpose of this message is to notify any available nodes that device A is prepared to proceed with the range measurement exchange.If device B is within the communication range and is listening to the correct UWB channel, it receives the blink frame and replies by sending the range measurement initiation message, using the address of device A.Device A receives the ranging initiation message, then it sends back a poll message to the other side and records the precise time of sending the poll frame.Device B gets the poll message and stamps the arrival time. Then, device B sends a response message to device A and record the reply time  ensures that the error due to clock offset is minimized compared to the single-sided method. The LocSpeck node implements the double-sided two-way method using three messages for ranging in addition to one final message to share the calculated range between the nodes pair, as shown in By the end of the messages exchange, the measured range value is available at both devices for further processing. The positioning algorithm uses the range information along with the other local sensors readings to update the position state of the nodes.The LocSpeck nodes are designed to operate in an unpredictable environment, in terms of the number of the surrounding nodes and in terms of the possible structure of the network formed using these nodes. The nodes should support rapid deployment with no or minimum effort from the operator. To meet the target operating conditions, LocSpeck nodes implement a simple and light-weight random-access medium access protocol based on the pure ALOHA protocol ,45. FiguAfter powering up LocSpeck nodes, they start in the sleep state. Each node sleeps for a random duration between 50 ms and 80 ms. When the sleep duration elapses, nodes wake-up and power-up its receiver, waiting for any incoming frames. If a node receives a complete frame, it decodes the frame and checks if the incoming frame is a blink message. In the case of receiving a blink message, the node operates as an anchor. If the node receives a complete frame, and it is not a blink message, it concludes that the channel is currently occupied with the ranging sequence of another pair of nodes, and goes back to the sleep state, so it would not interfere with the current exchange. If no frames are received and the listening period elapses, the receiver timeout flag is asserted. In this case, the node assumes that the channel is free, so it switches to the tag mode and starts sending a blink message to any active node.While at the anchor or the tag states, each node sends or receives a sequence of messages. If any of the received messages does not match the expected message at that stage, the node will switch back to sleep mode. Also, the nodes will go to sleep mode, if there is any problem with messages transmitting or receiving, such as a receiver timeout, or any other problem related to the radio interface. Once a ranging sequence is completed successfully, the node saves the range and the node ID of the collaborator. Finally, each node sends a notification to the host smartphone before returning to the sleep mode for another random duration.The LocSpeck medium access protocol assumes that all the nodes use the same UWB channel and the same preamble code for both communications and for ranging. The DW1000 supports the use of 6 RF channels out of the 16 channels defined in the IEEE 802.15.4-2011 standard. Each node operating in the channel is assigned a preamble code from a set of two or four possible codes. The exact values of the preamble sequence are defined by the standard and are selected to ensure minimum cross-correlation between different codes. Assigning different nodes to different channels and assigning different preamble codes to the nodes operating in the same channel can increase the number of nodes operating in close proximity . AlthougThis section describes the two-dimensional positioning algorithm implemented in the LocSpeck system. It highlights the dynamic motion model of the nodes, along with the measurements update model. The LocSpeck nodes use different measurement update models: relative range measurement updates, Wi-Fi fingerprinting updates, and map information update.The standalone position algorithm is implemented using a particle filter (PF) which performs better than the extended Kalman filter (EKF) or the unscented Kalman filter (UKF), due to the non-linear nature of the Wi-Fi fingerprinting update . The PF pThe weights of the particles in the filter are updated using the relative-range, the Wi-Fi received signal strength indicator (RSSI) measurements, and the map information, using Equation (5):Equation (5) is a simplified version of the particle filter weights update equation in which the proposal density is the state transition model. After the weight update step, a weight normalization and resampling steps are implemented to remove the undesired particles. The filter is implemented with relatively low particle count (150 particles) to reduce the processing time required. The details of the measurement likelihood equations for different measurement updates equations are presented in the following subsections.k; The measurement update using a relative-range between a pair of collaborating nodes involves two pieces of information: the relative-range and the position of the collaborating node. The measurement can be described using Equation (6).Note that Equation (7) not only depends on the local node state, but it also depends on the collaborating node state. In this case, the particle filter running locally on each node needs to account for the collaborating node uncertainty and the cross-correlation that stems from the collaboration between different nodes.This is accomplished by considering the joint distribution of the state of the local and the collaborating nodes, conditioned on the relative range measurement, mulation ,50, in wFollowing the RBPF formulation, the marginalized particle filter equation, As shown in Equation (9), the local particle filter keeps track of the local state only, and the state of the collaborating nodes need to be sent over the communication channel after each collaboration exchange. Sending the full posterior, represented as particles, will consume the available bandwidth by occupying the radio channel for more time. Instead, each node stores a simplified representation of its posterior as a Gaussian distribution, which can be fully described by the mean and variance of the node position. The mean of the state can be described by Equation (10).The variance of the state is described by Equation (11).Similarly, the cross-covariance matrix expression is given by Equation (12).The values represented by Equations (10) and (11) are exchanged between the different nodes, along with the measured relative range, approach  but exteWhen the local node initiates a new range measurement with another node, the conditional distribution, ectively .(13)x¯kAlgorithm 1 Distributed Relative-Range Measurement UpdateInput:Range measurement and collaborating node state parametrized with the mean and the covariance: Output:Local state posterior, for each particle 1 2   Evaluate the conditional distribution, 3   Evaluate the measurement likelihood, 4   Evaluate the marginal likelihood: 5   Evaluate mulation ,506   Evaluate particle weight: 7   Time update step: end for8 9 Normalize the particle weights: 10  Evaluate the mean and variance terms, 11  Update state cross-covariance term: return12  Algorithm 1 summarizes the steps to perform the distributed relative-range measurements update, taking into consideration the uncertainty in the collaborating node and the possible cross-correlation between the two nodes:The previous discussion illustrates that the proposed positioning system is a distributed system, which does not need a centralized processing element to estimate the state of the collaborating nodes. Each local node keeps track of the collaborating nodes and keeps track of the cross-covariance between its local state and the state of the collaborating nodes.The Wi-Fi RSSI fingerprinting method used in this work employs a Gaussian process model to represent the RSSI map . The Gauectively . Using tfound in .The filter also uses floorplan information to ensure that the effective particles are contained within the area of interest and to eliminate the out-of-bound particles. Equation (16) shows the weight update equation using the floorplan information:This section summarizes the performance results for several aspects of the LocSpeck positioning system. It highlights the performance of the different components of the system: the timing of the ranging frames, the efficiency of the medium access protocol, and finally, the positioning accuracy of the LocSpeck system using several realistic test scenarios.The timing characteristics of the ranging sequences and the individual components of the sequence were measured and analyzed using DW1000 on-chip high precision clock for time-stamping the different sequences. To obtain the range and ranging frames duration, two LocSpeck devices were placed 60 cm apart, and the firmware was modified to enable logging the frame duration measurements to a computer. The total duration of the ranging frame is affected by the DW1000 chip settings, such as the preamble length and the data rate. The DW1000 chip settings used for the rest of this section are summarized in This section examines the performance of the LocSpeck medium access protocol in terms of the ranging efficiency as a function of the number of the collaborating nodes. It is expected in a random-access protocol such as ALOHA, the basis of the LocSpeck medium access protocol, that the utilization of the medium is reduced due to collisions between different nodes attempting to initiate ranging sequence at the same time. Using the settings outlined in 2, the length of the testing region is 48 m, and the average width is 7.5 m. This section describes the positioning results of the LocSpeck system and compares the results with those obtained by the Pozyx system. The experiment took place on the second floor of the engineering block E (ENE) building at the University of Calgary. The surface area of the testing region is 360 mThe experiments were carried out using four dynamic nodes—one of them is the main node, while the other three nodes are the supporting nodes. Each node consists of a smartphone and a ranging device—both were held by a human participant. The participants were moving within the test area randomly. The test area was open to the public; however, there was light traffic during the experiments. The different smartphones used in the experiments were equipped with different sets of sensors. When inertial sensors were available, they were used to provide a PDR solution using the standalone filter. The rest of this section will evaluate the positioning performance of the main node, which was connected to the Pozyx system to collect the ground-truth trajectory.The complete experiment consists of three separate trajectories. Each trajectory starts with the four nodes at rest. Once the experiment starts, the nodes move in random trajectories inside the testing area. The nodes occasionally stop on one of the reference position markers on the floor. The reference trajectory of the main node is captured using the Pozyx reference system. The data logging application runs on each of the smartphones and collects readings from the available sensors, the Wi-Fi received signal strength indicator along with information about the corresponding access points, and the UWB range measurement along with the address of the collaborating node.The raw data of the three trajectories are processed using different scenarios. The first scenario evaluates the standalone positioning performance of the main node. In this scenario, all the sensors available to the main node are used by the positioning filter. The Wi-Fi fingerprint map used in this scenario is the reference map, which was created previously using a dedicated run. The objective of this scenario is to establish a performance baseline to which the performance of the collaborative approach is compared.The next two scenarios are collaborative positioning scenarios. In the first collaborative scenario, the main node is not using any of the available sensors, except for the UWB ranging device. At the same time, the supporting nodes are estimating their positions using all the sensors available to them, along with the Wi-Fi reference map. The main node uses only the relative range measurements to estimate its position. The objective of this scenario is to evaluate the effect of collaboration in the case of node asymmetry. The main node, in this case, is in a disadvantageous position where it could not estimate its location without external aid from the collaborating nodes.In the second collaborative scenario, the main node and the supporting nodes use the complete set of available sensors. The objective of this test is to assess the effect of collaboration when the active node already has a good estimate of its position using only measurements local to the device, without any external sources.In the final collaborative scenario, the positioning filter is providing position estimates based on a random-walk model only. This final scenario, though seems trivial, is used to establish the lower bound of the positioning performance.The rest of this section is divided into three subsections. The positioning performance of the LocSpeck system is evaluated using three different trajectories covering the same test area. The reference solution for each trajectory is compared to the pre-surveyed reference points on the ground. The performance of the Pozyx solution is summarized in The reference radio map is created using the Pozyx reference trajectory in a separate run. The fingerprint map is built by observing the signal strength indicator at the reference points, then fit a Gaussian process model for each visible access point, using the position and signal strength pairs. During the positioning scenarios, the resulting Gaussian process models are used by the different dynamic nodes to aid the positioning filter.The standalone scenario results comprise three trajectories for the main node. Due to the stochastic nature of the particle filter, each trajectory is run through the positioning filter 20 times to produce more robust statistics of the filter performance. The standalone positioning error statistics for the three trajectories are summarized in It is worth noting that the inclusion of the results in In this section, two collaboration scenarios are considered. The first scenario consists of four nodes, the main node, and three supporting nodes. The main node will not use any of its onboard sensors. However, the main node will only use the UWB device to measure the relative ranges between itself and the other collaborating nodes. The other supporting nodes will use all the sensors available to them, along with the range measurement device. The objective of this scenario is to evaluate the achievable performance using relative range measurements to dynamic nodes. The second scenario is similar to the first one with one change: the main node will be using all the available sensors, in addition to the range measurement device. The objective of this scenario is to assess the effect of the collaboration on the participating nodes. In addition to these two cases, the results of the positioning using the random-walk model only is showed as well.Positioning using relative range measurements• Another factor that can affect the performance of the main node in the collaborative setting is the availability of the supporting nodes. The availability of the nodes is illustrated in Positioning using relative range measurements and all sensors• This scenario evaluates the effect of the collaboration on the main node while using the full set of sensors available on board. When using all sensors, the main node should achieve a performance level similar to the performance of the standalone solution. Positioning using random-walk model only• Before proceeding to evaluate the performance of the collaborative approach using relative range measurements, it would be useful to consider the error in the absence of the collaboration between the main node and the other nodes. This previous section provided an overview of the positioning performance using the LocSpeck collaborative framework. However, for the strong nodes in the process, the performance can suffer a hit. One possible explanation of the performance degradation could be related to the fusion of an erroneous estimate from one of the collaborating nodes. This case is exacerbated when the filter of a collaborating node suffers from particle depletion. As a result, the filter will generate new particles through resampling. These particles will have less diversity, and their corresponding covariance will be small. When a filter with these characteristics collaborates with another, it will provide false confidence in its position estimate. Consequently, it will drive the estimates of the collaborating nodes in the wrong direction. The performance degradation due to an erroneous state estimate or an overconfident remote note could be mitigated by implementing filter integrity measures to ensure that each filter has a realistic covariance estimate, possibly by utilizing the actual values of the measurement likelihood function. Such measures could include divergence monitoring  or by inThis paper described and evaluated the LocSpeck system, a collaborative and multimodal framework for indoor positioning using smartphones. The collaboration between the different nodes is achieved using a dynamic and ad-hoc network. The network architecture and a light-weight medium access control protocol were highlighted and evaluated. The paper provided an overview of the hardware and software components of the individual nodes.Different simulations and experiments were performed to evaluate the performance of the standalone positioning algorithm of the nodes, along with the performance of the collaborative positioning approach. These experiments confirm the advantage of the proposed framework, as discussed. The results shown in this paper demonstrate that using collaborative positioning approaches can provide pronounceable improvement in the performance, especially for situations involving asymmetric nodes, where the weak nodes can benefit from the superior sensing or capabilities available in nearby nodes. However, the performance of the system can be improved by implementing state integrity checks to ensure that the positioning filters will not suffer from depletion, which can negatively affect the overall performance of the system.