This work introduces a super-resolution (SR) algorithm for range images on the basis of self-guided joint filtering (SGJF), adding the range information of the range image as a coefficient of the filter to reduce the influence of the intensity image texture on the super-resolved image. A range image SR recognition system is constructed to study the effect of four SR algorithms including the SGJF algorithm on the recognition of the laser radar (ladar) range image. The effects of different model library sizes, SR algorithms, SR factors and noise conditions on the recognition are tested via experiments. Results demonstrate that all tested SR algorithms can improve the recognition rate of low-resolution (low-res) range images to varying degrees and the proposed SGJF algorithm has a very good comprehensive recognition performance. Finally, suggestions for the use of SR algorithms in actual scene recognition are proposed on the basis of the experimental results. With the advancement and development of technology, obtaining three-dimensional 3D) range images using laser radar (ladar) is easy and popular in recent years ,2. The rD range iSuper-resolution (SR) of images can also be called up-sampling of images, which is a signal processing technique for estimating a high-resolution (high-res) image from a low-res image . SR algoThis work aims to propose an SR algorithm for range images based on self-guided joint filtering (SGJF). The algorithm adds the interpolation result of the range image as a guide image to the filter kernel to reduce the influence of the intensity image texture on the super-resolved image. In addition, an SR recognition system for range images is constructed and a range image acquisition platform is designed to test the effect of SR algorithms in recognizing ladar range images. Four SR algorithms are tested, including Bicubic, JBF, MRF and our proposed SGJF. The SR recognition system uses combined moment invariants composed of seven Hu and three affine moment invariants to describe the global features of the image. Moreover, it uses a back propagation neural network (BPNN) to classify and recognize the features. The recognition rates of the SR recognition system under different SR algorithms, SR factors, model library sizes and noise conditions are tested via experiments. The results show that the SGJF algorithm we proposed can largely avoid the problem of ‘texture transfer’ in the JBF algorithm and is also robust to noise. All of the SR algorithms we tested can improve the recognition rate of low-res range images and a high SR factor increases the improvement. Regardless of the resolution of the model library, high resolution of the super-resolved scene raises the recognition rate and using the model library with the same resolution as the super-resolved scene can obtain the highest recognition rate. Finally, following the experimental results, the application suggestions of SR algorithms in actual scene recognition are proposed.The remainder of this work is organized as follows. The interpolation algorithm estimates unknown pixels on the basis of known pixel values in the neighborhood and image resolution conversion is one of its specific applications. Strictly, the resolution conversion must first reconstruct (or interpolate) discrete data into a continuous curve and then sample at different sampling rates. However, in a real interpolation algorithm, these two steps can be performed in one operation . BicubicMRF is based on the Markov model and Bayesian theory. The Markov model describes the local statistical dependency relationship between image pixels and considers that a certain pixel value of an image is only related to some pixel values in its neighborhood. Bayesian theory is used to solve the uncertainty problem described by the MRF model, converts the prior knowledge of the image into the prior distribution model and then the maximum posterior probability is used to establish the objective function of the problem to transform the image processing problem into an optimization problem.J. Diebel and S. Thru  used theSpecifically, (1)(2)The image segmentation algorithm in the VLFeat image processing library provided by  is used (3)(4)Allowing the range values to propagate freely with only very sparse data constraints can lead to notable range ambiguity. Therefore, the guide range image e filter :(9)ENLS to obtain vector Reconstruct The specific implementation steps of the MRF algorithm are summarized as follows:The edge is one of the most important structural features of the image, which reflects the details of the image and belongs to the sensitive high-frequency information of the human eye. Generally, the smoothing effect of the filter will cause blurring in the image edge area, which will reduce people’s ability to distinguish the details of objects. Therefore, in many computer vision applications, filters are required to have edge preservation capabilities.The bilateral filter is an edge-preserving filter, which has edge-preserving capability whilst smoothing the image and has been widely used in image processing . The kerJ. Kopf et al.  first prThe filter of the JBF algorithm only considers the spatial Gaussian kernel and the intensity Gaussian kernel; that is, it assumes that where the intensity image gradient is large, the range image gradient is also large. However, the range image actually often makes smooth changes in sections; that is, the range information belonging to the same area of the object is constant or gradual, whereas the corresponding color guide image may have a certain texture structure that will cause the latter to be introduced into the super-resolved images after filtering. This phenomenon is called ‘texture transfer’.To reduce the effect of intensity image texture on the super-resolved images, we propose the SGJF algorithm. This algorithm adds the range information of the range image as a coefficient of the filter and reduces the influence of the intensity image texture by strengthening its weight. The specific method is to interpolate the low-res range image  to obtain a high-res range guided image Input low-res range image Calculate the magnification according to the sizes of Set the filter window size Calculate the filter kernel Calculate the range value Repeat steps 4 and 5 for all pixels to obtain a high-res range image The specific implementation steps of the SGJF algorithm are summarized as follows:Target recognition based on ladar range images generally includes two processes of model acquisition and scene recognition and four steps of noise suppression, image segmentation, feature description and feature classification . The modPre-processing includes noise suppression and image segmentation. Since the acquired range images are prone to produce some holes or obvious noises at the edges and areas with low reflectance of the target object, it is necessary to fill in these missing areas and eliminate outliers before processing the image. We used a statistical-based method  to complAfter eliminating the outliers, we removed the ground and partial noise by fitting the plane. The least square method was initially used to fit the plane function of the ground through the outermost ring range data of the range image. The ground and partial noise could then be removed by giving up the range data of the plane and its vicinity. After that, a binary operation was used to extract the region of the target object to complete the image segmentation. However, when there was strong noise in the range image, the above operation was not enough to remove all the noise. It was also necessary to clean up the remaining noise spots by the method of extraction of connected components to obtain a clean binary image.To improve the robustness, we acquired three range images for each pose of the target object and pre-processed them separately. The final binary image was obtained by voting after accumulating the three pre-processed images.Moment invariant as a global feature description vector has important applications in the field of ladar range image target recognition . In geneAll of the features of all of the models described by combined moment invariants were then used to train the classifier for scene recognition. BPNN as a classifier is currently widely used to classify ladar range image features . It usesGenerally, BPNN includes at least three layers: an input layer, a hidden layer and an output layer. The input layer to the output layer is one-way propagation and the relationship between the input data and the output result is non-linear. The three-layer BPNN used in the experiment can be expressed as (32)gk three at 40°, (2) five at 20° and (3) nine at 10°. For azimuth, the numbers of range images taken at different intervals were as follows: (1) three at 90°, (2) four at 60°, (3) seven at 30°, (4) 10 at 20° and (5) 19 at 10°. Combining the view and azimuth angles, 15 training sample sets with different sizes were obtained. From the table, the recognition rate of the test sample set by the BPNN was typically positively related to the training sample set size. However, when the training sample set had fewer samples selected at the view angle (or azimuth angle), the recognition rate was low even if many samples were selected in another direction. By contrast, when the training sample set size increased to a certain number, the recognition rate grew slowly or even stopped. If we wanted to maintain the recognition rate above 95%, a minimum of 232 training samples were required and if we wanted to maintain the recognition rate above 90%, a minimum of 104 training samples were required.The SR recognition of low-res scenes without noise was tested and the resolution of their super-resolved scene was the same as the resolution of the model. The down-sampling factor was defined as t object a were acs scenes b. Finallame as d c. OHRIIsThe binary image could well represent the contour of the target object and the effect of the down-sampled images slowly worsened as the factor To test the recognition rate, the training sample sets were established by OHRRIs and the test sample sets were established by down-sampled images and their SR images. Compared with In practical applications, the ladar range image often contains various noises, mainly Gaussian noise. To test the influence of noise on the SR algorithm in the system, this section used the same experimental procedure as The five BPNNs trained by the training sample set in Overall, a larger factor hastened the decrease in the average recognition rate and 25 dB decreased faster than 35 dB. Specifically, the Bicubic algorithm was extremely sensitive to noise and was unsuitable for use when the noise intensity and factor was large. The MRF algorithm performed best in noise and the recognition rate of its test sample set only decreased slowly with the increase in noise intensity and factor. However, it needed to adjust more parameters and the calculation was more complicated. The JBF and SGJF algorithms were also robust to noise. When the factor or noise intensity was low, they were less affected. However, when the factor and noise intensity increased simultaneously, the average recognition rate dropped significantly and the gap between the JBF and SGJF algorithms also increased. Taken together, the average recognition rate of the SGJF algorithm was significantly smaller than the MRF algorithm only when the noise intensity and factor were at the maximum, whilst it was superior or similar to other algorithms in other cases. Therefore, the SGJF algorithm had a very good comprehensive recognition performance.Recognizing multi-resolution (multi-res) super-resolved scene libraries by multi-res model libraries without noise was tested where the resolution of the super-resolved scene library may have been lower than, equal to or higher than the model library. Firstly, OHRRIs and OHRIIs of each target object  When a low-res scene is acquired, if it can be super-resolved then it should be super-resolved. (2) If multiple SR factors are available to choose from, the largest one should be selected even if the resolution of the super-resolved scene is higher than that of the model in the model library. (3) If only one SR factor exists to choose from, but multiple model libraries are available with different resolutions, the model library with the same resolution as the super-resolved scene should be selected.This work proposed an SR algorithm for range images based on SGJF, which can reduce the effect of intensity image texture on super-resolved images by adding the range information of the range image as a coefficient of the filter. We also constructed an SR recognition system for range images, which acquired range and intensity images of eight military target objects through the image acquisition platform to establish training and test sample sets for the simulation model and scene libraries and used the Bicubic, JBF, MRF and SGJF algorithms to test the effect of the SR algorithm on the recognition of ladar range images. The recognition of scenes with arbitrary attitude angles by model libraries of different sizes was initially tested. The results demonstrated that the recognition rate was positively correlated with the model library size. However, the recognition rate grew slowly or even stopped when it increased to a certain number. The recognition of the system under different noise conditions was then tested when the resolutions of the super-resolved scene and the model were the same. The results demonstrated that all the SR algorithms we tested can help improve the recognition rate of low-res scenes; moreover, a larger factor increases the improvement. The SGJF algorithm we proposed largely avoids the problem of ‘texture transfer’. It had the highest recognition rate when no noise occurred and showed noise robustness after adding noise. Therefore, the SGJF algorithm has a very good comprehensive recognition performance. Finally, the recognition of multi-res super-resolved scene libraries by multi-res model libraries was tested. The results demonstrated that regardless of the resolution of the model library, an increase in the resolution of the super-resolved scene also increased the recognition rate. In addition, using the model library with the same resolution as the super-resolved scene could obtain the highest recognition rate. According to the experimental results, suggestions on the use of SR algorithms in actual scene recognition are proposed. Our future work includes studying the application of the SR algorithm in the recognition of radar range images based on local features.


Modern visual SLAM (vSLAM) algorithms take advantage of computer vision developments in image processing and in interest point detectors to create maps and trajectories from camera images. Different feature detectors and extractors have been evaluated for this purpose in air and ground environments, but not extensively for underwater scenarios. In this paper (I) we characterize underwater images where light and suspended particles alter considerably the images captured, (II) evaluate the performance of common interest points detectors and descriptors in a variety of underwater scenes and conditions towards vSLAM in terms of the number of features matched in subsequent video frames, the precision of the descriptors and the processing time. This research justifies the usage of feature detectors in vSLAM for underwater scenarios and present its challenges and limitations. Knowing the position of underwater robots and obtaining maps of the surrounding environment is essential for a variety of robot tasks, from gathering geo-referenced data to autonomous navigation and exploration. Simultaneous Localization and Mapping (SLAM) offers a framework to incrementally build a map while a robot moves through an unknown area and to use that map to localize the robot simultaneously. A typical implementation of SLAM in the underwater environment involves the use of dead-reckoning, acoustic sensors and cameras . In the A fundamental part of feature-based SLAM is data association which allows extracted features from images as key-points to be recognized when re-observed in consecutive images as well as in loop-closing. To achieve this, features are extracted using an interest point detector, and then described including local information from the neighbors of the point through a feature descriptor. The descriptor is a vector which assigns a distinctive identity to the feature to be recognizable .In computer vision there are several feature detectors and descriptors which have been evaluated in terms of scale invariance, viewpoint changes (including rotation) and variations in illumination ,5, as weUnderwater images are subject to alterations to the light and characteristics of the medium resulting in blurry, hazy and tinted images . This prIn the second part of this Section, a selection of related works to feature detectors and their evaluation in vSLAM is presented. In , Younes For feature-based vSLAM, the front-end involves the detection of interest points, the creation of descriptors, and the data association performed by matching features from the current frame with previous frames.Visual SLAM approaches have been evaluated for indoor and outdoor applications over benchmark datasets. In  ORB-SLAMIn  an experSeveral feature detectors and descriptors have been evaluated in the past regarding correct matching against image alterations. For this purpose, features are extracted from a pair of images from the same scene or digitally altered and then matched against the other. In  the SIFTThe recall ratio Equation  equivaleThe Equation  which inJohansson et al. used the same performance ratios to evaluate more detectors and descriptors; and combinations (detectors/descriptors). They include SURF, ORB, BRISK, Fast Retina Keypoint (FREAK) finding the combination SURF/SURF and ORB/BRISK robust against geometric and photometric transformations . SimilarOther evaluations are carried out for customized functions such as for tracking objects  and visiBased on the performance of features extractors in the literature discussed in The SIFT algorithm follows two main stages in the detection part: (1) Scale-space extrema detection, where Difference of Gaussian (DoG) is applied to identify keypoint invariants to scale changes, then a local extrema check with adjacent pixels is performed; (2) keypoint localization, which rejects low contrast keypoints and then eliminates non-edge points based on Hessian matrix.For building the descriptor the algorithm follows two further steps: (3) orientation assignment, which forms orientation histograms from local gradients to determine the dominant direction of the keypoint; (4) keypoint descriptor, where the proper vector is constructed based on the course of the keypoints and local areas around them, and finally the descriptors are normalized to improve light invariance ,26.SURF follows a similar idea as SIFT, it was developed by Bay et al.  as a fasThe descriptor combines local gradient information, like SIFT, 2D Haar wavelet response to local areas and windows around they keypoints to approximate the gradients.ORB is based on Features from Accelerated Segment Test (FAST) and Rotated BRIEF. It creates a pyramid of blurred and subsample versions of the image which are then divided into cells and FAST is computed. Then the cells are subdivided to contain one corner per cell or the maximum number of cells allowed by a parameter of the algorithm, disregarding the features with low score per cell.The ORB descriptor modifies the FAST extractor adding an orientation component through first-order moments in a local patch. Then the Binary Robust Independent Elementary Features (BRIEF) descriptor is computed on a rotated patch. It reduces the descriptor vectors such as in SIFT and SURF to binary vectors .BRISK is based on the FAST detector, it extracts features from the image and different scales of it. For the descriptor, it uses a concentric rings sampling pattern to retrieve the gray values of their neighbors and process local intensity gradients to obtain the direction of the keypoint. Then it forms the binary descriptor comparing the intensity between pairs from the pattern .AKAZE focuses on multi-scale feature detection exploiting non-linear scale spaces. It is computationally efficient taking advantage of Fast Explicit Diffusion. It applies the Hessian determinant to the scaled images and performs a search of the maxima response in spatial location.Alcantarilla et al. proposed a Modified-Local Difference Binary (M-LDB), that exploits gradient and intensity from the extractor stage, as a descriptor. It is based on BRIEF performing over the average of areas instead of pixels. It includes intensity values, and the orientation of the keypoint is similar to KAZE .Images captured in underwater scenarios are altered in every aspect due to the changes in radiant energy when traveling through water rather than air. Light gets scattered by suspended tiny particles in the water  and it is also absorbed by the water itself causing blur and loss of contrast a 28]. T. T28]. TSunlight flickers (caustic waves) are observed in very shallow water which are formed by trespassing a wavy water layer . These lArtificial light sources are used when gathering images at night or in murky water to increase the lightness of the scene. The source is usually located near the camera and the light is reflected by particles in the medium yielding the back-scatter component e 30,31],3130,31]There are several approaches of image processing to enhance underwater images regarding the lighting effects presented before. In , Wang liOther methods developed mainly to diminish the effect of sunlight flickering such as the works presented in ,32,33,34External hardware have been used for mitigating the lighting problems when gathering underwater images. Treibitz et al. placed polarizers on the light source and the camera to achieve back-scatter reduction ,42. In , a barriIn this Section we present the evaluation framework followed, based on the literature described in Three key elements to apply the detector/descriptor sets described to a feature-based SLAM are: the capacity to extract keypoints from an image, the capacity to associate re-observed keypoints, and the computational time. In this regards the detectable features and frame sequence matching tests are proposed. Additionally, we include pre-processed images from the datasets through an enhancing underwater image by fusion  and backWe use two detectors’ profile tuned manually to expose features proportionally to a limit. One is set to obtain around 1000 features . We used the BlueROV2 robot to acquire 1024 × 768 pixels images which are collected on an average of 12 frames per second. Images include part of the structure of the Remotely Operated Vehicle (ROV) (lights). Eight datasets are selected for the present chapter.We collected different datasets for a variety of underwater scenarios in rivers, beaches, ports and open sea in the surroundings of Perth, Australia  files keeping the record of the number of features found, matches and processing time.In The overview shows an overall homogenous performance detecting around 500 features in It is worth mentioning that the detailed graphs for both profiles have similar behavior, the only difference is the number which is proportional to the maximum number of features per profile. Therefore, in most cases, we analyze the We have selected two datasets to show the performance of the feature detectors in the underwater scenario. In Dataset_8 is mostly sandy with some frames capturing partial poles as objects. The illumination is uniform and has a greenish tint . As obseIn overall, detectors are able to find features in underwater scenarios specially over rocks, defined algae patches (from far), objects and even sand (when exposing patterns). The detectors struggle in sandy areas from far as well as where turbidity and blurriness appears.It is important to quantify the number of features that can be re-observed (matched) in the following frames under the vSLAM scope. In this framework, the inliers (correct matches) are obtained after applying NN and homography between the keypoints detected in two consecutive frames. In Similar to the average number of features found, Datasets 3, 4, 6 and 8 present the lowest average number of inliers, despite having a high average number of features found in In the case of AKAZE, which showed a lower number of average features extracted in In We applied two image enhancement algorithms for underwater images to Datasets 3, 4, 6 and 8 which showed the lowest number of features or inliers found. In The average number of features extracted increases for Datasets 4, 6 and 8 a. DataseIt can be seen in Dataset_4, which was taken at night with artificial illumination on a sandy background with few algae and rocks, gets the most significant improvement in the number of inliers. The filter by fusion gets better results than the backscatter filter.Datasets 6 and 8 also increase their number of inliers, especially with the filtering by fusion. These two scenarios present a sandy background with few objects on the seafloor. Both present illumination problems, Dataset_6 presents a caustic pattern and Dataset_8 a non-uniform illumination.The processing time is measured for the detection and describing, NN matching and homography for the two profiles. In ORB is the fastest set detector/descriptor with an average processing time of 43 ms and 97 ms for The enhancement algorithms applied are highly time-consuming showing values above 1 and 2 seconds for the filtering by fusion and backscatter removal algorithms respectively.The experimental results provide a detailed analysis of SIFT, SURF, ORB, BRISK, and AKAZE detectors/descriptors for underwater environments towards their application to vSLAM.In this analysis, the detectors selected showed a satisfactory performance on images containing color distortion, low non-uniform illumination and low turbidity. Sandy environments with algae patches, algae recorded from near and far; small particles, such as debris and rocks; and objects, such as poles and rocks presented detectable features for the detectors.Different datasets were categorized according to the characteristics of the seafloor, types of objects, lighting, tint, and turbidity. The influence of these effects on the images is seen in the number of features extracted and subsequently matched in the following frames. The results showed a decrease of features and matches in presence of turbidity and blurriness, as in The number of inliers when matching keypoints from consecutive frames was homogeneous among the detectors, in The two enhancement algorithm applied in this survey showed an improvement in the performance of the detectors/descriptors. The filter by fusion  showed tThe survey provides abundant information and detailed insights valuable for making decisions in applications towards vSLAM. ORB detector/descriptor stood out in detection and matching performance, shaping up as a good selection for implementing vSLAM, with the lowest computing time.


Due to its widespread presence and independence from artificial signals, the application of geomagnetic field information in indoor pedestrian navigation systems has attracted extensive attention from researchers. However, for indoors environments, geomagnetic field signals can be severely disturbed by the complicated magnetic, leading to reduced positioning accuracy of magnetic-assisted navigation systems. Therefore, there is an urgent need for methods which screen out undisturbed geomagnetic field data for realizing the high accuracy pedestrian inertial navigation indoors. In this paper, we propose an algorithm based on a one-dimensional convolutional neural network (1D CNN) to screen magnetic field data. By encoding the magnetic data within a certain time window to a time series, a 1D CNN with two convolutional layers is designed to extract data features. In order to avoid errors arising from artificial labels, the feature vectors will be clustered in the feature space to classify the magnetic data using unsupervised methods. Our experimental results show that this method can distinguish the geomagnetic field data from indoors disturbed magnetic data well and further significantly improve the calculation accuracy of the heading angle. Our work provides a possible technical path for the realization of high-precision indoor pedestrian navigation systems. The application of global positioning systems (GPS) in the field of navigation has resulted in greater convenience for human society. However, in some scenarios, such as rural areas, indoors, and during field rescue, navigation based on satellite positioning technology is far from being sufficient in meeting the requirements. At the same time, with the development of microelectronics manufacturing and algorithm theory, researchers have begun to study new technological solutions to meet diverse navigation requirements in the case of GPS failure ,2,3,4. HIn order to take advantage of MEMS-based inertial autonomous navigation, Suprem et al.  adopted Geomagnetic field signals can provide value direction information in indoor pedestrian inertial navigation due to their flexible application scenarios and high degree of integration. Magnetic-assisted inertial navigation technology is able to use geomagnetic field data without being constrained by the scene ,17. HoweIn recent years, rapidly developing machine learning algorithms have provided researchers with effective tools for signal processing. Deep learning algorithms, Bayesian networks, etc., have been used in many applications in classification scenarios ,22. As aThe rest of this paper is organized as follows. The Earth’s magnetic field is a naturally occurring planetary phenomenon which follows the principles of dynamo theory and is created by the constant motion of Earth’s outer core, composed mostly of molten iron . The geoIn indoor pedestrian inertial navigation, the heading angle can be obtained using a MEMS gyroscope. The error in the heading angle calculated by the gyroscope accumulates twice over time because the gyroscope contains measurement errors. The accuracy of the heading angle obtained by geomagnetic information is not related to time. Hence, the geomagnetic field can be used to correct the accumulation error of the heading angle calculated by the gyroscope. However, there are a large number of artificial interference sources randomly distributed throughout indoor spaces, as shown in In order to make full use of the magnetic data information and avoid interference from using magnetic data anomalies, we designed a novel algorithm flow to calculate the heading angle. As shown in The magnetic data sequence collected by the magnetometer will then be transmitted to a neural network. Once this neural network is well trained, it can be used as an indicator in the algorithm. It can determine whether the magnetic data sequence belongs to a magnetic anomaly sequence or is geomagnetic sequence data. When the neural network verifies that the received magnetic data sequence is a magnetic anomaly sequence, which contains incorrect information, our algorithm will round it down to avoid erroneous effects on the system’s heading angle calculation. When the neural network indicates that the collected magnetic data sequence is undisturbed data, the algorithm will transmit this set of data to the second part. The second part of the algorithm integrates an extended Kalman filter, which is a special form of particle filtering and can be used to correct magnetic data with a smaller computation in our devices, transmitted from the previous part, and fuse the information obtained by other inertial sensors to calculate the heading angle ,29. BecaFor magnetic data screening, the second part of the algorithm includes a convolutional neural network (CNN). In order to get good screening results, we carefully defined the topology of the neural network based on the characteristics of the magnetic data. Since the collected magnetic data were three-dimensional vectors, we defined four channels consisting of three axis values and the model of the magnetic data vector for input into the neural network. We can define the dimensions of the input according to the length of the input time series. As shown in After passing through a pooling layer and a fully connected layer, we used the SoftMax function to obtain two outputs, which, respectively, indicate the network’s probability of judging whether they are in a magnetic anomaly state or geomagnetic field data state. Since the learning network is unsupervised, our training process uses a combination of forward and backward propagation algorithms. We used the K-cluster theory to define the loss function. If we note the network as function The input sequence has four channels as mentioned above, and its length is 13. The dimension of vector M is Therefore, the distance of the two feature vectors for the cluster isSince the SoftMax function is used, we use The objective of training is maximum:The negative log-likelihood of the training set is:The Z-axis. We use the blue point to indicate the geomagnetic data extracted from the magnetic data by our 1D CNN, and the red points are the interference magnetic data. As we can see, the magnetic data are divided into two groups, and we will verify the effectiveness of the separation in the following sections.As shown in We can enter the geomagnetic field data obtained in the second part of the algorithm in In the process of attitude angle calculation for pedestrian inertial navigation, the quaternion calculation with a simple arithmetic process can simplify the arithmetic process and improve efficiency without loss of accuracy. The relationship between the quaternion and the attitude angle can be obtained through a certain attitude angle operation in the attitude matrix, which can be represented by Equations (11) and (12), respectively. The attitude angle equation based on quaternions is:When using the extended Kalman filter for the attitude angle calculations, the measurements of the MEMS gyroscope were used to update the equation of the state, and interference-free magnetic data obtained through 1D CNN classification were used as the observation to correct errors in the one-step prediction of the equation of state. Hence, the prediction and measurement models Gyroscope measurements contain errors, and though these errors only have a small effect on state predictions over short periods of time, they can accumulate over time. Indoor geomagnetic field data can be disturbed for short periods of time in some areas, but interference-free geomagnetic field data can accurately calculate heading angles of information at any time. The extended Kalman filter can combine the features of both, using interference-free magnetic data to correct for deviations in the heading angle, resulting in greater accuracy of the attitude angle. In order to reduce the amount of computation and ensure accuracy, we use quaternion to update the attitude angle. The gyroscope has zero bias instability, we also estimate the zero bias of the gyroscope in real time. Based on the above information, we can establish the state vector One-step prediction of the quaternion was calculated using the derivative of quaternion:The quaternion differential equation isThe Jacobian can be calculated to produce the state-transition matrix The calibration algorithm proposed in this paper corrects the heading angle deviation in pedestrian inertial navigation using the geomagnetic field information obtained by 1D CNN screening without interference from the indoor environment. As the heading angle obtained from the geomagnetic signal is an absolute angle, which is not related to time, it can compensate for the phenomenon of divergence of the heading angle error with time in inertial navigation. Then, the observation vector and measurement equation of the extended Kalman filter is as follows:During the initialization of pedestrian inertial navigation, we can obtain geomagnetic field data The Jacobian can be calculated to produce the observation matrix Linearization of the non-linear state equations and observational equations can be achieved using Taylor expansion, which allows the use of a Kalman filter for attitude angle calculations. The process is as follows.Predicted state estimate:Predicted estimate covariance:Measurement residual:Kalman gain:Updated state estimate:Updated estimate covariance:We implemented an algorithm based on 1D CNN magnetic anomaly classification for simulation and testing according to the above procedure of extended Kalman filtering, and we analyzed and discussed the test results for heading angle accuracy.To verify the effectiveness of our proposed algorithm, we built a wearable navigation platform and used it to perform a series of experiments in indoor scenes. As shown in In order to explain the working principle and performance of our algorithm, we first analyzed the magnetic field signal distribution of the indoor test track. As shown in Next, we studied the calculation of the heading angle in two cases, aiming to verify the effectiveness of magnetic data screening based on 1D CNN on the pedestrian inertial navigation performance of the system. In one case, we used the raw magnetic data collected by the device to directly calculate the heading. In the other case, the magnetic data are screened out by a well-trained 1D CNN before being used to calculate the heading angle. The 1D CNN discards the bad magnetic data and extracts the geomagnetic data to improve the accuracy of heading angle calculation. We plotted the trace of the tester, as shown in The navigation trace obtained by our algorithm with the 1D CNN screening model was plotted in Based on two different methods for calculating the heading angle, we can obtain two pedestrian inertial navigation trajectories as shown in For a more intuitive comparison, a new trajectory was used to verify the validity of our proposed algorithm, and we compared this algorithm with the magnetic anomaly detection algorithm  . In FiguIn This paper proposes an indoor pedestrian navigation algorithm that can significantly improve navigation accuracy using magnetic data. Based on the CNN network, we designed an unsupervised learning model and obtained an algorithm that can screen out anomalies in indoor magnetic data from the measured data. In addition, by using real magnetic data obtained through screening combined with the characteristics of the gyro error mainly derived from the accumulated errors, we proposed a novel heading angle calculating method which is based on an extended Kalman filter. This method can use real indoor magnetic data to correct the accumulated errors of the inertial navigation system, thereby achieving higher accuracy. Our proposed algorithm demonstrated good adaptability and effectiveness in indoor environments when analyzed using trajectory tests in different environments. Our research provides an effective solution that addresses the issues encountered in pedestrian navigation based on wearable equipment.


Recent studies show that changes in one of the brain areas related to empathic abilities ) plays an important role in violent behavior in abusers of alcohol and cocaine. According to the models of James Blair, empathy is a potential inhibitor of violent behavior. Individuals with less empathic abilities may be less susceptible and motivated to inhibit violent behavior, which causes a higher risk of violence. Recent neuroscientific research shows that modulating (stimulation or inhibition) certain brain areas could be a promising new intervention for substance abuse and to reduce violent behavior, such as the neurostimulation technique transcranial direct current stimulation (tDCS). This study aims to investigate tDCS as an intervention to increase empathic abilities and reduce violent behavior in forensic substance use offenders.A total sample of 50 male forensic substance abuse patients (25 active and 25 sham stimulation) will be tested in a double-blind placebo-controlled study, from which half of the patients will receive an active stimulation plus treatment as usual (TAU) and the other half will receive sham stimulation (placebo) plus TAU. The patients in the active condition will receive multichannel tDCS targeting the bilateral vmPFC two times a day for 20 min for five consecutive days. Before and after the stimulation period, the patients will complete self-report measurements, perform the Point Subtraction Aggression Paradigm (PSAP) and a passive viewing empathy task. Resting state electroencephalography (rsEEG) will be performed before and after the treatment period. A follow up will be conducted after 6 months. The primary outcome is to investigate multichannel tDCS as a new intervention to increase empathic abilities and reduce violent behavior in offenders with substance abuse problems. In addition, we will determine whether electrophysiological responses in the brain are affected by the tDCS intervention. Finally, the effects of tDCS on reducing craving will be investigated.This study is one of the first studies using multichannel tDCS targeting the vmPFC in a forensic sample. This study will explore the opportunities to introduce a new intervention to improve empathic abilities and reduce violence in forensic substance use offenders. Specifically, this study may give insight into how to implement the tDCS intervention in the setting of daily clinical practice in this complex, multiple-problem target group and with that contribute to reduction of recidivism.NTR7701. Registered on 12 January 2019. Prospectively registered before the recruitment phase. https://www.trialregister.nl/trial/7459.Dutch Trial Register, Recruitment started on the 1st of February 2019 and will be finished approximately in the winter of 2019. Protocol version 1. 22 May 2019. Repeatedly using substances has been found to lead to neuro-adaptations in the ventral striatum and ventral tegmental areas and with that, decreased dopamine secretion . DecreasEmpathy is crucial for social enhancement, social interactions and relationships, and our emotional and social life . A deficSeveral studies show evidence that impaired prefrontal cortex areas lead to the emergence of aggressive behavior. Most notable are impairments in the vmPFC , and the DLPFC (e.g. disinhibition and impulsiveness), which is associated with aggression and violent behavior –28. ThesFigure The vmPFC is particularly relevant to empathic abilities and antisocial behavior , 36, 37.Furthermore, a recent study  combinintDCS is a non-invasive neuromodulation technique that modulates the brain region of interest by increasing or decreasing neuronal excitability through constant, low-direct-current electrodes. tDCS has been proven to be an effective intervention to modify brain activity –45 and hIn this current study, the effectiveness of tDCS as a new intervention to increase empathy and reduce violent behavior in substance abuse offenders will be investigated. Through modulating (stimulating and inhibiting) certain areas of the brain, tDCS causes a change in the function of the brain, due to an increase in susceptibility to generate and facilitate brain-related electrical impulses. This susceptibility is achieved through repeated sessions of brain stimulation and is thought to produce long term potentiation (LTP)-like “learning” in stimulated neurons. Functional alterations in the brain due to long-term substance abuse are hypothesized to improve with the application of tDCS –61 and tSeveral studies showed that emotional processes can be influenced through anodal tDCS of the prefrontal cortex (PFC). tDCS can modulate emotional pain , 63 and In addition to empathy, antisocial behavior, including aggression, is also associated with dysfunctions in the prefrontal cortex –24. ReseIn addition to aberrant functioning of specific brain areas, recent research , 73 alsoThe aim of this study is to investigate the effectiveness of tDCS in increasing empathy and reducing violent behavior in offenders with substance abuse problems.Does stimulation with multichannel tDCS targeting the vmPFC increase empathic abilities in forensic patients with substance abuse problems during an empathy task from pre intervention to post intervention?Does stimulation with multichannel tDCS targeting the vmPFC reduce aggressive response and risk of violence risk in patients with forensic substance abuse from pre intervention to post intervention?Does multichannel tDCS targeting the vmPFC reduce craving in patients with forensic substance abuse?Does multichannel tDCS targeting the vmPFC affect electrophysiological response in the brain measured through EEG from pre intervention to post intervention?The following research questions will be addressed in this study:Outcomes will contribute to the development of more effective diagnostics and treatment of patients with substance abuse. Furthermore, these data will potentially contribute to improving treatment through increasing understanding of specific targets for treatment interventions. In addition, outcomes may provide better insight in the functioning of the vmPFC and the relationship between empathy and risk of violence.Forensiche Verslavings Kliniek (FVK), the forensic addiction clinic of Bouman, Antes. The institution is located in Rotterdam, The Netherlands. The sample size is based on other studies; we will operate according to the “Evidence-based guidelines on the therapeutic use of transcranial direct current stimulation (tDCS)” published in 2017  in order to assess risky and impulsive behavior. The RISQ is a 38-item self-report questionnaire-based measure, assessing eight domain-specific factors .We use the Dutch version of the BIS-11  in orderWe use the Dutch version of the AUDIT  in orderWe use the Dutch version of the DUDIT  in orderWe use the Dutch version of the OC-DUS  in orderWe use the Dutch version of the OCDS  in orderThe primary outcomes are empathic abilities, aggressive behavior, and the electrophysiological response in the brain. Empathic abilities will be measured using the Passive Viewing Empathy Task , followeAggressive behavior will be assessed using the PSAP task. The PSAP task is known to be one of the best-validated instruments in provoking aggression in the laboratory . In thisAs a standard for effectiveness of the reduction of violent behavior next to the PSAP, the results of a violence risk assessment instrument (HKT-R)  will be Functional brain changes will be measured using rsEEG, with a resting-state task to measure the baseline of the brain activity of the participant in a resting-state condition and the passive viewing empathy task to measure the electrophysiological changes in brain function during empathic induction caused by the intervention. The expectation is that patients who receive tDCS will show higher event-related potential (ERP) towards the pictures of the victims after the intervention compared with patients who have receive the sham condition.After 6 months, as a standard for effectiveness of the reduction of violent behavior, the results of the aggression task (PSAP) and the risk-assessment tool (HKT-R) will be compared to the results obtained before the intervention. The HKT-R risk level will be obtained to see whether there is a longitudinal effect of tDCS on aggression and risk reduction. In addition, to check whether the effect of the tDCS intervention is long-lasting, the participants will complete the self-report questionnaires, the empathy task, and the aggression task once more.Beside the primary outcomes, the current study distinguishes multiple secondary outcomes. These will be measured using the self-report questionnaires described in the 'Instruments' section. All the secondary outcomes will be assessed during T0 and again at T6.Different general linear models (GLM) in SPSS will be used to analyze the main parameters. For instance, analyzing the empathy and aggression outcome (empathy ratings and b responses in the PSAP task), the outcome variables will be handled as the dependent variables. In addition, the group variable (active versus sham) will be used as the independent between-group variable, whereas the pre-post time will be included as a within-subject independent variable. A mixed design will be used in which the pre-post intervention in the experimental group will be compared with the pre-post intervention aggression outcomes in the sham group. The mixed design will be used for the PSAP outcomes, empathy ratings, self-report questionnaires, and assessment of risk of violence with the HKT-R instrument.The sample size is based on the primary outcome and other studies from the aforementioned guidelines  and seenThis study protocol describes the design of an intervention with multichannel tDCS targeting bilateral vmPFC next to TAU, in comparison with the sham condition and TAU. This study will explore the opportunities to introduce a new intervention to improve empathy, reduce violence, and reduce craving in substance-use offenders.The present study has several strengths. First, to our knowledge only a few studies focus on increasing empathic abilities –67, 101 Second, tDCS may also influence substance abuse –61 and, Third, we will use a wide range of instruments and will gather information through multiple sources. We will obtain rsEEG, questionnaires about psychopathic traits (SRP-SF) , aggressMoreover, we will also include the participants who drop out during the intervention in our 5-day intervention program. This will give insight into the consequences of the formal procedures preceding the intervention and may eventually enable us to describe profiles of the intervention with tDCS in treatment success and failure.Finally, this study will contribute to development of a cheaper and less invasive treatment for substance abuse. As mentioned before, problems that patients with substance abuse bring an enormous burden to the community . Previous research has found that substance abuse, especially alcohol and cocaine, are related to (violent) criminal behavior. Money invested in treatment may lead to a large reduction in the costs associated with substance abuse. Nevertheless, current interventions seem insufficient in the treatment of substance abuse in forensic mental health care and are not sufficient enough to reduce the risk of violence risk; 66% of these patients reoffend.Despite the strengths of this current study, several limitations may threaten the quality of our study. The greatest challenge will be to have an adequate response rate at baseline and at the end of the 5-day treatment. Our target group will be conducted in a forensic institution and is participation is voluntary, so the participants can decide to quit at any time. The population has an extensive history of treatment and criminal justice, so they might have negative experiences with treatment or research studies. Moreover, the participants already have filled in numerous questionnaires in their time spent in institutions, such that they may have become tired or suspicious about the purpose of the proposed study. To keep the nonresponse rate as low as possible, the researchers conducting this study are trained to motivate and encourage the participants to complete the program. The interaction will be transparent, attentive, and flexible and the researchers will be prepared to give clear information about the purpose, goal, and aim of the current study at any time. Furthermore, a treatment that uses neurostimulation and EEG might seem scary and new to the participants as compared to TAU, so extra explanation might be needed to assure the participants that it is safe. So, to recruit as many participants as we aim to in this study will require a high degree of motivation.Finally, an active control group (i.e. sham modulation and TAU) might lead to smaller effect sizes with active tDCS than when the tDCS treatment is compared to no treatment or a wait-list group. However, considering the seriousness of the problems of the forensic substance use offenders it would not be ethical to let them wait for an intervention and it remains important to get a sense of the effects of placebo versus the effects of active stimulation.In conclusion, with the present study design we are able to explore what the added effectiveness is of tDCS plus TAU in comparison to sham plus TAU, which could provide valuable information for institutions, researchers, psychologists, and the professionals in the field of criminal justice. In addition, this study may help to reduce craving in forensic substance use offenders and thus reduce not only violent behavior but also recidivism. The study will contribute to knowledge about increasing empathic abilities and the functioning of the vmPFC. Specifically, this study may give insight into implementing tDCS on the vmPFC in the TAU and the setting of daily clinical practice in this complex, multiple problem target group.The current study started in February 2019. The data collection will run until the 50 participants all completed the 5-day program. Follow up will be conducted after 6 months to test whether the effects of the intervention are still present. Until then, the intervention effects are unknown.


Optical 3D geometry reconstruction, or more specific, fringe projection profilometry, is a state-of-the-art technique for the measurement of the shape of objects in confined spaces or under rough environmental conditions, e.g., while inspecting a wrought-hot specimen after a forging operation. While the contact-less method enables the measurement of such an object, the results are influenced by the light deflection effect occurring due to the inhomogeneous refractive index field induced by the hot air around the measurand. However, the developed active compensation methods to fight this issue exhibits a major drawback, namely an additional cooling of the object and a subsequent transient illumination component. In this paper, we investigate the cooling and its effect on temporal phase reconstruction algorithms and take a theoretical approach to its compensation. The simulated compensation measures are transferred to a fringe projection profilometry setup and are evaluated using established and newly developed methods. The results show a significant improvement when measuring specimens under a transient illumination and are easily transferable to any kind of multi-frequency phase-shift measurement.Dataset License: CC-BY The optical 3D geometry measurement  of hot aIn this paper, we investigate the transient illumination component occurring during the forced air flow cooling of a hot measurand using pixel intensities and thermal imaging. Its effect on the phase reconstruction using a multi-frequency phase-shift algorithm  is analyThe second part of the paper contains an extensive analysis of the newly proposed projection sequence. In order to reveal the positive effect of the proposed projection sequence, a synthetic intensity gradient is incorporated into the projected images. The effect of the new projection sequence as well as of the compensation by approximation is analyzed in this simplified experiment. Additionally, the shown concepts are used for a geometry measurement of a hot object with the added cooling of the forced flow actuator. For the experiment with the hot object under approximately industrial conditions, the new projection sequence is enhanced for improved robustness. This experiment is analyzed w.r.t. the acquired reconstruction quality. The acquired results are discussed and conclusions are drawn.In this paper, a fringe projection system similar to  is used.A recent review of temporal phase unwrapping techniques is given by Zou et al. . The autn can be described [f and its phase Generally, the intensity escribed  as(1)Inn of N acquired sine signals can be estimated from three or more measurements of known phase-shifts The phase Using an K, resulting in fringe images with smaller wavelengths. The frequencies While these four patterns are sufficiently accurate for an ideal system with an unlimited resolution, its accuracy is limited for a limited resolution, e.g., when using an 8-bit camera sensor. For this case, Peng  proposedCalculating the fringe order is part of a process called phase unwrapping. It is a hierarchical process, estimating This requires full phase-shift sequences being projected for each frequency ty Ia(u)  would reThis reduces the required number of images from m and the deviations of corresponding 3D points are used as quality metric to relatively describe the magnitude of the refractive index influence, according toIn a previous paper, we have presented a reconstruction quality metric for optical geometry measurements under the influence of a refractive index gradient based on the deviation of corresponding 3D points . For theter cf. ,34) is u is unc=4In the above equation, In reference to the reconstruction quality metric, we presented a setup for compensating the light deflection effect when measuring the geometry of hot specimen . The setA simplified model is used to estimate the influence of a transient illumination on the phase reconstruction using a multi-frequency phase-shift algorithm. Here, the phase quations  and 7)  0≤ϕ≤1 ofThe full 3D measurement setup comprises four cameras (AV Prosilica GT), one projector (WinTech PRO4500) and one thermographic camera (InfraTec Vario-CAM HD—infrared (IR) wavelength range (7.5–14) enced to ,19.To analyze the effect of the reduction of radiation from self-emission during cooling, the acquired images during a full-field illumination of the specimen are examined. The exposure time is chosen to be 80 Assuming a linear intensity gradient n. Assuming a sequence with In this equation In this expression, For the regular 12-image and high-frequency 8-image sequence (cf. Equation ), both wFor the low-frequency 8-image sequence, Equation  under thIn this case, the index differences are not equal reduce the influence of compensate for the intensity gradient through an estimation.The observations from the use of the forced air flow actuator show a darkening of the captured images over time, which is similar to the reduction of radiation from self-emission caused by the cooling of the object through the air flow cf. . Due to To reduce the absolute intensity difference between two subsequent images (1.), it is proposed to change the order in which the patterns are projected. Projecting the images that exhibit a phase-shift of To compensate for the intensity transient, it is necessary to estimate its magnitude in a fringe projection sequence. By this means, Equations –10) can can10) cThe influence of a temporal intensity gradient in a sequence of multi-frequency phase-shift images is examined using the simulation model. By this mean, different gradient functions are modeled see a. For thThe projection magnitude is kept constant at see both c,d. The For the stated reasons, the difference to the nominal phase is only a function of the intensity transient in the highest projected frequency. This is observable through the comparison of the differences in Due to the shown significance of the intensity gradient on the phase calculation, the proposed solutions need to be examined. Both are separately analyzed using a synthetic intensity gradient induced into the projection sequence in the hardware setup. Using 8-bit projection images, the amplitude is set to Equation ). The seTo analyze the effect of an intensity-gradient compensation based on an intensity estimate (cf. (2.) from The presented methods are also evaluated in the design application, i.e., when measuring hot components. By this means, the compensation method for the light deflection effect  is used While the intensity estimation from the used regular sequences has not yet shown to be difficult, the necessary extrapolation of the intensity to compensate the gradient in the outer parts of the sequences, i.e., image indices In our opinion, the enhanced adapted sequence offers a solution to the trade-off between the added measurement time through the addition of new projection images and the expected quality of the phase measurement. It contains the same number of images as a regular 12-image sequence, but offers a high number of support points for the intensity estimation, as the average intensity can be calculated from a rolling average through the images quations ) for theThe reduction of the influence of the intensity transient through the use of the adapted sequence as well as the compensation of such a transient through intensity approximation are evaluated using different methods. The adapted sequence is evaluated under the influence of a deterministic gradient in an experiment with a synthetic intensity transient while the enhanced adapted sequence is evaluated under the influence of a hot measurand in a workshop experiment. The workshop experiments are conducted w.r.t. the delay time between activation of the flow actuator and the start of the measurement. This delay time y metric . In the  used in  and was  used in .The results for the experiment with the synthetic intensity gradient are shown in For the evaluation of the adapted sequence cf. , the difExamining the results of the intensity-gradient compensation by estimation see c, both uThe following results were acquired in the experiment with the hot cylinder in the workshop. For these experiments, the quality of the intensity estimation is evaluated on a per-pixel basis using the variance of the average intensity This approximation quality For the proposed adapted sequence from The direct influence of the intensity gradient compensation on the reconstruction of 3D geometry can be analyzed using the previously established reconstruction quality estimation based on the variance of corresponding 3D points using multiple cameras . The recThe acquired results show similar median deviations for all four measurements , e.g., The limits of the intensity transient compensation through approximation appear to be mainly due to the used approximation function. The linear approximation fails to produce high quality geometry reconstruction results for In this paper, we have investigated the increased reconstruction deviations when trying to compensate for the light deflection effect caused by a hot measurement object using a forced flow actuator. The intensity gradient caused by the cooling of the object through the air flow was postulated to be the reason for this increase. A synchronized acquisition of grayscale images and thermographic images confirmed this effect. Its influence on the multi-frequency phase-shift measurement was investigated on a theoretical level, in a 1D model simulation as well as in a 2D experiment using a synthetic intensity gradient. From the conclusions of these experiments, a new adapted projection sequence was proposed and implemented. A method for the approximation of the intensity gradient was presented and evaluated. The evaluation was conducted using different methods based on the novel projection sequence and the previous established reconstruction quality metric. The new method did improve the reconstruction quality level of the measurements of the hot object significantly compared to not using a flow actuator and compared to using a flow actuator with a regular projection sequence, but could not reach the level of the cold reference measurement.The enhanced adapted projection sequence in combination with the established forced flow actuator and the intensity transient compensation is significantly improving the reconstruction quality when measuring the 3D geometry of hot objects. The main disadvantage of the adapted enhanced sequence is the extension of the necessary projection time Generally, the adapted sequence with swapped projection image indices cf.  can be aTo generalize the results beyond a singular measurement object, a semi-automatic selection of the delay time is needed with the use of a single-camera single-projector FPS system. By this means, the projection of a rolling high-frequency phase-shift sequence can be used in combination with Equation . In the ibration  with a pibration .


Networks in the human brain are extremely complex and sophisticated. The abstract model of the human brain has been used in software development, specifically in artificial intelligence. Despite the remarkable outcomes achieved using artificial intelligence, the approach consumes a huge amount of computational resources. A possible solution to this issue is the development of processing circuits that physically resemble an artificial brain, which can offer low-energy loss and high-speed processing. This study demonstrated the synaptic functions of conductive polymer wires linking arbitrary electrodes in solution. By controlling the conductance of the wires, synaptic functions such as long-term potentiation and short-term plasticity were achieved, which are similar to the manner in which a synapse changes the strength of its connections. This novel organic artificial synapse can be used to construct information-processing circuits by wiring from scratch and learning efficiently in response to external stimuli. The human brain was first described as a complex network of innumerable neurons connected via synapses in the early 20th century. This discovery inspired various mathematical models for information processing, including models by McCulloch and Pitts , which hNew processers based on the concept of neuromorphic hardware have emerged in recent years, where several studies have aimed to reproduce the functions of brain synapses by skillfully utilizing the unique electrical properties of various materials ,4. SpeciPEDOT:PSS has been previously grown into the shape of a wire during electrodeposition under a bipolar square-wave alternating current (AC) voltage between two electrodes immersed in a precursor solution of 3,4-ethylenedioxythiophene (EDOT) monomer and PSS ,14,15,16A previous study by the current authors demonstrated an increase in conductance value between electrodes due to linking of conductive polymer wires to achieve synaptic long-term potentiation (LTP) ,22, whicThis study aimed to demonstrate the use of bridging wires to control the conductance between two electrodes, thereby emulating the way in which a real synapse changes its strength of connection with another neuron. Conductance modification was achieved by controlling the diameter of the bridging wires and reproducing a synaptic LTP function. Synaptic short-term plasticity (STP) describes a temporary change in synaptic connection strength ,27, whicThe growth and linking of the polymer wires were conducted between the tips of an electrode gap, where the conductance was adjusted based on continuous conductance readings . The appp-p) with a 50% duty cycle applied to the working electrode, where the counter electrode was switched to ground (GND). The conductance value between the electrodes was measured at 4-s intervals, which consisted of a 3.5-s wire growth phase and a 0.5-s conductance reading phase. During the reading phase, the counter electrode was switched to a current–voltage (I–V) amplifier with a magnification of 2 × 106. The number of bridging wires was determined using in-situ optical microscopy. The polymer wire growth was manually stopped based on the number of bridging wires and their conductance.Polymer wire growth was conducted under a bipolar square-wave AC voltage  revealed a significant increase in wire diameter near the anode during voltage pulsing, compared to that near the cathode  was observed as the polarity was changed between negative and positive every 200 pulses. The conductance gradually increased with the continuous application of positive pulses, and gradually decreased when the polarity of the applied pulses was negative. This change in conductance continued after repeated switching, demonstrating that the artificial polymer wire synapse had a high switching repeatability. Thus, the conductance can be consecutively and reversibly modified by controlling the number and polarity of the applied voltage pulses. As the polymer wire synapse was capable of reproducing both potentiation and depression, it may be regarded as an artificial synapse with plasticity due to its ability to repeatedly rewrite its conductance value. This conductance control has never been successfully achieved using normal polymer wires with a uniform thickness, and was found to be dependent on an asymmetric polymer wire shape.The application of bipolar voltage pulses with a lower amplitude to the asymmetric polymer wires acquired by the expression of LTP induced a further change in conductance, which included both an increase and decrease a. The chThe retention characteristics of the polymer wire synapse in the low-resistance state (LRS) and high-resistance state (HRS) were evaluated, where the change in conductance over time with an open-circuit was observed b. The inV = 2.5 V, W = 10 ms, T = 2 s) was applied 100 times. Then, the sample was removed from the precursor solution, dried, and immersed in a PSS aqueous solution. While attempting to achieve the STP characteristic by applying small voltage pulses , the conductance change was found to be less likely to occur at a lower PSS concentration . First, the polymer wires were linked between the tips of the electrode gap via electrodeposition in the precursor solution, and the shape of the wires became asymmetric as a periodic voltage pulse :The application of a voltage to the electrode gap linked by polymer wires led to simultaneous doping near the anode and dedoping near the cathode. These reactions would cancel one another in a typical two-terminal system, thereby minimizing the variation in conductance along the entire wire before and after applying voltage. However, an asymmetric diameter distribution along the longitudinal direction of the wire led to disproportionate doping efficiency at the anode side and dedoping efficiency at the cathode side due to the difference in the wire surface area around the anode and cathode. Consequently, the total number of carriers varied along the wire, leading to a change in conductance before and after applying voltage.The changes in conductance caused by the doping and dedoping reactions decayed and converged to the same value within a period of a few tens of seconds to a few minutes. This may have been attributed to the uniform concentration gradient achieved due to PSS anion diffusion over time. The application of voltage to the electrode gap immersed in the PSS solution generated electrostatic forces that caused PSS anions to aggregate near the anode, which enhanced the doping reaction. When the voltage was halted, the PSS anions that had not engaged in the doping reaction diffused readily, while the PSS anions engaged in doping dissociated more gradually. Consequently, the concentration gradient in the electrode gap became uniform over time. Considering the effect of Le Chatelier’s principle on Equation (1), the change in the concentration gradient of PSS anions led to a new equilibrium state of the wire near the electrodes. The reverse reaction was promoted due to diffusion, thereby returning the conductance to the initial value.The polymer wire synapse allowed for easy LTP expression changes according to the frequency of the applied voltage pulses, thereby reproducing the multistore model of human memory . This prThis study facilitated conductance changes between two electrodes linked by conductive polymer PEDOT:PSS wires via the continuous application of voltage pulses to the electrode gap, thereby achieving synaptic functions including LTP and STP. The learning efficiency and retention time were dependent on the frequency and magnitude of the input voltage pulses, thus the system may be regarded as an artificial synapse capable of learning efficiently via selective storage of input information. The polymer wire synapse can be used to construct an information-processing network circuit from scratch and learn efficiently in response to external stimuli. This shows promise for future applications in highly integrated brain-type hardware to reproduce the structure and learning mechanism of the human brain.


The spread of SARS-COV-2 has affected many economic and social systems. This paper aims at estimating the impact on regional productive systems in Italy of the interplay between the epidemic and the mobility restriction measures put in place to contain the contagion. We focus then on the economic consequences of alternative lockdown lifting schemes. We leverage a massive dataset of human mobility which describes daily movements of over four million individuals in Italy and we model the epidemic spreading through a metapopulation SIR model, which provides the fraction of infected individuals in each Italian district. To quantify economic backslashes this information is combined with socio-economic data. We then carry out a scenario analysis to model the transition to a post-lockdown phase and analyze the economic outcomes derived from the interplay between (a) the timing and intensity of the release of mobility restrictions and (b) the corresponding scenarios on the severity of virus transmission rates. Using a simple model for the spreading disease and parsimonious assumptions on the relationship between the infection and the associated economic backlashes, we show how different policy schemes tend to induce heterogeneous distributions of losses at the regional level depending on mobility restrictions. Our work shed lights on how recovery policies need to balance the interplay between mobility flows of disposable workers and the diffusion of contagion. Consequences on economic activities and trade flows have been strong4. In addition to the direct effects of the contagion, mobility restriction policies have been put in place in many countries to limit the spread of SARS-COV-2. Indeed, during pandemics, different types of non-pharmaceutical interventions (NPIs) have been implemented to contain the spread of contagion5. These policy restrictions have produced relevant economic consequences, since disposable workers  are prevented from keeping up their activities9.SARS-COV-2 has caused almost 7 million confirmed cases and 400k fatalities globally as of June 6th 202011. Italy has been the first European country to experience a severe SARS-COV-2 pandemic12, with very high mortality rates in some of the northern regions. Moreover, it has been the first country to impose strong lockdown measures on a national scale, despite an already weak prospected economic outlook: before SARS-COV-2 outbreak, Italy was in fact experiencing some of the lowest growth rates in Europe (according to Eurostat), while the first quarter of 2020 indicates a fall of Gross Domestic Product (GDP) of about 5.3% with respect to previous quarter . Against this background, Italy represents an interesting case study to simulate lockdown lifting scenarios, due to a peculiar trade-off between economic and health conditions.This work aims to study the interplay between the SARS-COV-2 diffusion and mobility restriction measures, and how it affects the Italian productive system. Through a scenario analysis which models the transition into a post-lockdown phase, we investigate the economic consequences of alternative lockdown lifting policies. We study how economic outcomes are influenced by (a) the severity of virus transmission rates across Italian regions and (b) the timing and intensity of the release of mobility restrictions20 that simulates SARS-COV-2 diffusion in Italy. We partition the Italian population into districts21 and we introduce commuters that, moving between different districts, contribute to spread the disease , to highlight the challenge for policymakers in reaching a proper balance between the spread of contagion and economic recovery30. To investigate this trade-off, we explore the range of economic outcomes deriving from different lockdown lifting schemes for the Phase-2. We imagine two extreme cases: one, in which mobility restrictions continue while transmission rates come back to pre-lockdown values, the other, in which mobility is fully restored to pre-lockdown values while transmission rates are kept to the minimum reached during the lockdown phase. Then, we perform simulations with values in-between these two extreme cases by tuning parameters related to the transmission rate and to mobility flows.Our analysis relates with contemporary literature about the side effects on the economic consequences of measures put in place to circumvent the pandemic. Similar to Ref.31. This information is matched with the details on the economic sectors which, starting from 22th March, were allowed to continue their activity during the lockdown phase32.To show how the complexity of Italian productive structure can magnify losses beyond the direct effects of SARS-COV-2, we measure the share of employed individuals at the sector level, in each Italian district. We collect socio-economic variables at the district level from the 15th General Census of Population and Housing developed by the Italian National Institute of Statistics (ISTAT). The database contains information, at sub-municipal levels, on the demographic and social structure of the Italian population. In particular for each district, we rely on information on population size, average income, and the number of workers in different economic sectors, available in the Employment Register created in 2011 on the occasion of the CIS2011 Virtual Business Census and updated annually, starting from 2012Thus, our estimates of the economic backslashes of the epidemic will result from the interplay of three sources of variability: the range of possible values of the epidemic parameters, the evolution of mobility patterns and the productive structure of the economy.36. Accordingly, public health interventions have focused mainly on mobility restrictions, with a varying degree of intensity: school closures, bans of public events, curfews, social distancing and self-isolation37. Given the rapid spread of contagion and the high death toll in several countries, lockdowns have been enforced to err on the side of caution, hence the effectiveness of each measure has not been accurately tested and the debate is still open on which intervention (or combination of interventions) may lead to successful results38. Nevertheless, increasing evidence has focused on highlighting the role of mobility restriction in effectively curbing the rate of contagion43, hence in what follows we will use a simplifying assumption and we will attribute the variation in the effective rate of reproduction during lockdown entirely to the variation in mobility. This allows us to focus only on those details which are functional to estimate the economic losses during lockdown and after lockdown lifting. For the same reason, in the scenario analysis for Phase-2, we do not model the epidemic dynamics and we explore the space of mobility and contagion parameters, using as starting point estimates of the effective reproduction number obtained from Ref.27.The role of mobility patterns in explaining contagion dynamics has been a central finding in the recent literature on SARS-COV-244, which assumes individuals uniformly distributed in space and randomly moving (see Ref.45 for an interesting modeling exception). On this issue, our work improves over previous models by using real data on movements of individuals in the simulations within a metapopulation model tailored to address the role of mobility in the contagion. On the other hand, however, we have used a simplified model of losses, which includes only direct losses from infections and deaths  and job losses from NPIs. Following the classification made in Ref.44, there are three sources of economic damages which our model does not include: change in consumption and investing behavior of households and firms, disruption of trade and global value chains, and hysteresis effects, i.e. damages to the capacity of the global economy to achieve growth in the long run. Including all these effect in our model would increase our estimate of reported losses, but at the same time would also introduce new sources of uncertainty in the estimation procedure due to the number of parameters required to calibrate the model. For this reason we have chosen to focus here on the first two types of damages and use a more data-driven approach.While there is a well-established literature incorporating mobility patterns in epidemic modeling, economic models of epidemics have mostly focused on what is called the SIR macro modeli. According to the model, restrictive policies which affect national mobility have a two-fold effect: first, they contribute to slow down the infection dynamics, thus increasing the stock of disposable workers (not infected); second, they also limit the number of actual workers which are allowed to produce. For the Italian case, indeed, during the lockdown phase only a subset of essential sectors was allowed to continue their economic activities, while the others were forced to close when remote work was not feasible49. We assume that disposable labor force (from now on we will use “labor force” and “workers” interchangeably) in the ith district at each instant t, is a fraction of the total labor force N(t):actual labor force, i.e., the number of workers that are effectively active, on the contrary varies based on mobility restriction measures: it coincides with the disposable force during the pre-lockdown phase only (Pre), while during the lockdown phase (Lock) it is composed by the workers employed in the essential economic sectors which were allowed to continue their activities50 and, moreover, by the fraction of the disposable labor force which could perform “smart-working”, 10. Finally, during the Phase-2 (Unlock), we assume that a varying fraction i as:E indicates the set of essential sectors allowed to produce during the lockdown phase. Notice that the parameter The epidemic model that we adopt see “” provide51 and the daily Intensive Care Unit (ICU) cost at 54. We then collect information on the fraction of infected individuals needing different kinds of medical care from the Italian Civil Protection bulletins55; in particular, we take the average of both quantities for each district over the entire period of observation to avoid fluctuations in the measurements. These measures must be seen as a lower-bound for costs since they refer to services provided during ordinary time and do not account for the extra costs deriving from the epidemic. Moreover, we do not attempt to make any estimate of costs incorporating the value of statistical life57. On the one hand our focus is to estimate economic losses only and not perform a cost-benefit analysis as in Refs.60. On the other hand, we want to be parsimonious about our set of assumptions.To measure the aggregate economic loss for each district, we average the municipal level per-capita incomes to their corresponding district level, from which we obtain the daily district income per person  to 42 days, from the 26th of January to 8th of March. We then impose that the lockdown phase (Lock) is put in place for 46 days, from the 9th of March up to the 3rd of May. Finally, we start Phase-2 (Unlock) on May 4th  and retained only municipalities which are present also in our dataset . Besides, we build an averaged graph of mobility over a window of 14 days before national lockdown (9th March). Then, we performed Pearson’s correlation test to the following metrics: Degree and Strength of nodes, and Weight of edges. In all cases we find a significant positive correlation at level 16, designed at district level in order to match economic variables , which provides movements of workers/students travelling between municipalities that were recorded in the last census. We tread carefully in the comparison, first because the commuting network retains only residents who travel for either work or study reasons, and second because the information refers to the Italian mobility patterns of 9 years ago. On the other side this data is not biased towards individuals who own a phone and are registered on social networks, as in Facebook data, therefore we believe it is useful to further validate the representativity of the Italian mobility data observed before entering the lockdown phase. To make a consistent comparison we filtered out those paths where the number of travelers is less than 10  scenario national lockdown is first put in place and then lifted, assuming that the transmission rate is held constant  during the re-opening phase; however, as we show in the following, we mainly focus on the epidemic dynamics during the lockdown phase to assess the efficiency of restriction measures to contain the epidemic spreading. Hence from Pre-lockdown values. During the lockdown phase , whereas southern regions are spared from the contagion . Figure However, as Eq.  suggestsThe combined effect on the Italian labor force of the epidemic and mobility restriction policies per region is reported in Supplementary Fig. The two contrasting effects of the epidemic and lockdown measures on the economy are translated into monetary losses by multiplying the per-capita daily income (at the district level) and the actual labor force in the district Figure So far, we did not take into account the uncertainty related to Phase-2, in which information on transmission rates and mobility restrictions are not available in advance. Therefore, we evaluate the impact of lockdown lifting through a scenario analysis, i.e., we simulate the number of infected individuals after lifting the lockdown by varying parameters Figure i at simulation time t. Again, for sake of interpretability, we mapped the parameter We further leverage our pool of simulations to model both the actual labor force and the economic loss in each district using Eqs.  and 4)..4). In FFor increasing values of Finally, we compare the evolution of the aggregate economic loss across regions in Phase-2 for different simulation scenarios keeping constant the total aggregated loss. Indeed, despite the fact that multiple configurations of see Fig. . Top panThe political debate on the transition to a post-lockdown phase  is affected by several unanswered questions on the timing and modality in which governments should lift mobility restrictions. On the one hand, a second wave of the epidemic could take place after lifting the restrictions. On the other hand, prolonged mobility restrictions would damage the economy. The uncertainty on the economic consequences of a second wave of restrictions is strengthened by the complexity of the global economy, where direct losses to a specific sector also entail second order effects to other sectors and to international trade partners. Including all these variables in a scenario analysis would require complex dynamic macroeconomic models in which policy alternatives embed welfare functions and citizens utility preferences, along with the impact of investments and fiscal interventions, heterogeneity of policies across regions, and macroeconomic unbalances from world trade shutdown.Against this background, this paper proposes an illustrative micro-level framework to study the impact of possible alternative policy responses. We recognize that economic losses arise mainly from two main sources: the number of infected people and the restrictions in mobility that prevent individuals to work. Mobility restrictions, at a first glance, are beneficial for containing high losses in the number of available workers as they mitigate the epidemic spreading. However, restrictive policies have also a direct detrimental effect on economy as they force non-infected workers to vacate workplaces. Our study reveals that the aggregate reduction of disposable income can vary from − 10% up to − 40% in the worst case, and that the outcome is the result of non-linear interactions between mobility policies and infection transmission rates. In addition, we show that different combinations of parameters producing a given economic loss at the national level correspond to heterogeneous regional losses. These effects are shown to depend on the geographical impact of the disease and of the socio-economic structure of each territory. Despite the simple SIR model governing the spreading disease and the inherent approximation due to how we track mobility, our findings contribute to inform policy design as they suggest the adoption of geographically tailor-made policy actions instead of a one-rule-fits-all approach to mobility restrictions.66. Districts differ not only in their number of inhabitants, but also for what concerns transportation networks and commuting patterns among areas. To account for such heterogeneity we leverage a Susceptible-Infected-Recovered (SIR) model with metapopulation67 by subdividing the Italian population into districts. Each district has its own independent epidemiological dynamics, and commuters can further spread the disease into other districts. Since our main focus is on the economic effect of the SARS-COV-2 epidemic, we keep the transmission scheme as simple as possible to avoid confounding effects68.The spatial structure of populations is a key ingredient for understanding the dynamical spreading of an epidemic, in particular in a territory where areas have a high degree of heterogeneity, such as the Italian peninsulai that live in district j. The metapopulation SIR model is described by the following equations:i that live in district j is Let ith district. Finally, the parameter Matrices 69, i.e., to ensure that the output is realistically informative on the epidemic spreading. Thus, we leverage a massive dataset of near real-time observations provided by Facebook24 through its Data for Good program. In this way we can tune model parameters concerning the mobility rates before and during the lockdown phase, Real data is of paramount importance to calibrate the modelFacebook Disease Prevention maps provide information on movements of individuals with their geo-positioning option enabled during the period of observation, which goes from February 24th to March 23rd. Our dataset consists of approximately 800,000 distinct observations (recorded with a 8-h frequency) covering movements of nearly 4 million daily individuals (on average) across 3000 Italian municipalities. This data is not publicly available but it can be released to non-profit organizations and academics upon agreement with Facebook.70, which we aggregated at increasing territorial levels to perform our analyses. The aggregation process consists in summing together all observations belonging to the same territorial level , hence it does not entail any loss of information as each tile is assigned to a unique territorial unit. Furthermore, we represent mobility using a directed weighted graph formulation where nodes are territorial units and edges are weighted based on the amount of traffic of individuals flowing between two locations.Each observation in the Facebook dataset describes movements of individuals across tilesPre and Lock phases by aggregating together daily mobility over two 14-day windows, before and after the day of the intervention (March 9th), respectively. Let x to municipality y at day t, i.e. the number of individuals moving between the two locations. The estimates of entries in matrices i at time t for district j.For the mobility analysis we use data at the municipality level, whereas in order to simulate our epidemic model we further aggregate mobility flows among municipalities at the district level. In particular, we built two snapshots corresponding to 26.The binary version of the matrix 27z indicates an Italian region, for each district located in region z. In formula:69, where authors assume that the effect of national lockdown and other governmental interventions, such as school closure, social distancing or self-isolation, is to reduce the original 36. Besides, the reduction of internal mobility estimated with Facebook data is in accordance with latest reports on the decreasing trend of Italian mobility provided by Google71. However, we remark that the model only aims to show the economic effect of different scenarios rather than providing an accurate forecasting of the epidemic evolution.For calibrating the district specific transmission rate during the pre-lockdown phase (To study the economic impact of the epidemic after the lockdown phase, we perform simulations with values in-between two extreme cases. An over-pessimistic scenario in which mobility restrictions continue while the transmission rates turn back to their pre-lockdown values and an over-optimistic scenario in which mobility is fully restored to pre-lockdown values while transmission rates are kept to the minimum reached during lockdown phase. Thus, let Supplementary Information.