Writing is a means of communication which requires complex motor, perceptual, and cognitive skills. If one of these abilities gets lost following traumatic events or due to neurological diseases, handwriting could deteriorate. Occupational therapy practitioners provide rehabilitation services for people with impaired handwriting. However, to determine the effectiveness of handwriting interventions no studies assessed whether the proposed treatments improved the kinematics of writing movement or had an effect at the level of the central nervous system. There is need to find new quantitative methodologies able to describe the behavioral and the neural outcomes of the rehabilitative interventions for handwriting. In the present study we proposed a combined approach that allowed evaluating the kinematic parameters of handwriting movements, acquired by means of a magnetic resonance-compatible tablet, and their neural correlates obtained simultaneously from a functional magnetic resonance imaging examination. Results showed that the system was reliable in term of reproducibility of the kinematic data during a test/re-test procedure. Further, despite the modifications with respect to an ecological writing movement condition, the kinematic parameters acquired inside the MR-environment were descriptive of individuals’ movement features. At last, the imaging protocol succeeded to show the activation of the cerebral regions associated with the production of writing movement in healthy people. From these findings, this methodology seems to be promising to evaluate the handwriting movement deficits and the potential alterations in the neural activity in those individuals who have handwriting difficulties. Finally, it would provide a mean to quantitatively assess the effect of a rehabilitative treatment. Handwriting is historically believed to be one of the most difficult fine motor skills to learn. It takes years of practice before a person has mastered the mature handwriting skill. Despite the difficulties this ability can pose to the learners, handwriting is one of the first motor expertise that children acquire at school and is nowadays part of the individual motor-cultural baggage. Although the increasing use of personal computer has reduced the time dedicated to handwriting, adults still use handwriting to communicate with others, to record ideas and for creative expression.Handwriting is a complex functional activity, which involves fine motor skills, cognitive and visual-perceptual processing . In adulThe restoration of handwriting is matter of the occupational therapy treatments. Unfortunately, many of the conventional methods used to evaluate handwriting movements during the clinical routine are not standardized and are dependent on the therapist’s expertise . From twA step further toward a comprehensive description of the laws that govern handwriting is the acquisition of the neural correlates of writing movements with the aim to investigate the relationship between the behavioral data and the cerebral activity. In this context, functional magnetic resonance imaging (fMRI) has become a powerful tool to study the functional organization of the brain , allowinper se. To achieve this goal we: (a) assessed the reproducibility of repeated measurements of handwriting movements acquired by means of a newly designed and developed MR-compatible tablet outside the magnet bore, (b) tested the reliability of the kinematic data acquired inside the MR scanner, where participants had a visual simultaneous visual feedback of their written trace, by comparing them with those obtained in ecological condition, and (c) investigated the neural correlates associated to handwriting by means of fMRI sequences.Aim of the present work was to implement a methodology able to quantitatively characterize the kinematics of handwriting movements and the related neural substrates. Indeed, in this study we considered the temporal and spatial patterns of handwriting, omitting the evaluation of writing This study was composed of two experiments: Experiment 1 assessed the reproducibility of the kinematic data acquired with the MR-compatible tablet by means of a test–retest approach. Since this was the first study that used this system  and this methodology, it was crucial to test its reliability. Experiment 2 tested whether the individual kinematic features associated with handwriting movements performed in ecological conditions are maintained when participants performed the task inside the MR scanner. Further, in a sub-group of subjects, who took part to Experiment 2, we tested whether the proposed kinematic paradigm was able to evoke the activation of the cerebral areas usually active during a handwriting task.A total of 44 subjects were enrolled in this study. Twenty-two of them  participated in Experiment 1, whereas 22  took part in Experiment 2. Seven out of the 22 subjects who took part to Experiment 2 were also recruited for an fMRI examination. Participants who manifested MR contraindications, as claustrophobia, pregnancy, and presence of a pacemaker and other metallic parts not magnetic resonance compatible were not considered eligible for Experiment 2. All subjects were right-handed according to the Edinburgh Handedness Inventory  and naivFigure 1).An innovative MR-compatible tablet, the SMART TAB , was used to acquire handwriting movements during the evaluation sessions outside and inside the MR scanner. The SMART TAB consists of a touch-sensitive tablet, a plastic-made stylus, a USB controller box, and a cable that connects the tablet with the controller outside the magnet room. All the equipment inside the magnet room is non-ferromagnetic. The surface of the tablet is the AccuTouch Five-Wire Resistive Touchscreens , where both X and Y measurements are made on a stable rear glass layer. The touch surface  is mounted within a frame that delimits the sensitive area while offering some protection from unintentional touches. The subject can use the fingers or the plastic stylus to write on the tablet (Touch Activation Force less than 113 g). The tablet is connected to the USB controller box in the operator console room via a shielded cable. In turn, the controller box is connected via USB cable to the PC that manages acquisition  of the pen-tip acquired at a sampling frequency corresponding to the refresh rate of the video projector (60 Hz). In the present study the pen was either an ink stylus  or a plastic made stylus without ink . When the stylus touches the screen the PC starts to acquire the data, and a black line reproducing the written trace appears. Thus, participants can monitor in real-time what they are writing. None of the subjects reported latency between their executed movement and representation of the PC monitor.The software handling the acquisition of writing movements was developed in our laboratory in MatLab Toolbox , a free Participants were seated on a chair at a table. The SMART TAB was placed on the table and the participant could adjust its position to feel as comfortable as possible. Once a comfortable position was achieved the participants were requested to keep the orientation of the tablet and of their forearm constant. A plastic, MR-compatible stylus was used to perform the task directly on the tablet surface. Participants were required to look at the computer screen where the written trace was displayed simultaneously to the motor task. They were asked to write the Italian sentence “Il carro sale al colle” , keeping the text in a single line. This sentence was chosen because it was short and easy for the subject to keep it in a single line, and because it was composed of simple words very popular in the Italian language. The sentence was repeated five times. Participants were tested twice , one month apart, to assess possible changes in measurements taken under the same condition and to define the motor performance parameters showing good repeatability .This experiment was composed of two sessions, which involved 22 participants. The first session (OUTSIDE) was aimed at evaluating handwriting movement outside the magnet bore, in ecological condition, i.e., participants were seated on a chair at a table. As in Experiment 1, the tablet laying on the table was positioned by the participant to feel as comfortable as possible. In addition, in order to mimic a conventional handwriting condition while recording movement kinematics, a paper with black, horizontal lines was positioned over the SMART TAB surface and the subject was provided with an ink stylus. In this way, the participant had a visual feedback over the paper of what she/he was writing and could appreciate the normal resistance of the pen over the paper during the handwriting movement; in the meantime, the tablet acquired the trajectory of the stylus and the experimenter could simultaneously monitor the writing motor output on the computer screen. The subject was asked to write three times the Italian sentence “Il sole scalda”  when a “go” signal was provided by the experimenter. As in Experiment 1, this sentence was chosen because it was composed of simple words very popular in the Italian language. We adopted a shorter sentence with respect to that of Experiment 1 in order to make the task easier for the subject inside the MR scanner.Magnetic Resonance Imaging Protocol). For this reason, participants were instructed to write the sentence on a first line and to repeat it on the lines below until a “stop” signal was provided. Each subject completed three task blocks, containing at least one complete sentence each, interleaved with 30-s periods of rest. The sentence was the same adopted for the OUTSIDE condition. During rest blocks subjects had to handle the stylus without moving and to keep the eyes opened, as in the task blocks, to avoid spurious findings related to visual activations. Each session was preceded by a familiarization phase which allowed subjects to train themselves with the SMART TAB in this non-conventional writing condition.The second session (INSIDE) took place inside the magnet bore. The subject laid down with the SMART TAB positioned on her/his body at the level of the pelvis. The forearm was positioned over an MR-compatible support made of soft material, adjustable in height to minimize the subject’s forearm movements. A pad was placed under the tablet to help the subject to keep a position as comfortable as possible during the handwriting movement. A plastic, MR-compatible stylus was used to perform the task directly on the tablet surface. The participant had a simultaneous visual feedback of her/his motor performance by looking at a mirror positioned on the head coil. The mirror reflected the images projected onto a white screen placed inside the magnet room, where a video-projector, connected to the computer used for the acquisition, displayed the written trace. The horizontal lines over the writing space were physically created by mounting equidistant, black cotton threads over the white screen inside the magnet room. The task was adapted to cover a temporal interval of 30 s, which corresponded to the active task blocks of the fMRI acquisition  for fMRI. Particularly, each fMRI run included 63 brain volumes; the first 3 volumes were discarded because of non-steady magnetization. Within each run the subject performed the handwriting motor task  alternatively with a rest condition , according to a block designed paradigm consisting of 30-sec active task periods alternating with 30-s control periods (10 brain volumes per block).Seven out of 22 subjects, who executed the two experimental sessions of the Experiment 2, during the INSIDE session were also examined by means of an fMRI test. MRI examination was performed on a 1.5 T MR system  and included the following series covering the whole brain: axial FLAIR sequence  to exclude incidental findings in the enrolled subjects; axial T2-weighted sequence  used as structural reference for the fMRI acquisition; T2® software. The software automatically detected the beginning and the end of the sentence on the basis of the module of the velocity profile : the first and the last instants  in which the velocity was greater than zero value corresponded to the beginning and the end of the sentence, respectively. Total duration (s) , movement length (mm), and thickness (mm) of the sentence were considered as outcome parameters. The length corresponded to the size of the segment which connects the first and the last points of the sentence. The thickness was computed as the vertical distance between the top of the highest letter and the bottom of the lowest one.The kinematic parameters of handwriting movements were computed by means of a custom-made MatLabSPM12 software  was used for fMRI processing . For eact-test compared the mean kinematic parameters  acquired for each subject the first time the participant used the MR-tablet with those acquired one month later. Further, to assess the reliability of the acquisition, a linear regression model  tested the relationship among the single-values of the kinematic parameters acquired during the first evaluation and after one month on the same subjects. Pearson’s correlation coefficients are provided together with the associated p-value, the slopes and the intercepts values of the regression lines. To be more accurate the test–retest reliability of the motor performance parameters was assessed also by the Intraclass Correlation Coefficient (ICC) and the corresponding p-value , a linear regression model (∗ INSIDE + bOUTSIDE = a ) tested the relationship between the kinematic parameters in the two experimental conditions. Pearson’s correlation coefficients are provided together with the slopes (a) and the intercepts (b) values of the regression lines and the associated p-values. As in Experiment 1, the ICC and the corresponding p-value were computed for each parameter.To compare participants’ motor performance inside and outside the magnetic bore, the mean kinematic parameters  were compared by means of paired t contrast images were created for each subject, with a height threshold of p < 0.05 FWE-corrected and extent threshold arbitrarily set at k = 50 voxels.Concerning the fMRI data, a general linear model was used to identify the voxels with task-related signal changes at the individual level. Task-related Data in the text are reported as means ± SD.Figure 2A shows the first sentence written by a representative subject the first time she was tested (PRE) and the first sentence she wrote 1 month later (POST). No evident differences in the sentence dimensions can be observed from these representations of the written trace.t-tests showed that the length of the sentence , as well as the thickness  and the total duration of the writing movement  did not change when retested after one month from the first acquisition.The results of the Figure 2B shows the linear relationship between the kinematic parameters of each single trial acquired during the first and the second testing session . Significant correlations were found between the two sessions, as shown in Table 1, indicating that the system  provided reliable and repeatable data. Table 1 reports also the ICC values for the three kinematic parameters which were found to be statistically significant. The ICC values for length and total duration were between 0.40 and 0.75, thus suggesting a fair to good reliability, whereas the ICC value for thickness was higher than 0.75, a value conventionally considered as representative of an excellent reliability . Similarly, sentence thickness increased significantly when the subjects laid down inside the scanner , as well as the total duration of the writing movement .The length of the sentence was significantly longer inside than outside the MR scanner  between the two conditions for each parameter , indicating that the individuals’ movement features were preserved in the MR environment. Further, the ICC values for all the kinematic parameters ranged between 0.40 and 0.75 which suggested a fair to good reliability  in one representative subject on a rendered brain surface. As reported in details in Table 2, significant activations were found bilaterally in the frontal and parietal areas, as well as in the cerebellum. In particular, we observed significant clusters of activation with peaks in the precentral and postcentral gyri  including the left superior and medial frontal gyri. Further, the left superior parietal lobule (BA 7), and the right cerebellum were significantly activated during handwriting with respect to rest.Concerning the analysis of fMRI data, as a first result, the healthy volunteers showed full feasibility of the procedure. In the activation maps of the single subjects, no artifactual cluster was observed. The present study describes the development and validation of a new methodology that enables to acquire the kinematics of handwriting movements during fMRI studies by means of an innovative MR-compatible tablet. In the literature few studies analyzed the kinematics of handwriting and the related neural correlates , and nonExperiment 1 assessed the reproducibility of the kinematic data in two sessions performed one-month apart, giving information concerning the reliability of the system. The results of the acquisitions showed that the kinematic parameters characterizing the sentence the first time the subjects wrote on the tablet did not significantly differ from those acquired after 1 month. Indeed, although a tendency toward the decrease of movement duration and amplitude could be found 1 month later, the existence of significant and positive linear relationships between the first and the second testing times, whose slope values correspond to a moderate-to-strong correlation, suggests that the system is able to provide repeatable and reliable data, as also confirmed by the significant ICC values.Experiment 2 was performed to test whether and how the kinematic parameters of the sentence were modified when the subjects performed the writing test inside the magnetic bore. Indeed, several factor could potentially affect the measurements and invalidate the test. Firstly, the posture of the subjects was dramatically different from a natural writing posture; the subjects laid down inside the scanner and the tablet was positioned at the level of the pelvis. Some studies specifically focused on evaluating how different body postures influence handwriting movements, producing contrasting results. Although someone did not find any significant tendency regarding posture variation in participants’ signature , others As expected, the results of the comparisons between the kinematic parameters obtained in a writing ecological condition and when the subjects laid down inside the MR scanner showed significant differences; the total movement duration, as well as the height and the length of the sentence significantly increased inside the MR bore. On the other hand, for each parameter, the analysis of the relationship between the acquisition sessions INSIDE and OUTSIDE showed weak-to-moderate, but nevertheless significant correlations, as well as significant ICC values. Therefore, even if the slopes values do not approach the identity , these findings suggest that the individual features that characterize the subject’s movement with respect to the other participants were sufficiently preserved in the MR environment despite the previously mentioned difficulties. Namely, a subject who was slower than others in ecological writing condition was slower during fMRI, too. Therefore, although with some obvious differences, the kinematic evaluation of writing movements acquired inside the MR scanner can be considered sufficiently descriptive of the individual motor behavior and of the individual differences among subjects. Although this methodology is informative about the temporal and spatial features of handwriting, this study was limited to the description of these kinematic features of writing movement, without assessing differences in the morphology and topology of the letters. This limitation could be considered in future studies that, applying the methodology here proposed, will provide a more comprehensive description of the handwriting outcome.During the fMRI study, the task required an overt movement execution, which involved hand and fingers’ movements (the forearm was positioned over an MR-compatible support to minimize the subject’s forearm movements); however, participants were able to maintain the head stable and this guaranteed to reduce motion artifacts. No image artifacts due to the presence of the MR-compatible tablet were observed. Further, the fMRI analysis presented here for a representative subject showed the significant activations of brain regions which have been previously described as candidate cortical sites for writing, i.e., the posterior part of the left middle frontal gyrus and the left superior parietal lobule , togetheIn the present study we proposed a combined approach that allows evaluating at the same time the kinematic parameters of handwriting movements and the related brain activity. The system has proven capable of providing reliable kinematic data. Then, despite the expected modifications with respect to an ecological writing movement condition, the kinematic parameters acquired inside the MR scanner were largely descriptive of individuals’ movement features. Further, fMRI results indicate that the cerebral regions activated in this condition were those expected to be involved in an handwriting task in healthy people. Following all these findings, we suggest that this methodology can be promising to evaluate the behavioral performance and simultaneously the brain activations in persons with handwriting difficulties, and provides also a tool to quantitatively evaluate the effects of a rehabilitation treatment. Further studies are also needed to investigate the correlations between the handwriting kinematic parameters and the imaging data in a large number of healthy participants and people with handwriting deficits, where abnormal brain activations might correspond to altered motor behavior.AB conceived and designed the work, performed the acquisition, analyzed and interpreted the data, wrote the manuscript. LP performed the acquisition, analyzed and interpreted the data, revised the manuscript. LB conceived and designed the work, analyzed the data, revised the manuscript. PR interpreted the data, drafted and revised the manuscript, gave the final approval of the version to be published. GB interpreted the data, drafted and revised the manuscript, gave the final approval of the version to be published. MB conceived and designed the work, interpreted the data, revised the manuscript, gave the final approval of the version to be published.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Crowd sensing is a powerful tool to map and predict interests and events. In the future, it could be boosted by an increasing number of connected vehicles sharing information and intentions. This will be made available by on board wireless connected devices able to continuously communicate with other vehicles and with the environment. Among the enabling technologies, visible light communication (VLC) represents a low cost solution in the short term. In spite of the fact that vehicular communications cannot rely on the sole VLC due to the limitation provided by the light which allows communications in visibility only, VLC can however be considered to complement other wireless communication technologies which could be overloaded in dense scenarios. In this paper we evaluate the performance of VLC connected vehicles when urban crowd sensing is addressed and we compare the performance of sole vehicular visible light networks with that of VLC as a complementary technology of IEEE 802.11p. Results, obtained through a realistic simulation tool taking into account both the roadmap constraints and the technologies protocols, help to understand when VLC provides the major improvement in terms of delivered data varying the number and position of RSUs and the FOV of the receiver. Vehicles are becoming sensing tools, paving the way to new challenging applications ranging from safety to environmental sensing and prediction of driver behaviors. A car today is equipped with over 4000 sensors that continuously collect internal and external data . In addiAmong the wireless enabling technologies, we are assisting at the race between IEEE 802.11p and Cellular-V2X (C-V2X) to win the podium of vehicle-to-everything (V2X) communications . On the Independently on which one of these two radio frequency (RF) technologies will reach the maximum diffusion for on board integration in the next future, recent years also assisted to the increasing interest in visible light communication (VLC) to enable vehicular visible light networks ,7,8,9. TRecent literature investigates the use of VLC in vehicular networks in terms of feasibility and in comparison or in addition to dedicated short-range communications (DSRC) . As for However, the performance of vehicular VLC networks, either when VLC is adopted alone or together with other wireless communication technologies still needs further investigations.How can VLC support sensor sharing?Can the use of VLC improve the rate of data delivered at the RSUs when adopted in addition to other RF technologies?Which is the impact of number and position of RSUs?Can typical routing algorithms impact differently varying the field of view (FOV) of the receivers?Given these considerations, in this paper we focus on how VLC can supplement crowd sensing and which is its role to achieve the vision of driving automation. Specifically, in this work we aim at evaluating the impact of VLC in vehicular crowd sensing applications to deliver the data, sensed and collected on board to the RSUs, assumed integrated in the traffic lights. Specifically, in this paper, we face the following questions:The paper is organized as follows: in LEDs are more and more diffused since cheap and energy efficient;VLC uses the visible light spectrum (380–780 nm) offering (about a 1000 times) greater bandwidth compared to the RF bandwidth;the visible light spectrum is unlicensed;VLC is safe for the human health;VLC does not interfere with electromagnetic devices ;the limited penetration capabilities allow higher reuse factors and lower interference from neighbors.VLC represents an alternative and a complementary technology to the classical RF wireless communications. The interest in this kind of communication has recently intensified and this is mainly due to the following reasons:Beside these advantages, it has to be remarked that VLC is susceptible to ambient light and weather conditions, it covers limited distances and does not cross obstacles ,9.The high reliability and efficiency of LEDs made this kind of lights widely used in vehicle headlights, turn signals, taillight and stop lights. Given also their lifetime, LEDs are diffused in the new city lights and in traffic lights as well. However, in spite the numerous advantages, VLC systems have been investigated in vehicular scenarios only recently.The several advantages of VLC systems over RF systems and some potential algorithms to jointly manage the two technologies, have been already surveyed and demonstrated in indoor scenarios . In 18]18], for Differently from indoor scenarios, in outdoor vehicular networks, challenging tasks still need to be faced .Some outdoor experiments successfully demonstrated data rate of the order of 1 Gb/s over hundred meters  and otheTo better understand which kind of receivers are most suitable for automotive application  presentsThe adoption of VLC as a complementary technology for the internet of vehicles has been considered in , where aThe increasing interest in this kind of communication is also demonstrated by the development of the IEEE 802.15.7 standard, which defines the physical (PHY) and medium access control (MAC) layers for the visible light spectrum and allows to support audio and video services also in mobile visible links .IEEE 802.15.7 defines three different PHY layers, that differ for the adopted modulation and coding scheme, as shown in At the MAC layer, the IEEE 802.15.7 standard foresees the use of carrier sensing multiple access with collision avoidance (CSMA/CA): each node transmits only if the medium is sensed as idle after a random backoff interval time; then, the message is considered correctly received only upon the reception of an acknowledgment at the transmitter.Since FEC codes are made available by PHY I and PHY II , in thisHowever, it has to be highlighted that even though the standard identifies vehicular networks as a potential application area, it does not provide any specific implementing rules regarding V2V or V2I. Hence, many developers in the field do not refer to the IEEE 802.15.7 when developing their prototypes. Given this context, a revision of the standard, known as IEEE 802.15.7r1, is under study, addressing vehicular communications as a fundamental VLC use case. In this case, the standard considers the requirements of vehicular communications and aims to enhance mobility, data rates, robustness and the networking protocols . To meetWe address vehicular crowd sensing applications, where vehicles cooperate to sense and collect urban data requested by data centers . SpecifiIndependently on the wireless interface, each vehicle attempts to transmit its data directly to the nearer RSU that will be in charge to forward them to a remote data center; if a direct transmission is not possible , multi hop communication is exploited. The RSUs are assumed positioned at the crossroads; specifically, RSUs for VLC can be considered integrated in the traffic lights or road lamps, thus avoiding expensive installation, whereas RSUs for IEEE 802.11p need to be set up when necessary.When VLC is addressed, transmissions happen through the head or rear LED lights, whereas receptions are typically left to photodiodes, assumed integrated in the vehicles lights themselves, as depicted in The transmitter (LED) is characterized by a certain angle of irradiance and the receiver (photodiode) by its FOV: wider angles provide larger service areas, but also lead to performance degradation because of higher probability of receiving undesired light interference.I the interference power, A transmission between two vehicles occurs if and only if (1) the virtual line connecting them is inside the transmitter angle of irradiance and the receiver FOV; (2) the virtual line connecting them does not cross any other vehicle or building; (3) the received power uated as ,30,31,32tributed . The recan model .H matrices it is possible to realize a simulation without defining single routes. More information on the model can be found in .SHINE reproduces the behavior of interworking access-networks, taking care of all aspects related to every single protocol level of each access technology affecting the achieved performance. Its realization was planned to overcome the limitations of off-the-shelf network simulation tools in terms of capability to simulate the behavior of heterogeneous networks operating simultaneously and exchanging users. SHINE is based on a client-server structure and is constituted by one server-core simulator  and a client simulator for each access technology considered . All aspects related to the access technologies adopted, hence related to the data-link and physical layers, are managed by the LLSs which are the client simulators and are specific for each access technology: IEEE 802.11p and IEEE 802.15.7 in this paper.We focus on crowd sensing, which today is a popular application to provide different types of information with high capillarity in time and space. In this work, we assume that each OBU acquires from on-board sensors several basic vehicle parameters such as vehicle identification, position, speed and acceleration, that are packed into Two scenarios, with fluent and congested traffic, are considered. The road-network layout of the scenario is plotted in 1-DSRC: 1 intersection equipped with 1 DSRC RSU (no VLC RSUs);1-VLC: 1 intersection equipped with 4 traffic lights with VLC capability acting as RSU (no DSRC RSUs);1-DSRC & 1-VLC: 1 intersection equipped with 1 DSRC RSU and 4 traffic lights with VLC capability acting as RSU;23-DSRC: 23 intersections, each one equipped with one DSRC RSU (no VLC RSUs);23-VLC: 23 intersections, each one equipped with four traffic lights with VLC capability acting as RSU (92 traffic lights and no DSRC RSUs);23-VLC & 23-DSRC: 23 intersections equipped with both one DSRC RSU each and 4 traffic lights with VLC capability acting as RSU (92 traffic lights and 23 DSRC RSUs);A variable number of RSUs are placed at some intersections of the scenario. Specifically, the following cases are considered:The four traffic lights are placed on the four directions of the considered junctions of the scenario; for each considered intersection, the RSU of IEEE 802.11p is placed in the same position as the northern traffic light of the four VLC RSUs.When only one RSU is considered for a given technology, it is positioned in the most travelled intersection of the scenario.VLC only: the sole VLC is used to communicate and reach the RSUs;VLC first: VLC is used anytime it is possible in order to maximally offload the DSRC network;DSRC only: the sole DSRC is used to communicate and reach the RSUs;DSRC first: VLC first is used only in those cases where DSRC is not possible,We assume vehicles equipped with both VLC and DSRC technologies, where DSRC indicates the use of IEEE 802.11p communication protocol, whereas RSUs can be installed for one technology only or for both. To verify the impact of the use of VLC in vehicular networks, we compare the following main approaches:Comparing the considered approaches, we can verify the performance of a sole VLC vehicular network with that of VLC used to offload DSRC. In fact, if from one side, VLC suffers of short ranges and limited visibility, on the other side it offers a large unlicensed bandwidth, reduced deployment costs and high spatial reuse, which makes the full bandwidth being used in almost all links.The system performance is evaluated in terms of delivery rate PHY and MAC layers. When V2V and V2I communications are carried by means of IEEE 802.11p, we assume a received power inversely proportional to the distance raised to the power of 2.75 and buildings obstructing the communications. Such a model, also adopted in [opted in , is wellopted in  where meWhen VLC is adopted, we assume a received power inversely proportional to the distance raised to the power of four and any obstacle obstructing the communications . The angRouting. Each OBU attempts to forward its packets to the nearest RSU adopting a greedy forwarding (GF) routing algorithm [it selects the nearest RSU;it considers as possible relays the neighbors that are closer than itself to the destination;it reduces the number of available relays by considering only those within the FOV of the RSU;among the remaining relays, it considers the one nearest to the RSU and transmits its data. If no relay satisfy the conditions, the packets are stored in a local queue.lgorithm ,41. SpecTo keep a certain data freshness, if after a parametric time out (here set to 700 s) or after a maximum number of packets in queue (here set to 2000), no direct transmission (nor single hop neither multi hop) is available, the vehicle is supposed to transmit its data through the cellular network.Results show the delivery rate as a function of the packet generation rate for the two considered scenarios of fluent and congested traffic. If not specified, we assume a FOV of both vehicles and RSUs equal to DSRC first is similar to that of DSRC only in fluent traffic conditions DSRC only and DSRC first cases, demonstrating the effectiveness of VLC to increase the available resources. It can be also observed the impact of VLC data rate on DSRC first. In this case, in fact, given the higher number of vehicles, the IEEE 802.11p network is prone to collisions and the introduction of a new radio improves the performance.VLC only, independently on the data rate. In addition, the FOV of the receivers implies that vehicles must be almost in a line to communicate. This factor, added to the rules of routing algorithm, implies that an important percentage of vehicles cannot forward their messages to the RSU. On the other hand, we underline that the use of VLC has the great advantage to exploit the traffic lights that are already deployed on intersections; differently, DSRC RSUs require new installations.On the opposite, DSRC first and VLC first. What can be underlined is that, when VLC only is adopted, To verify, now, the impact on the performance of the presence of both technologies at the RSUs, in VLC only politic is adopted, also with 10 Mb/s.The impact of different number of RSUs on the delivery rate is plotted in To better understand this aspect, VLC only and DSRC first: when Hence, to also verify the impact of a larger FOV on the delivery rate, in This also suggests further investigation on the joint impact of the routing algorithms and the FOV and the angle of irradiance of vehicles and infrastructures, taking into account also realistic heights of vehicles lamps.We investigated, through realistic simulations in urban scenarios, the performance of a vehicular crowd sensing application, in terms of packets delivered to the RSUs. By considering vehicles equipped with both DSRC and VLC communication interfaces, we showed the delivery rate when the RSUs were equipped with only one of the two technologies or both. Results highlight the improvement of performance due to the use of VLC in addition to DSRC, especially in congested scenarios. The outputs also show the main limit of VLC, which is the reduced connectivity due to the high directivity of LEDs and PDs: if from the one hand, this imply lower interference, on the other hand, it provides less packets delivered to the RSUs.Future investigations will be devoted to evaluate the joint impact of different routing algorithms and lights directivity on the performance. Additional studies could also consider the use of image sensors at the receiver side instead of PDs; since a major advantage of the image sensors is the ability to spatially separate multiple sources, in such case hence parallel data transmission and multiple VLC signal reception is possible. The use of image sensors would also allow to incorporate various image processing applications in VLC function, such as distance estimation of nearby vehicles, lane keeping application, or collision avoidance .


As eye movements are mostly automatic and overtly generated to attain visual goals, individuals have a poor metacognitive knowledge of their own eye movements. We present an exploratory study on the effects of real-time continuous auditory feedback generated by eye movements. We considered both a tracking task and a production task where smooth pursuit eye movements (SPEM) can be endogenously generated. In particular, we used a visual paradigm which enables to generate and control SPEM in the absence of a moving visual target. We investigated whether real-time auditory feedback of eye movement dynamics might improve learning in both tasks, through a training protocol over 8 days. The results indicate that real-time sonification of eye movements can actually modify the oculomotor behavior, and reinforce intrinsic oculomotor perception. Nevertheless, large inter-individual differences were observed preventing us from reaching a strong conclusion on sensorimotor learning improvements. Real-time auditory feedback of body movements appears promising in several applications such as sport training and rehabilitation, for instance after a stroke, for children with developmental disorders as in writing , and are thus an intrinsic part of a complex sensory-motor loop. Moreover, the repertoire of eye movements is very rich, ranging from small fixational eye movements, to fast saccades and smooth pursuit eye movements (SPEM). The latter is known to be impossible in the absence of a moving visual, auditory or illusory target to track  to notify the production of smooth movements but stopping the sound if the subject produced saccades; this represents a knowledge of performance feedback (KP). A knowledge of result feedback (KR) is also produced (2 beeps) in case of success in a trial, in addition to a video animation and money reward. However, in the absence of any interactive aspect of the auditory feedback, motor control is driven one way, only from perception to action without closing the loop.To our knowledge, no study described the usage of real-time and continuous sonification of eye movements for SPEM learning. Sound has been mostly used as a stimulus, especially exploiting the ability of the auditory system to localize sound sources in surrounding space. Gauthier and Hofferer  and thenIn this study we use real-time continuous sonification of eye movement , which can provide gradual information, within short time scales. One initial hypothesis is that eye movement sonification could enhance proprioceptive feedback during motion and possibly provide a positive reinforcement feedback regarding the production of smooth pursuits. The lack of proprioceptive feedback during eye movement is supposed to be detrimental to free SPEM with reverse-phi, as the necessary visual percept itself is conditioned to movements initiated. For a beginner the lack of proprioceptive feedback could also lead to poor inverse models development for the sensorimotor system. To evaluate the extent to which learning to master one's eye movement would benefit from such sound feedback, we choose to couple eye movements to sound in an experimental protocol comprising two different tasks. Moreover, we attempt to train participants to control SPEM, rather than other eye movements, as this type of movement can hardly be generated in the absence of a moving target. In this way, we ensured that learning, if any, would occur without strong prior cognitive or motor control. To this aim, we employed a visual paradigm where the eye movements themselves entail the perception of visual motion in the same direction as the eye movement . Two audio examples of sonified movements were then presented to the participants, one containing mainly smooth pursuit and small catch-up saccades, and one containing many saccades. Participants were asked to indicate which of the 2 recordings was the most successful under the criteria of the experiment, which is to produce as much SPEM as possible. Two participants were withdrawn from the panel following the preliminary session, due to poor pursuit abilities and concentration issues, the final number of participants being 12 .www.sr-research.com). The eye-camera and its infrared light were positioned underneath the stimulation screen, 57 cm away the participants' eyes. The reflection on the retina and the cornea captured by the camera were analyzed by a host computer to provide the absolute gaze position . A five-point calibration was performed before each recording, or each time participants moved their head out of the chin rest. The visual stimuli (moving targets and reverse-phi stimulus) were presented on a 1,024 × 768 pixels 60 Hz screen  facing the participants, and operated by a second computer . Data recording includes horizontal and vertical gaze position and pupil diameter (500 Hz sampling frequency). The reverse-phi stimulus consisted in 500 disks  whose random positioning on the screen was renewed every 50 frames (833 ms), to avoid fixating the shapes they may form. On a single frame, all disks had the same luminance, and all disks changed their contrast polarity at 10 Hz, switching from lighter than the background to darker, and reverse. This flickering stimulus is designed to allow endogenously generating SPEM, as shown in a previous study , driven between 100 and 500 Hz, operating on a pink noise. The low end of the spectrum was cut off for clarity. From an ecological point of view, the SPEM sound coupling was designed to evoke the sound of a wind flow, or of rubbing a surface. It was built to be smooth and continuous as well as respecting the dynamic range of the motion. Saccade sounds were generated from the gaze acceleration signal. After computing the squared norm of the acceleration vector, the signal was logarithmically smoothed (20 samples window). The resulting signal commanded a monopole low-pass Filter between 400 and 1,000 Hz filtering pink noise. The envelope of the sound was then shaped with a 5 ms up and a 500 ms linear down-ramp. The average latency of sonification was evaluated around 50 ms. If the velocity of the eye exceeded 100°/s, the saccade sound was triggered, illustrating the saccade profile that was being produced. Below this threshold, no saccade sound was produced. This velocity threshold was set as the upper velocity range of smooth pursuit in humans  received the oculometric data from the EyeLink 1000 at 250 Hz and generated sound feedback in real-time through closed headphones. A custom program has been developed using the eye-tracker built-in API to transmit the eye-tracker data with OpenSoundControl (www.opensoundcontrol.org) and a UDP connection protocol to the third computer. Incoming data were processed with a patch built under the Max/MSP environment (www.cycling74.com). The sonification of the eye data was based on two processes: one for SPEM and one for saccades. The processes generated pursuit and saccade sounds from horizontal and vertical gaze speed signals. Specifically, pursuit sounds were generated from the squared norm of tangential velocity gaze vector. This signal was then filtered by a 20 samples median filter  and finished with four sessions without sound feedback. Group 2  did the opposite. Each session was composed of 4 blocks of 5 repetitions of each randomly chosen pattern for a total of 140 trials per sessions. Performing one block took approximately 12 min. A 5 min break was taken between each block. A full data set therefore contained 1,120 trials for each participant. The time course of a trial was as follows , effective displacement of the eye (>2.5°) and standard deviation of spatial distribution. Both saccades and pursuit segments smaller than 3 data points (6 ms) were excluded as they were taken for measurement artifacts. An example of segmentation obtained with the method described is shown on Figure During the tracking task, performance was assessed by the average gain of tracking pursuits. Specifically, the gain was measured on the pursuit segments validated by the detection algorithm. The number of saccades counted during a trial was also computed. During the production phase, the main evaluation criteria were the cumulative duration of SPEM produced during a trial and the number of saccades. As only few participants were able to reliably control SPEM and reproduce significantly the patterns, it was impossible to quantitatively assess the quality of reproduction of the patterns. Performance during sessions with and without sonification were compared.The data set collected in this experiment is large, as the 12 participants underwent 8 sessions, each comprising 4 blocks with 5 repetitions for each of the 7 patterns, with each trial comprising a 6 s tracking phase with the visible moving target (tracking phase), followed by 6 s of pursuit production without the target (production phase). Eye data were analyzed for each trial, separating the two tasks: number of saccades, speed of the saccades, overall duration of pursuits, mean duration of pursuit, longest duration of pursuit, and the pursuit velocity gain for the tracking task. Pupil data were not analyzed. We did not consider the production of micro-saccades and eye tremor, since most data of interest were smooth pursuit and saccade generation. The criteria for identifying saccades and pursuits were the same for all participants (see Section 2).We focused on the extent to which eye movement sonification helped participants to generate longer and smoother pursuits, as compared to trials without sound feedback. Since participants could improve their performance over time, within a session and across sessions, due to intrinsic learning effects independent of eye sonification, we also considered the evolution of performance, independent of the eye sonification. Below we describe the analysis performed on the different variables derived from the collected data set. Although we observed a slight advantage for the horizontal Lissajous pattern (data not shown), no significant effect of the type of figure was found on SPEM production. Similarly, no significant effect was found on gain or saccade production, therefore we averaged the eye data across all figures for each participant. Similarly, we averaged the data across the 5 repetitions and 4 blocks of each session, in order to obtain tractable mean data between sessions. We report below the results related to sonification and to learning on this averaged data set, separately for the tracking and production phases. We then present the results related to the subjective ratings of performance, before discussing the significance of our findings.The typical behavior exhibited by the participants in the tracking phase is a succession of smooth pursuits, of various lengths, interrupted by catch-up saccades. An example of recorded trajectory for pattern 1 (circle) is presented Figure Figure t-test , while others mostly generated saccadic eye movements. On average, the mean duration of smooth pursuit across participants was greater than 1s . This finding is in itself a remarkable result, as there is a general agreement to consider that generating SPEM in the absence of a moving target is impossible , while others only showed moderate effects, knowing that learning and sound feedback could have potentially both played a role. This is confirmed by the observation that in the SA group, some participants were also able to generate long episodes of smooth pursuit with no sound feedback.During this phase the performance of both groups is closer, only the variance of the SA group is higher. Overall no clear pattern emerges, which points to idiosyncrasies in the mastering of SPEM. Looking at the individual performance Figure  nine subThe number of saccades counted during the production phase follows a fairly similar pattern to the tracking phase. The SF group exhibits a steady number of saccades –although participant #12 stands out in sessions three and four, where he also produced many pursuits—throughout the experiment. The SA group shows a slight increase in saccades when sonification comes in (mostly driven by participant #8) with an increasing variance between participants. The presence of sonification in the second half of the experiment for the SA group seems to have an impact on the level of activity of some participants.In spite of large differences between the participants, the results of the second phase of the experiment also indicate a potential effect of sonification, positive or negative, on the ability to produce SPEM and the level of activity during the tasks (which does not appear correlated).At the end of each block, participants gave a subjective assessment of their performance during the block. Below we present the analysis on the average ratings performed with and without sound feedback, and on the correlation between ratings and smooth pursuit production. Figure Overall, the lack of correlation between ratings and pursuit performance indicates that cognitive access to eye movement is poor. However, the difference in ratings between both groups before and after the introduction of the sound feedback suggests that the feedback brought additional information that was taken into account by the participants, especially for the participants of the SA group who discovered their oculomotor behavior through the sound feedback only after the 4 initial sessions.We present here an exploratory study to use sound coupled to eye movements in order to provide a sensory feedback to an eye activity which is otherwise mostly out of reach of conscious cognition, due to poor proprioceptive and kinesthetic feedback . This is a problem given the large inter-individual variability we observe, which prevents from computing reliable between-group comparisons. A second aspect, more technical, must also be considered: the speed of the tracked target was between 16 and 21°/s depending on the patterns, and chosen to allow good tracking in the tracking phase. However, the endogenous generation of smooth pursuit on the flickering background should probably occur at a lower speed to “catch” the illusory motion that allows generating smooth pursuit regarding the stimulus parameters used in this experiment  varied with the ocular velocity. This choice was motivated to offer a “neutral,” spectrally rich and non-invasive sound, similar to the sound of writing on a hard surface and evoking velocity. Nevertheless, other choices might be more adapted to eye-sound coupling, for instance more clearly pitched or harmonic sound which variation could be easy to discriminate, could offer a better evaluation of one's eye speed. Stereo panning effects or loudness might also be used as additional variables to design the sound coupling (Parseihian et al., Sonification of eye movements is a novel approach to oculomotor control which opens a large field of investigations, both to characterize which parameters are relevant in the large space of possible couplings between eye activity and sound, and to evaluate its interest and usability in a variety of contexts: arts, education, gaming, clinical applications, etc. The results of this first study are encouraging, but also point to the need for large scale studies to better appreciate what characteristics of eye-sound coupling are most efficient, what tasks are the most appropriate to allow improved oculomotor control, and what is the time course of learning needed to use it.All authors conceived and designed the experiment together. AP and EB conducted the experiment. AP, EB, and JL analyzed data. EB, JL, and FB wrote most of the manuscript with support from AP. All authors reviewed the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Real-time dense mapping systems have been developed since the birth of consumer RGB-D cameras. Currently, there are two commonly used models in dense mapping systems: truncated signed distance function (TSDF) and surfel. The state-of-the-art dense mapping systems usually work fine with small-sized regions. The generated dense surface may be unsatisfactory around the loop closures when the system tracking drift grows large. In addition, the efficiency of the system with surfel model slows down when the number of the model points in the map becomes large. In this paper, we propose to use two maps in the dense mapping system. The RGB-D images are integrated into a local surfel map. The old surfels that reconstructed in former times and far away from the camera frustum are moved from the local map to the global map. The updated surfels in the local map when every frame arrives are kept bounded. Therefore, in our system, the scene that can be reconstructed is very large, and the frame rate of our system remains high. We detect loop closures and optimize the pose graph to distribute system tracking drift. The positions and normals of the surfels in the map are also corrected using an embedded deformation graph so that they are consistent with the updated poses. In order to deal with large surface deformations, we propose a new method for constructing constraints with system trajectories and loop closure keyframes. The proposed new method stabilizes large-scale surface deformation. Experimental results show that our novel system behaves better than the prior state-of-the-art dense mapping systems. Simultaneous Localization and Mapping (SLAM) plays an important role in the navigation system. RGB-D-based dense SLAM draws much attention since the birth of the consumer depth cameras such as Microsoft Kinect and Google Tango. Feature-based visual SLAM emphasizes the accuracy of camera pose ,2. RGB-DIt is very memory inefficient to represent the reconstructed space with voxels of equal size. Systems, for example, KinectFusion ,4, cannoCurrently, the surfel model is also used in RGB-D dense mapping systems ,13,14,15Many approaches have been proposed to improve the accuracy of camera tracking . The RGBThe approaches discussed above reduce camera tracking drift by improving camera tracking accuracy. Some other methods distribute the camera tracking drift through pose graph optimization or bundle adjustment. Bundle adjustment and pose graph optimization are commonly used to improve camera tracking accuracy in feature-based visual SLAM systems ,35. KintWhile loop closure optimization is able to distribute camera tracking drift, it is very challenging to re-integrate the generated TSDF voxel values when system revisits a place. Instead of re-integrating the produced TSDF voxel values, Kintinuous  correctsAt present, some dense mapping systems are capable of reconstructing large-scale scenes. However, we have found that when the camera tracking drift becomes large, the system may produce unsatisfactory surfaces around loop closures. Kintinuous  can recoIn summary, many dense mapping systems have been developed. However, these systems may struggle when the reconstruction region expands significantly. The dense mapping system should ideally have the following characteristics.Scalability. The system can handle large-scale dense surface reconstruction. The size of the area that can be reconstructed is unbounded if enough hard disk storage space is provided.Efficiency. The frame rate of the dense mapping system is maintained high and constant even if a large surface has been produced.Model correction. Camera tracking drift is distributed by pose graph optimization or bundle adjustment. The positions and normals of the model points are also updated.Model re-integration. The dense model can be re-integrated when camera revisits the previously scanned region.With these four items in mind, we propose a novel real-time dense mapping system. We use the surfel model in our system and maintain two maps: a local map and a global map. The new RGB-D images are integrated into the local map, and the old surfels in the local map far from the camera frustum are moved to the global map. The surfels in the local map and the global map around the loop closures are re-integrated when the camera revisits a place. There are three threads in our system. The thread of camera tracking and dense mapping is responsible for computing camera poses and fusing RGB-D frames. Loop closure thread detects loop closure and builds pose graph constraints. We optimize the pose graph if a loop closure is determined. In addition, the embedded deformation graph will also be optimized if the output error of the pose graph optimization is below the threshold. The positions and normals of the surfels in the map are corrected with the optimized parameters of the embedded deformation graph to keep them consistent with the updated poses. In order to deal with large-scale surface deformation, we propose a new method for establishing constraints using system trajectories and loop closure keyframes when optimizing the embedded deformation graph. Better 3D models of large-scale scenes are produced than the prior state-of-the-art dense mapping systems as shown in As displayed in T is the current estimation of the transformation from the current frame to the last frame. The estimated incremental value of the transformation is defined as i of the current RGB image. The camera tracking and dense mapping thread follows the pipeline of KinectFusion ,4. We usicFusion ,12, the lgorithm  to searcThe new RGB-D frame is integrated with the surfels in the active area. We find that registering and fusing the new frame with the surfels in the active area works better than those in the entire map. Because system tracking drift persists, the generated map may not be accurate enough. Registering and integrating the new frame with the entire map may paralyze the dense surface produced formerly and result in the wrong estimation of camera pose. The images of point, normal and color are generated by projecting the surfels in the active area to the current image. The surfels in the local map that need to be updated are selected by comparing the new frame with the images projected from the local map. If the values of the position and normal are close enough at the same pixel coordinates, the surfel in the local map is updated according to the equations:The old surfels in the local map far away from the camera frustum are moved to the global map. We provide more details about surfel streaming in An active region is defined as a sphere containing the current camera frustum as displayed in The memory block of the global map is equally divided as displayed in We use randomized ferns  to detecElasticFusion ,12 detecWe build a pose graph with the poses of consecutive keyframes and the pose constraints calculated with the loop closure keyframes. We optimize the pose graph with g2o  when look-nearest nodes in the Euclidean distance are selected. We optimize the embedded deformation graph if the output error of pose graph optimization is below the threshold. The positions and normals of the surfels in the maps are also updated using the optimized parameters of the embedded deformation graph. The equation for updating the surfel position is given by:In order to find out the nodes to connect with for each surfel, we first search all the nodes for the closest in time. Then the nodes nearby in time are collected, and the i are given by j on surfel i. The node close to the surfel would impose a large influence. The value is given as:k is the number of the nodes used to update the surfel. We set k to 4 in our experiments.The normal is updated according to the following formula:We find that the prior state-of-the-art dense mapping systems updating model points with embedded deformation graph may produce unsatisfactory results when the reconstruction spatial region expands significantly. Kintinous  also optThere are three terms in the cost function when optimizing the embedded deformation graph. The first term maintains the rigidity of the affine transformation matrix held in each node. The equation is given below:j. j. The third is the error term created with the constraint set. We utilize the translations of the optimized and unoptimized camera poses in the pose graph to build the constraint set. The constraint is given as:The second is a regularization term to guarantee the smoothness of the node parameters,Equation . The conT is the transformation between the current frame and the loop closure frame. The positions of corresponding features are commonly used to establish constraints for rigid and non-rigid registration and deformation [We find that the trajectory constraint is not strong enough to align the generated surface around loop closures. The generated surfaces at the same place may be misplaced and cannot be fused. We add additional constraints to align the surfaces produced around loop closures. Additional constraints are built with the sampled 3D points from the depth image that projected from the local map. The constraint is defined as:ormation ,46,47. HThe cost function is minimized by CPU. The positions and normals of the surfels in the local map and the global map are updated according to Equations  and 6)..6). The Our system is compared with the state-of-the-art dense mapping systems on the accuracy of the produced trajectories. Firstly, we use absolute trajectory error (ATE) to evaluate the accuracy of camera tracking on the TUM RGB-D benchmark . The fouAs can be seen from the table, our system achieves better results than the other four dense mapping systems. DVO SLAM calculates camera pose by combining the photometric error and depth error  of all pWe also evaluate the camera tracking accuracy of DVO SLAM , KintinuWe also evaluate the reconstruction surface accuracy of Kintinuous , Elastichttps://pan.baidu.com/s/1obi1KbpDN46HZivRni4j5A). We compare with the prior state-of-the-art dense mapping systems including BundleFusion [Most sequences in the TUM and ICL-NUIM datasets have small-sized scenes recorded. We record several sequences to evaluate the performance of large-scale region reconstruction. While recording sequences, we move the camera a long distance and close a loop at the end. When reconstructing the sequences in our datasets, dense mapping systems introduce large camera tracking drift when they revisit a loop closure area. It is very challenging to close the loop and align the surfaces created around the loop closure. Since the dataset is very important when developing new algorithms, we expose the recorded sequences to other researchers (leFusion , ElasticleFusion ,12, InfileFusion  and KintleFusion  to our oFirst, we use the four prior dense mapping systems and our system to reconstruct an office room. As displayed in BundleFusion usually works well when the features in the scene are rich, for instance, the office room in The conclusion can also be proved by the reconstruction results of the third-floor sequence. When recording the third-floor sequence, we also move the camera around and close a loop at the end. BundleFusion and InfiniTAM v3 failed in this sequence as displayed in We also compare the system efficiency. Our system is compared to ElasticFusion because both systems use surfel model. We recorded the average execution time per 100 frames when the system processed the stairs sequence. The execution time is displayed in We propose a novel large-scale dense mapping system with surfels. Old surfels far from the camera frustum are streamed from the local map to the global map. The space that we can reconstruct is theoretically infinite, and the frame rate of our dense mapping system remains high and constant even if a large surface has been produced. We optimize a pose graph when loop closure is detected and correct the model surfels to keep them consistent with the updated poses. In order to deal with large surface deformation, we propose a new method for constructing constraints using system trajectories and loop closure keyframes. Experimental results show that the dense surfaces produced by our system are superior to those produced by the prior state-of-the-art systems, especially with large-scale reconstruction.There are limitations and opportunities for future work. ElasticFusion uses a map-centric approach which works well with small-sized region reconstruction. However, the systems utilizing pose graph have more advantages in sensor fusion. IMU SLAM systems have demonstrated very robust camera tracking performance ,51,52. S


This paper focuses on the design, realization, and verification of a novel phonocardiographic- based fiber-optic sensor and adaptive signal processing system for noninvasive continuous fetal heart rate (fHR) monitoring. Our proposed system utilizes two Mach-Zehnder interferometeric sensors. Based on the analysis of real measurement data, we developed a simplified dynamic model for the generation and distribution of heart sounds throughout the human body. Building on this signal model, we then designed, implemented, and verified our adaptive signal processing system by implementing two stochastic gradient-based algorithms: the Least Mean Square Algorithm (LMS), and the Normalized Least Mean Square (NLMS) Algorithm. With this system we were able to extract the fHR information from high quality fetal phonocardiograms (fPCGs), filtered from abdominal maternal phonocardiograms (mPCGs) by performing fPCG signal peak detection. Common signal processing methods such as linear filtering, signal subtraction, and others could not be used for this purpose as fPCG and mPCG signals share overlapping frequency spectra. The performance of the adaptive system was evaluated by using both qualitative  and quantitative measures such as: Signal-to-Noise Ratio—SNR, Root Mean Square Error—RMSE, Sensitivity—S+, and Positive Predictive Value—PPV. The monitoring of fetal heart sound by general practitioners dates back to the second half of the 17th century when these signs were considered essential to distinguish between live and dead fetuses. To listen to these heart sounds, the so-called Pinard’s stethoscope or its modern versions have been in use for several decades ,3. With Currently, the most frequently used fetal monitoring methods by clinicians are mainly centered on the ultrasound modality. For example, fetal echocardiography (fECHO) is used for diagnosis of congenital heart defects (20th–23rd week of pregnancy), and cardiotocography (CTG) simultaneously measures fetal heartbeat and maternal uterine contractions ,8,9,10. One of the most important parameters monitored by EFM is the fetal heart rate (fHR). The fHR data can be acquired either invasively , or non-invasively. The invasive method produces a better quality signal , due to the fact that the heart rate is measured directly from the fetus’s head. The invasive method provides a very important tool in improving the diagnostic quality of conventional CTG as well as the validation of noninvasive ECG methods. Unfortunately, the invasive approach has a number of disadvantages. Obviously, the invasive nature of this method does not allow its utilization before delivery (before the rupture of membranes and the release of amniotic fluid). Also during delivery, it increases the possibility of infections and endangers the pregnant woman as well as her fetus. We can state with a fair degree of confidence that the current preferred trend in medicine is to use noninvasive methods ,13,14. EConventional phonocardiography (PCG) is based on a noninvasive scanning of the acoustic signals  by means of a microphone placed on the subject’s thorax. Therefore, the maternal PCG (mPCG) signals are acquired by placing a microphone on the maternal chest and the fetal PCGs (fPCGs) are recorded by positioning a microphone on the maternal abdomen. Here we describe, for the first time, an innovative method for extraction of fPCG signals from mPCG signals using a fiber optic sensor (based on two interferometric components) and an adaptive filtering system that implements two gradient-based algorithms (LMS and NMLS) for continuous monitoring of fHR. As such, we present a completely new and yet unused concept and technique in continuous fetal heart rate monitoring.Some of the main advantages of such a system are due to the desirable characteristics of optical fibers and fiber-optic interferometers. Fiber-optic measurement sensors are resistant to technical artifacts such as electromagnetic interferences; therefore, they can be used in situations where it is impossible to deploy conventional EFM methods, such as during Magnetic Resonance Imaging (MRI) examination ,21 or inBecause the conventionally used CTG Method exposes the fetus to ultrasonic radiation, it could be considered potentially harmful and invasive and therefore the safety of the fetus may become compromised in long-term monitoring ,31. In cFiber-optic technologies, or more precisely fiber-optic sensors (FOS), are gaining increased use for biomedical purposes in a number of fast developing applications ,54,55,56The specific aim of this paper is to describe the utility of a fiber-optic interferometric sensor and an adaptive system implementing the LMS and NLMS Algorithms for the effective extraction of fHR information from maternal phonocardiographic recordings by performing fPCG signal peak detection. Adaptive systems are able to change their parameters based on actual information in the processed signal. We should also emphasis at this point that there are other powerful methods such as the Independent Component Analysis (ICA) ,62,63, BBased on the analysis of real measurement data acquired by our sensor and adaptive signal processing system, we developed a simplified dynamic signal model for the generation and distribution of heart sounds throughout the human body. In developing our signal model we were guided by the contributions made by ALMASI et al. ,76, who Our aim here is not to fully describe the details of the models for the generation and distribution of cardiac electrical and acoustic signals in the pregnant woman’s body , but to evaluate the performance of our adaptive system on synthetic data, which morphologically conform to the data acquired by means of interferometric sensors used in our research. Our dynamic signal generation model, which is based on the works reported in ,83,84,85The role and importance of such a dynamic signal model capable of generating realistic synthetic data at this stage of our research and development cannot be taken lightly, as it is essential to carry out quantitative evaluations of the experimental results and validate the soundness of the overall system. Currently, there are no databases (Gold Standards) available for fHR signals acquired from interferometric sensors, which were used in our experiments and in testing our methods for effective fHR extraction. In addition, in the initial phases of developing any technology and its related clinical applications, performing tests on pregnant women are not considered as a suitable option. It is imperative that new technology developments and their applications to human subjects undergo comprehensive approved clinical trials and meticulous legislative processes first. Such requirements are notably strict and rightly so for pregnant women because their fetus is extremely sensitive to external energies and stimuli . Considering the above factors, it is evident that the utilization of synthetic data is absolutely essential in the initial phases of research and development.Fetal Phonocardiography was discovered by Kergardec, Marsac and Kennedy during the 17th century ,87. The Generally speaking, electrocardiograms (ECGs) and phonocardiograms (PCGs) belong to a group of basic diagnostic and monitoring measurements used to describe the cardiac electrical and mechanical activities, respectively . This clPhonocardiograms also include other heart sounds. The third heart sound  ,93.In phonocardiography, different characteristics such as rate, frequency, and duration or changes in individual parts of the recorded cardiac acoustic signal are measured. Thanks to PCG and ECG signals clinicians are able to diagnose a variety of heart diseases in adults  and in fn is the coupler’s output index with a value of 1, 2 or 3, Our noninvasive adaptive system utilizes two interferometric sensors (described in e (PDMS) ,96 constribed in .The measurements probe was encapsulated in PDMS with the designation of Sylgard 184, which is a two-component casting compound: the A component creates its own pre-polymer and the B component is a curing agent. Both components are mixed together with a weight ratio of 10:1 (A:B) according to the manufacturer’s datasheets. Bubbles and microbubbles that result from the combination of the pre-polymer and the curing agent can be removed using an ultrasonic bath. Homogeneity of the connection is achieved by using a laboratory shaker. The measurement probe contains two connectors of FC/APC type.The basic scheme for the noninvasive PPG-based fiber-optic adaptive system for fHR monitoring by means of two interferometric measurement sensors placed on the chest and abdomen is shown in Direct Signal Subtraction: The signal detected on the abdomen called as aPCG signal is the sum of fPCG and mPCG signals. The easiest method for removing the undesirable mPCG from the aPCG signal is a direct subtraction of the values Linear Filtering: The next method is to use linear filtering, i.e., frequency selective filtering. However, like direct signal subtraction, linear filtering cannot effectively eliminate the undesirable mPCG signal as the desired signal (fPCG) and the unwanted signal (mPCG) share overlapping spectra  are realized in the so-called pre-processing block. Pre-processing is realized by means of the well-known techniques, thoroughly described in technical literature, such as ,99,100.The amplification and digitization of the maternal and fetal signals are realized by means of the NI-USB 6210 card, which has a 16-bit Analog-to-Digital Convertor (ADC) with a built-in amplifier enabling the choice of amplification gains in steps of 1, 5, 10 and 50. The associated software support development environment of this card, LabVIEW, enables further digital processing of the data. In this application, the scanning and symmetric demodulation of data is implemented in the form of immediate variance of phase information and digital filtering.It is assumed that the o 400 Hz ,100,101.In our experiments, two inputs signals were considered (since we used two interferometric sensors). The first input (reference) signal that was fed into a pre-processing block is the maternal thoracic PCG signal represented by The second (primary) input to the experimental system shown in By using linear filtering we were able to eliminate only those signal components whose spectra did not overlap with the useful fPCG signal .To carry out our experiments, we used an adaptive system based on the root mean square error (RMSE) criterion. Stochastic approaches require a large number of measurements to enable us to perform adequate statistics. This approach leads to the Least Mean Square Algorithm or its normalized version , which are the basic representatives of stochastic gradient-based adaptation methods ,103,104.N-th order FIR filter with transversal structure and implemented by using the LMS Algorithm. Each iteration of the LMS Algorithm requires the implementation of 3 different steps, respectively. First of all, a value of the output of the FIR filter Equation , valid iSubsequently, a value of the estimated error signal Equation  in R, thAnd finally, the values of vector scales Equation , valid iThe implementation of the LMS Algorithm in The Normalized LMS Algorithm (NLMS) is a variant of the LMS Algorithm with the added advantage of an accelerated convergence speed and a reasonable computational cost . In this process .If we consider the step size in Equation  to be a ut x(n):w→(n)=w→. Considering these practical facts, the necessity to use synthetic data in the early stages of our developmental work becomes indispensable.The implemented fPCG and mPCG signal models used in our research are based on signals that were produced by means of our ECG signal generator and modelling of sound distribution in the human body. The models for fPCG and mPCG signals are inspired by real data acquired from interferometric sensors. The models predict that heartbeat activity causes (triggers) subsequent mechanical effects, which are distributed throughout the human body, and then reflected and attenuated after a certain period of time ,109,110.The advantage of this conceptualization of modelling the fetal heartbeats is that it allows us to superimpose the fPCG signal on an abdominal signal, i.e., The implemented model capturing the maternal and fetal heartbeat activities shows that the fetal heart activity is decreased while dispersing through the body and is not seen in the thoracic part. This is a necessary condition for being able to use this signal for adaptive filtering. Our signal generator enables us to model the variability of respiration and heart rate activity as well ,114,115.N = 38 and the convergence constant N = 61 with an initial The parameters of both adaptive algorithms were set empirically based on experiments. As for the LMS Algorithm, we set the filter order Note the peak at the beginning of the estimated signal in Objective methods for the evaluation of filtering quality require a reference fPCG signal from which we can determine the fHR. Currently, there are no references available to represent the outputs of interferometric sensors, and thus this seems to be quite a suitable way to objectively verify the utilization of the proposed model. In order to compare the differences between the reference and predicted fHRs, we made use of the Bland-Altman plots . The difFurthermore, the Bland-Altman statistics show that the error between the reference and the predicted fetal heart rate is twice as high when using the LMS Algorithm. Possible explanations for this are described in the text above.The value of quation:SNRinp=10The next filtering performance metric is the output Equation . Based oEquation , it is nBy using the values of  filters . The pre filters .(11)RMSTo evaluate the peak detection success rate of our algorithms we can use two parameters. The first parameter is sensitivity, further labelled as S+, which shows what percentage of all peaks included in the examined signal were detected. The Sensitivity parameter S+ is defined by Equation .(12)S+[The second parameter used for specifying the successful detection rate of peaks is the Positive Predictive Value—PPV. It is the probability of the real occurrence of a peak (significant point) during its positive detection by a detector. The PPV is expressed by Equation .(13)PPVIn Equation , TP is tThe results  show tha or fS1s a and con or fS1s b, howeveFetal PCG monitoring is based on a passive acoustic technique (our fiber-optic interferometric sensors), where no energy is transmitted to the human body ,120,121.We had to use synthetic data in our research as noninvasive and nonintrusive sensors (like those used in our system) are not currently used in clinical practice: thus it was completely unrealistic to think about carrying out experimental tests using real maternal and fetal data to noninvasively detect fetal hypoxic conditions during different stages of pregnancy. For verification purposes, we used models that generate maternal and fetal signals with properties similar to real data acquired in clinical practice. These data also provided us with realistic reference signals for the quantitative evaluation of the realized systems, including the modelling of fetal hypoxic states without causing any risk or danger to the mother and her fetus.To the best of our knowledge, this is pioneering work and the first study of its kind to explore the frontiers of noninvasive continuous fetal heart rate monitoring by means of fiber-optic interferometric sensors. We present a number of original and yet unpublished results. The follow-up research will be focused on the extension of our experiments and tests in clinical practice; at that time, measurements will be carried out on pregnant women. Fiber-optic sensors are resistant to technical artifacts such as electromagnetic interferences (EMI), and they can be used in Magnetic Resonance Imaging (MRI) environments or under wet conditions. The functionality of our sensor and adaptive filtering system was assessed by both qualitative (assessments made by gynecologists) and quantitative signal processing performance measures such as: SNR, RMSE, sensitivity S+, and positive predicative value PPV.The basic drawback of fiber-optic interferometry is the need for a reference channel. The basis of interferometry is the measurement of differences between sensor and reference channels. The reference channel of the interferometer must be designed so that it allows to eliminate most of the unwanted signals. Thermal phenomena, mechanical changes, and other factors generate a background signal (noise). To address this issue or more precisely the need for the reference channel (position and encapsulation) is critical according to sensor sensitivity which must be taken into account in the design of a fiber-optic interferometer ,133,134.Even though the diagnostic potential of the fPCG signal is high, its processing and analysis are rather challenging. Fetal heart sound (fHS) signals could be relatively weak and have a low SNR due to the following factors: the uncertainty associated with the presence of the G signal , the proG signal , the sprG signal . As a coG signal ,128.The next stage of research activity within the framework of this paper is to carry out animal studies  monitoring. The probe utilizes sensing elements operating on the basis of the Mach-Zehnder Interferometer, which offers great potential for applications in modern noninvasive medicine. The performance of our system was evaluated by using generated (synthetic) data comparable to real data with the added advantage of providing the reference signal necessary for the performance evaluation of the adaptive filtering system as well as the capability to model hypoxic states without causing any risk or danger to the fetus and the mother. The system’s performance was evaluated by using both qualitative  and quantitative measures such as SNR, RMSE, S+, and PPV.Our experimental results demonstrated that the NLMS Algorithm produced better outcomes in the determination of fHR information based on S+, PPV measures, whereas the LMS Algorithm performed better when SNR and RMSE parameters where used to evaluate the system’s performance. Our findings further confirmed that signal filtering efficiency was not influenced by the form of fPCG signals  when evaluated based on SNR, RMSE, S+, and PPV parameters.In summary, our paper presents the first attempt of its kind towards developing the techniques and methods for noninvasive continuous fHR monitoring by using interferometric fiber-optic sensors. The area of research dealing with fetal PCGs measured by these sensors is an uncharted territory that warrants further extensive clinical investigations to be carried out in the future.Our realized noninvasive adaptive system using fiber-optic interferometric sensors can significantly contribute to the future research in fHR monitoring and can even be considered for application in the Magnetic Resonance Imaging (MRI) and wet environments. We are very excited that the clinicians at the University Hospital, Brno, in the Czech Republic, intend to use the system reported here for their short term clinical studies in the near future.