Continuous monitoring of vital signs, such as respiration and heartbeat, plays a crucial role in early detection and even prediction of conditions that may affect the wellbeing of the patient. Sensing vital signs can be categorized into: contact-based techniques and contactless based techniques. Conventional clinical methods of detecting these vital signs require the use of contact sensors, which may not be practical for long duration monitoring and less convenient for repeatable measurements. On the other hand, wireless vital signs detection using radars has the distinct advantage of not requiring the attachment of electrodes to the subject’s body and hence not constraining the movement of the person and eliminating the possibility of skin irritation. In addition, it removes the need for wires and limitation of access to patients, especially for children and the elderly. This paper presents a thorough review on the traditional methods of monitoring cardio-pulmonary rates as well as the potential of replacing these systems with radar-based techniques. The paper also highlights the challenges that radar-based vital signs monitoring methods need to overcome to gain acceptance in the healthcare field. A proof-of-concept of a radar-based vital sign detection system is presented together with promising measurement results. The four major vital signs are body temperature (BT), heart rate (HR), breath rate (BR) and blood pressure (BP). They provide almost a complete picture of individuals’ body vital functions and help to assess their general physical health. The assessment of body temperature is relatively simple, low cost and does not often require continuous monitoring. On the other hand, detection and monitoring of breath rate and heart rate usually require complex systems involving sensors and computers that are physically connected. Breath rate or breath frequency determines the number of respiration cycles performed by an individual in one minute while the heart rate or pulse rate corresponds to the number of heart beats per minute. The standard breath rate and pulse rate of a healthy adult individual ranges between 12–20 breaths per minute and 60–100 beats per minute, respectively [Monitoring of human cardio-pulmonary rates exhibits a variety of applications from medical diagnostics to fitness assessment and emotion recognition. Heart and respiration rates monitoring can be used to predict certain pulmonary and cardiovascular diseases, which account for more than 31% of all deaths worldwide . In 2016In this paper, we review different methods used in contact-based  and contThis section reviews some of the major contact-based methods for detecting human respiratory and cardiac rates. The review highlights the fundamental principles of the various methods as well as some of their shortcomings. The contact-based methods addressed in this section include electrocardiography, photoplethysmography, airflow sensing, chest-wall mechanical displacement sensing, blood pressure sensing and sound-based sensing. We will explain each type in the next sectionsElectrocardiography is a process of acquiring the electrical activity of the heart resulting from the action potential generated by heartbeats. It is based on measuring the potential difference between at least two points on the body surface of the subject . The sigIn the 12-lead ECG system, 10 electrodes are attached to the surface of limbs and chest of the subject, generating 12 groups of signals  Figure a. ElectrOn the other hand, ambulatory ECG systems have typically three leads or fewer and are smaller in size, as shown in A typical ECG waveform is represented in Traditional ECG systems use hydrogel between the contact electrodes and the skin  to increPhotoplethysmography is a technique that uses optical means to instantaneously measure the changes in volume in human tissue . The timMoreover, the respiration rate can be estimated from the PPG data since breathing causes an amplitude and frequency modulation of the received signal ,41, evenDespite being cost-effective, PPG systems most often produce signals that are affected by several factors including the measuring site, ambient temperature and the posture of the subject ,37,43. IMany recent works on PPG remote monitoring are based on the use of a camera, which can be from a cellphone or a laptop ,44,45,46Breath rate generated from ECG and PPG is obtained from the breath modulation in amplitude and frequency of the heart signal and therefore is approximate. A more accurate method of acquiring BR can be done by sensing the breath air components, temperature and humidity. Next, we will discuss vital signs detection using breath air-sensing methods.2), its humidity and temperature.Human breath rate can be measured using characteristics of the inhaled and exhaled air such as its carbon dioxide . This difference can be as high as 15 °C . The airThermistors are resistors with resistance non-linearly dependent on temperature . ThermocSimilar to the capnography method previously discussed, breath rate can be measured by acquiring the water vapor level of the inhaled and exhaled air. The related humidity of the inhaled air differs from that of the exhaled one by approximately 20% to 60% . This diLike the aforementioned air-based vital sign detection and monitoring systems, the breath rate detection systems employing humidity sensors require the subject to wear a face mask or a tube around the nostril to avoid the corruption of the respiration signal by MAs. As we will see in the next section, vital signs measurement using chest displacement sensing can be non-invasive while providing good accuracy.Human breath and heart rates can be measured by sensing the mechanical or physical activities of the heart and lungs at the surface of the body.During normal human respiration, the diaphragm contracts and expends along with the intercostal muscles, allowing air to enter and exit the lungs. These physical activities of the diaphragm and respiratory muscles result in displacement of the chest that can expand up to 7.37 cm circumferentially . DiffereThe strain-sensing method employs resistive, capacitive, inductive and fiber-optic sensors to record the instantaneous change in strain. Piezo-resistive strain sensors employ elements that change their shape when subjected to a physical displacement. This change in shape results into a change in their resistance values which are measured using an electronic circuit a. The stMoreover, the impedance pneumography consists of measuring the impedance of the thorax, which is related to the volume of the lungs. Finally, respiratory rate can be detected by measuring the acceleration, the angular velocity and the magnetic field strength of the abdomen during respiration. An accelerometer can be used to convert the mechanical movement of the abdomen into an electrical signal. The movement of the thorax is related to the inertial response of the accelerometer. By attaching this electromechanical device to the upper thorax, it is possible to record the human breathing pattern ,81. The Heart rate can be monitored by sensing blood pressure. The rhythmic pulsation of the heart causes pressure of blood on the walls of blood vessels. A sensor can be appropriately placed on the human body  to capture the instantaneous pressure of blood resulting from heartbeat . The bloDuring the opening and closing of the valves, the heart produces sounds that can be detected by a microphone. The production of this acoustic effect results from several actions including contraction and relaxation of the heart muscle, rising and falling pressures of cardiac cavities, opening and closing of the valves and blood circulating and stopping. The record of this cardiac acoustic phenomena may help cardiac events to be visualized . The sysUnlike the aforementioned contact-based vital signs monitoring systems, radars do not require any contact probe to be attached to the body of the human being to acquire respiratory and cardiac rates with high accuracy. Vital signs detection radars rely on the modulation effect of a radio signal sent by a transceiver towards the patient. This modulation is due to the chest-wall displacement of the patient, which contains both the respiratory and heart signal along with environmental and electronic noises. The noise is removed during signal processing to provide clean vital signs of the subject. Note that the vital signs radars do not have severe power emission requirements for short distances application (up to a couple of meters). Typically, the power transmitted by the radar does not exceed 12 dBm for a two-meters distance application, which is less than the average power emitted by a smartphone. Therefore, these radars are safe. Depending on the type of signal it transmits, the radar can be of type continuous-wave (CW), pulsed, frequency-modulated continuous-wave (FMCW) or stepped-frequency continuous-wave (SFCW).The CW radars are the most common type due to their simplicity. A typical radar has a transceiver unit connected to the transmitting and receiving antennas and a digital signal-processing unit, as seen in t and Ar are the amplitudes of the transmitted and received signals, respectively; ω the angular frequency of the transmitted signal; λ the carrier wavelength; c the speed of the light; d0 the constant distance between the antennas and the subject; x(t) the instantaneous displacement of the chest, given by (3):b, Ah, φb and φh are the amplitudes and the phase shifts of the chest displacement due to breathing and heartbeat, respectively. As seen from (1) and (2), the signal sent by the radar is modulated in frequency and phase due to the displacement of the chest. This modulation is called the Doppler effect.The time-domain transmitted signal, denoted by T(t) and the received signal by R(t) are:The transceiver can be of type heterodyne, where the intermediate frequency (IF) is non-zero or type zero-IF (direct-conversion), where the IF is zero. As shown in I (t) is obtained by mixing a replica of the transmitted signal in (1) with the received signal in (2):The baseband signals are obtained after demodulation by the quadrature mixer then subsequently filtered, further amplified by the variable-gain amplifier (VGA) and digitized by the ADC before being processed by a computer. For a zero-IF radar, the in-phase baseband signal BSimilarly, the quadrature baseband signal is obtained by mixing the received signal with a replica of the transmitted signal shifted by a phase of 90°:I and Q baseband signals are used in signal processing to determine the vital signs.The above Various analysis methods can be used with the data of CW in order to accurately extract the vital signs. These methods can be classified as time-frequency analysis, numerical analysis, classification and training and algorithms based on mathematical and experimental modelling .I and Q baseband signals [I and Q signals into a complex form using Bessel’s function and taking the complex Fourier transform of the result [The time and frequency domain analysis have been extensively used in the literature to obtain HR and BR from CW Doppler radars. Peak detection methods are mainly used as a time-domain method to detect the peaks of respiration and heart signals. These include the autocorrelation output with fast Fourier transform (FFT)  and meth signals . On the e result   of the subject, separation of heartbeat signal from respiration signal, multiple targets detection etc. The null-point detection and phase-noise issues can be adequately dealt with by using a proper CW transceiver architecture (low-IF single SSB) and signal processing (CSD) . HoweverDuring the acquisition of the vital signs, the subject may often move body parts like hands and legs or his entire body. These unwanted body movements are called random body movements. The signals reflected due to RBMs are stronger than the vital signals, therefore corrupting the latter. Hence, it is necessary to incorporate an RBM cancellation mechanism in the radar. Several methods of RBM mitigation were proposed in the literature. One method of cancelling the RBM is the use of two identical CW transceivers and the CSD . AnotherFurthermore, the ability of the radar to detect a precise and accurate heart signal is challenging. The frequency of the human heartbeat (1–3 Hz) lies close to that of the respiration (0.1–0.9 Hz). Since the heartbeat signal is much smaller in amplitude compared to the respiration signal, it can easily be corrupted by the harmonics of the latter. Therefore, it is often required to take adequate measures for recovering the heartbeat signals. In , a signaMoreover, for applications such as surveillance and human presence detection, the Doppler radar must be able to detect the vital signs of multiple human subjects at the same time. Simple CW radars cannot fulfil this functionality since single tone CW signals cannot provide range information . TherefoIn FMCW radar systems, the frequency of the output signal is varied linearly with respect to time. Represented in The FMCW radars are capable of providing both the range and Doppler information. Therefore, the vital signs of multiple subjects located in the line-of-sight of the antennas can be detected. The basic algorithm of used in FMCW radar data analysis involve localization of the subject and detection of HR and BR associated with each range as seen in Many recent research on non-contact vital signs sensing are based on FMCW radar techniques as they overcome the shortcoming of the difficulty or inability in CW radars to provide the range information. A portable short-range FMCW radar was developed in  for locaThe use of FMCW radars for vital signs detection does not eliminate issues such as RBM effect and separation of the respiratory signal from the cardiac signal as previously discussed. Another important issue arising in the operation of FMCW radars is the incoherence of the radar. Incoherence occurs when the radar is unable to detect the micro-doppler signature which is contained in the phase data . TherefoThe operation of SFCW radars is similar to that of FMCW ones. In this case, a series of N frames are linearly transmitted towards the target with a time interval of Δf between each frame, as seen in The SFCW radars are not common in human vital signs detection applications compared to the CW and FMCW radars. Nevertheless, they detain similar algorithms and signal processing techniques to their FMCW counterparts as they allow localization and multi-subjects detection. Meanwhile, most article on the implementation of SFCW radars for human HR and BR detection use frequency domain processing, peak detection from FFT being the dominant method. The baseband data obtained from the radar is put into a time-frequency matrix of J × N size, where J and N indicate the total number of frequency sweeps and the number of radio frequency (RF) steps, respectively . Next a SFCW radars have been actively investigated for biomedical applications. Due to their simplicity compared to the FMCW radar, this type of radar is becoming quite attractive for contactless cardio-respiratory signal detection and monitoring in recent years. In , experimSimilar to the CW and FMCW radars, the SFCW radars suffer from the motion artifact of the subject as the major bottleneck. Additionally, the presence of respiratory harmonics on the vital signs undermine the accuracy of the heart rate.Even though the radars discussed in the previous sections have high sensitivity, they suffer from low power efficiency and are not preferred for the detection of humans behind concretes. On the other hand, pulsed radars have the advantage of higher power efficiency and can be used to detect the vital signs of humans located behind non-conductive materials such as concrete walls and tables. This type of radar will be discussed in the following section.In pulse-based sensing radars, a short time  domain modulated or un-modulated pulse is sent by the radar’s transmitter towards the patient. The reflected echo is captured by the receiver and processed in time domain to obtain the cardio-respiratory signals of the subject. The most common pulse-based radar for vital signs sensing is the impulse radio ultra-wideband radar (IR-UWB). The typical architecture of IR-UWB radars is depicted in Various algorithms are used to retrieve the cardio-respiratory rates of human subjects using UWB radars depending on the specific applications. A simple UWB employing body movement cancellation was designed in . It usesThe UWB radars are extensively used in many biomedical applications. Due to the ability of the UWB pulse to penetrate concrete, a plethora of research has been conducted into detection of humans behind walls and other concrete objects. This feature gives the advantage of UWB radar for applications such as search and rescue surveillance and security. In , a UWB rThe IR-UWB radars have been proven to have great feasibility in search and rescue and security applications due to their low power consumption and excellent ability to penetrate non-conductive material. However, a number of limitations impede their popularity over the Doppler radars. One of the fundamental issues with the IR-UWB radars is their power density limitation. For safety and interference with other devices purposes, power density masks are provided by telecommunication regulators such as the Federal Communication Commission (FCC) and the European Electronic Communication Commission (ECC) . These mAs stated in the previous sections, RBM remains the main challenge in human vital signs detection using radar techniques. Several RBM mitigation techniques are studied in the literature.One method of cancelling the RBM is the use of two identical CW transceivers and the CSD . The twoAnother method used to cancel the RBM effect in vital signs acquisition is the radar implementation with self-injection locking . As showFurthermore, signal processing techniques can be used to remove unwanted body movement in Doppler radars for vital signs detection. In , the empThe study on vital radars in this paper was mainly focused on the detection of HR and BR and, therefore, does not address the beat-to-beat cardiac activities in details. Heart rate variability (HRV) provides the time variation in beat-to-beat interval and is particularly important for correlating the cardiovascular regulatory system response to stress, illness and demands . In addiIn , the autThe effect of frequency on detection accuracy of human vital signs using Doppler radar has not been intensively studied in the literature. However, several factors will need to be taken into account to fully understand how the operating frequency of the radar impact vital signs acquisition. First, the dielectric properties of the human body tissue must be studied. The complex dielectric properties of human tissue was modelled by Gabriel et al. in  using thNext, the results from an experiment setup using a CW radar and two human subjects will be presented and discussed. Future works needed for the application of vital signs radars in the consumer market will be emphasized. Next, we will evaluate the performance of a CW radar in respiration and heart rates acquisition using two human volunteers.A proof-of-concept experimental setup has been constructed using a vector network analyzer (VNA) and two horn antennas operating from 8 GHz to 15 GHz. As seen in The aim of the signal processing is to obtain the BR and HR of the subject without employing much complex algorithms. The summary of the signal analysis algorithm employed in this experiment is depicted in The experimental results are given in As seen from the above experiment, CW radars can be used to detect human BR and HR without employing complex signal processing.Since human cardio-respiratory rate monitoring is crucial for prediction and diagnosis of cardiovascular and pulmonary diseases, this paper presents methods used for detecting and monitoring these vital signs. The most popular modalities are contact-based with ECG being the gold-standard technique. These methods involve the use of body sensors and cabling, which may not be practical for long-term monitoring. As a result, many recent research activities were focused on contactless-based methods using radar techniques, which do not require the use of electrodes and offer good vital sign detection accuracy. The challenges faced by vital signs radars were discussed along with their state-of-the-art solutions. A proof-of-concept experiment was carried with two different volunteers to demonstrate the potential of Doppler radars in vital signs detection. Even though radars show promising results in detecting human cardio-respiratory rates, the issues of RBM and separation of HR from BR remain the bottleneck of the widespread application of this type of system. The state-of-the-art solutions for motion artifacts usually deal with one-dimensional motions and often exhibit high power consumption. Furthermore, accurate heart rate detection usually requires advanced signal-processing methods. Therefore, future works on vital sign radars are needed for their proliferation in the consumer market. These include more accurate mechanisms for mitigating the 3D motion artifacts of the SUT and less computational loads for vital signs acquisition while keeping the power consumption low.


In continuous-variable quantum key distribution (CVQKD) systems, high-quality data synchronization between two legitimate parties, Alice and Bob, is the premise of the generation of shared secret keys. Synchronization with specially designed frames is an efficient way, but it requires special modulating devices to generate these special frames. Moreover, the extra requirement of special modulating devices makes it technically impossible for some passive preparation schemes. We propose a novel approach to realize synchronization in this paper, which is different from those special-frame-based methods. In our proposed scheme, Alice publishes parts of the original signals as the synchronization frames and Bob takes these frames to perform the synchronization algorithm. Besides, a synchronization feature is applied to deal with phase shifts. The simulation results based on practical data demonstrate that the proposed synchronization scheme not only maintains a high success rate but simplifies the data processing flow at the same time, which dramatically reduces the computational complexity. Quantum key distribution (QKD) has become a popular topic for its confidentiality, which allows two legitimate parties far away to share secure secret keys through an untrusted channel with unconditional security ,2,3. GenX or P measurement bases. Afterward, he compares them with Alice’s bases and selects the variables with the same bases. After reconciliation and privacy amplification processes, Alice and Bob will share the same key data.In a typical CVQKD system, Alice first prepares quantum states. Then secret information, which is produced from the true random number generator, is encoded on the position or momentum quadrature of quantum states by amplitude and phase modulations. After that, the modulated quantum states, which can be expressed as It is worth noting that synchronization in CVQKD plays an important role. Simply speaking, if the data of Alice and Bob are not aligned, decoding key information in Bob’s side will be independent with the one Alice prepared, which results in inconsistent secret key strings after the reconciliation process, and thus deteriorates the overall performance. In a CVQKD system, clock synchronization makes two communication entities share the same clock to acquire accurate data. So far, clock synchronization schemes include the transmitted local oscillator (TLO)  and locaBesides, in practical CVQKD applications, the quantum state will suffer unpredictable nonlinear effects, and the quadrature components of the optical field of quantum states will suffer phase shifts during signal transmissions ,20, whicTo simplify the frame synchronization scheme and improve the efficiency and robustness of the CVQKD system, we propose a novel scheme here. In particular, a new feature is designed, which can tolerate phase shifts and synchronize in a strong noise environment. Each synchronization process requires only a few addition and subtraction operations and a Hamming distance comparison. In particular, we analyze the performance of this method under different phase shifts and various SNR settings. The results show that this scheme can tolerate different phase shifts and performs well at a low SNR. Moreover, the proposed scheme also keeps a good balance between performance and computational complexity.The rest of the paper is organized as follows: In In the common frame synchronization scheme of CVQKD ,16, the These issues prompted us to improve the training-frame-based scheme into a modulation-free one without specified synchronization frames. In addition, phase drifts between the LO and signal will introduce extra trouble into the synchronization process. A practical scheme should overcome the phase drifts to successfully implement synchronization. In classical optical and wireless communications, synchronization can be performed by measuring the Hamming distance between the outputs of the transmitter and the received signals . The HamT is the transmittance of the quantum channel and In this part, we will mainly analyze the influence of the phase shift on the synchronization process, then introduce a robust feature. Alice sends quantum states eference , XA and x. The sign of the above conditional expectation is,To cope with the phase shifts, here we introduce a new operator So when In the above discussion, we do not consider one case that P into consideration is another good way. In the following section, we will show how the above conclusion can be applied to real synchronization.From the above expression, we can see that if The conditional expectation of the incremental label and its sign give us some ideas that the sign of X, which can be expressed as YBased on the above analysis, the incremental label can be constructed. This labeling method will transform a random number sequence L numbers of the current position, such as shown in L numbers, the output is used as a descriptor. We call the Step 1. Sum the next Step 2. If N bits of conversion sequence Y are seen as a feature, and we can calculate the Hamming distance of the two signal sequences to measure their similarity.Step 3. After all the received signals are marked, the synchronization process begins. Every successive It should be mentioned that noise with zero expectation will be suppressed and their impact on synchronization is weakened. This transformation method is simple and efficient, and we will show its performance in the next section and analyze computation complexity in the computational analysis section.To compete with phase drifts, the sender Alice can prepare four transformation sequences of her synchronization frames. Firstly, Alice generates the binary sequences After the sequence transformations, the similarity can be measured by calculating the Hamming distance between the transformed sequences of Alice’s synchronization symbols and every segment of Bob’s received signal. Here we want to make the cost function reach its peak value when synchronization succeeds, so the cost is rewrittenThe location of synchronization is where the function From the above derivations, we have now found a stable feature to endure phase drifts. The following synchronization scheme is based on this feature.Step 1. Alice (the sender) selects parts of the random strings as the synchronization frame see a.X and P components must be transformed. We the get two 0–1 sequences: Step 2. Alice transforms the selected sequences into 0–1 sequences using the incremental label algorithm proposed above. Both Step 3. Alice publishes the two 0–1 sequences Step 4. Bob transforms TXB) see b.Step 5. Alice and Bob synchronize at the position where the function To verify the correctness of the proposed scheme, the synchronization process is simulated as follows. The encoded random strings in Alice’s side are The cost function Equation .L should be longer than 10 and the threshold To explore the influence of SNR  and phase shifts on the performance of the proposed frame synchronization algorithm, we prepare several strings of data with natural Gaussian distributions generated from ASE output signals with length 200,000. We add Gaussian white noise of different variance to the output signal to simulate different noise environments. Here we randomly select some segments of the signals as synchronization frames, and we define the proportion of the times of successful synchronization as the success rate. To improve the success rate, the parameter X or P. If Bob applies a heterodyne detector to measure X or P can also synchronize well when the phase shifts occur.N to maximal 2048, the fraction, which is used for synchronization, is 2.048%. Our synchronization scheme requires only a small sacrifice of data.In L is mentioned above, usually it can be set as N times of subtraction operation and an N-bit Hamming distance operation. This is because data reuse is not considered. In a practical synchronization, every synchronization step needs only Complexity is another important factor for a practical synchronization algorithm. In a practical CVQKD system, the synchronization algorithm should work in real-time for high efficiency, otherwise, it will require a mass of storage to store all the received data. At first sight, according to the algorithm flow, every step has N, the calculation of a correlation needs multiplication operations to get the result of every Comparing to the frame synchronization method based on the correlation calculation , the proxity see .The realistic system may incur loopholes due to the imperfections of the implementation process, although the CVQKD protocols are theoretically proven to be secure. Traditional frame synchronization methods are performed by alternately transmitting a strong pulse. Although there has not been any practical attack on these frame synchronization schemes, the use of strong pulses can be manipulated, which may incur potential loopholes. Moreover, the frame synchronization methods based on the designing of the special frame may also introduce potential risks, since the synchronization frames can be distinguished from the key data. A well-designed synchronization method should conceal its synchronization frames into key data so that it is hard for an eavesdropper to distinguish them.Synchronization frames of this proposed scheme are similar to data frames but they are uncorrelated, which is different from the traditional schemes. In our proposed synchronization scheme, we regard parts of the signals as synchronization frames, so the signals and synchronization frames have the same distribution and the same power. If an eavesdropper intends to attack the CVQKD system through the potential loopholes in the frame synchronization method, she must distinguish the synchronization frames from the quantum signals. So she will detect the quantum signals and this will inevitably cause an increase in excess noise. Her attacks will be then found by the legitimate parties in the following key generation steps. Although the synchronization method in this article uses parts of data as the synchronization frame. The revealing of these synchronization frames does not leave any useful information about the secret key.N denotes the block length and n denotes the size of the samples used for final key generation.In our synchronization scheme, a fraction of data is used as a reference frame. It means that the scheme will also work at the cost of a slight drop in the secret key rate as the previously proposed frame synchronization schemes. We can evaluate the influence of using synchronization frames on the secret key rate when considering the finite-size effects,n in Ref . These tIn Another important thing needs to be considered here is whether the proposed frame synchronization scheme is valid or not when the attenuation of the quantum channel fluctuates. Actually, except the threshold We simulate the process of quantum channel attenuation fluctuation and test the performance of the proposed synchronization algorithm in this condition. We first simulate the synchronization performance of Bob receiving signals through a constant attenuation channel. Afterward, we change the channel into a fluctuation one and compare the matching cost of these two situations see . The matSynchronization is a crucial step in the CVQKD. Traditional methods always need to construct special synchronization frames. We propose here a simple and robust synchronization scheme without particularly designing the frame for the CVQKD system. In the proposed scheme, the sender Alice only needs to transmit parts of the quantum signals as synchronization frames to the receiver Bob. A novel feature is designed to help find the correct synchronization location. The analysis of our scheme shows that the feature we designed can tolerate phase shifts among range


It is clear that the haptic channel can be exploited as a communication medium for several tasks of everyday life. Here we investigated whether such communication can be altered in a cognitive load situation. We studied the perception of a vibrotactile stimulus presented under the foot when the attention is loaded by another task (cognitive load). The results demonstrated a significant influence of workload on the perception of the vibrotactile stimulus. Overall, we observed that the average score in the single-task (at rest) condition was greater than the overall mean score in the dual-task conditions . The walking task was the task that most influenced the perception of the vibrotactile stimulus presented under the foot. Humans constantly interact with their environment, which exposes them to conditions that can be distracting . In motion, for example, humans’ attention is mostly occupied by tasks such as walking while visualizing their space or walking while speaking to someone. These activities and/or tasks mostly involve haptic perception through the haptic channel . Indeed,This article is organized as follows. The introduction in Attention is a complex cognitive function that is paramount in human behaviour. It allows for focusing on a main task while ignoring the other aspects, but it also involves cognitive efficiency, whether in perceiving, memorizing, or solving problems . In everWhen humans interact with computers, visual rendering is mainly exploited. Although this modality is suitable in many situations, there are scenarios in which the visual channel may be overloaded by the amount of presented data. To fully take advantage of the sensory capabilities of the human system, a benefit would certainly come from exploiting other sensory channels, such as the haptic and audio ones. Moreover, several studies support the use of haptic feedback as a means of communication ,14 becauOakley and Park have evaluated the recognition rate of tactons (structured vibrotactile messages) under workload . The tacPsychologists have studied various types of activity presented in daily living situations. For instance, Kaber and Zhang reviewed the use of virtual reality and haptic technologies for studying human performance in tasks involving the tactile sense . StudiesThe wearable device was designed based on some previous works ,25,26. Ww (Equation (1)) and sent on Pacinian corpuscles mechanoreceptors field under the plantar surface of the foot.One vibrotactile stimulus (Equation (1)) with a duration of one second was designed according to various studies ,30,31,32This experiment aimed to evaluate the influence of dual tasks on the perception of a vibrotactile stimulus conveyed randomly to the plantar surface of the foot.Twenty-eight healthy students, aged from 20 to 35, from the University of Quebec at Chicoutimi (UQAC) participated in the study . They weWhat follows is a description of the setup, including the protocol.The experimental phase took place in a calm space, specifically, in our laboratory at the university, equipped with chairs and a table for the preparation of the participants. The laboratory was equipped with the flooring surface conditions and a hygienic kit to clean the device after each session was finished. This environment remained constant during the experiments.To achieve the goal set in this study, we chose four tasks/activities that, based on preliminary studies ,19,33, wOur protocol is summarized in The experiments conducted to achieve the goal of this study involved two sessions: a control session and an experimental session. The control session concerned the evaluation of the vibrotactile stimulus perception in the normal condition , whereas the experimental session concerned the evaluation of vibrotactile stimulus perception under the influence of distractors. Each session  featured a familiarization phase followed by a test phase. For all conditions, the vibrotactile stimulus was conveyed when the participant had the left foot on the ground. The control session had an average duration of 25 min, whereas the experimental session lasted for around 45 min; there was a break of 5 min between the two sessions and between each phase. The space dimension used to walk was width × length (470 cm × 80 cm). The entire method  consisted of 336 measures . In all phases, a signal (vibrotactile stimulus) is sent. Each sending of a signal is spaced by at least five seconds and the user must have his foot on the ground. A delay of three seconds is granted for the user’s response. Otherwise, the answer is perceived to be a false positive.We had different configurations for the familiarization phase. It was performed at rest (sitting on a chair) for the baseline session and while walking for the control session. During this phase, we explained and demonstrated all aspects of the experiment to the participants. For the two conditions, the participants had ear protection and the haptic device attached to the left foot. We chose the left foot for technical reasons. Nothing in the literature suggests that there may be differences in lower extremity perception due to dominance. The participants had a smartphone on the hand outside of their field of vision. They also had to look at a black spot on the opposite wall. The participants were asked to press the smartphone screen whenever they perceived a vibrotactile stimulus. We recorded 28 participants × 1 vibrotactile stimulus × 3 trials (84 measures). Each good tacton recognition was computed as haptic perception recognition rate (score). Once each participant was trained (after achieving a recognition rate of 95%), the test phase began.The test phase was performed at rest (sitting on a chair) for the baseline session. During this phase, the participants wore the haptic device and ear protection. For each condition phase, their vibrotactile stimulus perception was tested. When they perceived the vibrotactile stimulus, they pressed the smartphone screen. Thereafter, the identification was saved in the database. No results were shown to the participants during the test phase. All identifications were considered for measures. The test phase conditions were counterbalanced (randomized) between participants. When the test phase was finished, the participants were invited to fill in a form about the experience. The device was cleaned up after each test condition for the next participant.All the participants completed the experiment successfully, with 336 measures recorded . We took the mean of the three trials for each measure. Overall, we observed that the best mean perception score was achieved during the single-task condition  and the worst was achieved in the presence of distractors , as illustrated in These results indicate that the perception accuracy was greater in the single-task condition than in the dual-task conditions, as illustrated in Hypothesis 1 1)., we performed an analysis of variance with repeated measures on the accuracy of the perception score. Indeed, as our goal was to examine the effect of dual tasks on the perception of a vibrotactile stimulus presented under the foot, we investigated whether distractions  had an impact on vibrotactile stimulus perception under the foot. Therefore, we made the following assumption:1‘s null hypothesis (H01) were true and that at least one mean would be different from the others if H1’s alternative hypothesis (Ha1) were true. Our significance level was  = 0.05. The dependent variable was the score, and our independent variables were the conditions. All the tests were performed using Minitab version 17, and the visualizations were created using Power BI Desktop, January 2019 release. The analysis of variance (ANOVA) evaluation was performed with post-hoc Tukey HSD (honest significant difference) tests. Our input data independent group factors were at rest (single task), counting forwards, counting backwards, and walking . Pairwise comparisons, which identified significant differences between conditions, were used in all analyses. Statistical significance was set at the 95% confidence level (p < 0.05). The sample observation was n = 28. The data satisfied the conditions to justify the use of ANOVA. Analysis of the distribution of the data suggested that they were normally distributed.We assumed that all the means would be equal if H = 24.71, Fcritical = 2.70, p < 0.05. This result suggests that one or more of the conditions  influenced the perception of the vibrotactile stimulus. According, to Cohen’s guidelines [2) = 0.407, was a very large effect size. It also indicates that 41% of the change in the perception accuracy (score) can be accounted for by the conditions , as depicted in A one-way ANOVA with repeated measures was conducted to determine whether the perception of the vibrotactile stimulus under the foot was different for the participants in the dual-task condition. Overall, there was a statistically significant difference between the conditions, as determined by the result of the one-way ANOVA: Fidelines , a smallTo identify the conditions that most influenced the perception of the vibrotactile stimulus, we performed the Tukey simultaneous test for differences of means .We observed a significant difference between the single-task condition and the three groups of dual-task conditions . This meWe performed an analysis on false positive to see how user’s responses referred to the stimulus delivered. We performed an ANOVA analysis on false positive response between conditions. We observed a non-significant difference between all conditions. This means that participant’s answers really referred to the delivered vibrotactile stimulus.We also analyzed the normality of the sample checked. We used the Anderson–Darling test. Results diagrams and plots are reported in the The objective of this study was to evaluate the influence of dual tasks on the perception of a vibrotactile stimulus presented under the foot. The data were analyzed in terms of the overall mean score of perception within each condition, single task (at rest) and dual tasks . The results obtained in this study indicate that being in a dual-task condition is detrimental to the perception of stimuli presented under the foot.During our study, we mainly used two tasks: pure cognitive (to count) and a cognitive-motor task . The results revealed a significant difference in response time to the perception stimulus in the walking task. According to p-value of 0.272, which is not significant. This is not surprising, because the dual-task counting forwards or backwards is related to the same cortex in memory as cognitive tasks [As pointed out previously, there was a significant difference between vibrotactile stimulus perception at rest (single task) and vibrotactile stimulus perception when the participants were counting forwards and/or counting backwards . However, the results of the difference levels  for counve tasks . Thus, ove tasks ,37, whicve tasks . Our resve tasks . In summve tasks . All thive tasks ,39.The results of this study show a poor performance in vibrotactile messages perception when walking, unlike cognitive tasks. These results indicate that it is not appropriate to communicate a risk of falling throughout a haptic message presented under the foot as suggested in ,5. IndeeFollowing the design and development of a shoe capable of preventing accidental falls linked to balance problems, we studied the use of vibrotactile returns to communicate a risk of falling. The objective of this study was to investigate the influence of workload on the perception of a vibrotactile stimulus presented on under the foot. The apparatus used was a wearable device capable of conveying a vibrotactile stimulus. The experimentation protocol included four conditions: one single task and three dual tasks. The measures were performed in the controlled environment of our laboratory. Overall, the results revealed a significant influence of dual tasks on the perception of a vibrotactile stimulus. Specifically, we observed that the dual task involving walking influenced the perception of the vibrotactile stimulus far more than the cognitive task (counting forwards or backwards). These findings suggest that dual tasks could bias perception and should be taken into account when conveying vibrotactile stimuli under the foot. For this, as an alternative we intend to investigate the use of clothing for the communication of haptic information. This avenue is privileged as it appears that this new method will have a wider range of applications including spatial orientation and guidance, attention management.


Virtual screening (VS) is a computational practice applied in drug discovery research. VS is popularly applied in a computer-based search for new lead molecules based on molecular similarity searching. In chemical databases similarity searching is used to identify molecules that have similarities to a user-defined reference structure and is evaluated by quantitative measures of intermolecular structural similarity. Among existing approaches, 2D fingerprints are widely used. The similarity of a reference structure and a database structure is measured by the computation of association coefficients. In most classical similarity approaches, it is assumed that the molecular features in both biological and non-biologically-related activity carry the same weight. However, based on the chemical structure, it has been found that some distinguishable features are more important than others. Hence, this difference should be taken consideration by placing more weight on each important fragment. The main aim of this research is to enhance the performance of similarity searching by using multiple descriptors. In this paper, a deep learning method known as deep belief networks (DBN) has been used to reweight the molecule features. Several descriptors have been used for the MDL Drug Data Report (MDDR) dataset each of which represents different important features. The proposed method has been implemented with each descriptor individually to select the important features based on a new weight, with a lower error rate, and merging together all new features from all descriptors to produce a new descriptor for similarity searching. Based on the extensive experiments conducted, the results show that the proposed method outperformed several existing benchmark similarity methods, including Bayesian inference networks (BIN), the Tanimoto similarity method (TAN), adapted similarity measure of text processing (ASMTP) and the quantum-based similarity method (SQB). The results of this proposed multi-descriptor-based on Stack of deep belief networks method (SDBN) demonstrated a higher accuracy compared to existing methods on structurally heterogeneous datasets. In recent years, chemoinformatics has been an active multidisciplinary research area that is beneficial to chemistry and drug discovery, with the use of various tools and technologies. The use of virtual screening (VS) in chemoinformatics is considered as pertinent to scrutinize records of molecules and identify those structures that are most anticipated to be able to be attached to a drug target. The two main classes of VS are ligand-based and target-based VS ,2. RecenThe process of measuring the similarity between any two objects involves comparing their features. Molecular features range from physicochemical properties to structural features and are stored in different ways, which are commonly called molecular descriptors. A molecular descriptor is the ultimate outcome of a logical and arithmetical process, which converts data encrypted within a symbolic depiction of a molecule into functional numbers. A molecular descriptor may also be the outcome of a regulated experiment . The perData fusion is a technique involving the combination of multiple data sources which are translated into a single source, in which the result for the fused source is expected to be more informative compared to the results of the individual input sources ,9. The cVarious techniques have been introduced to reduce dataset dimensionality. One of the best-known techniques developed to deal with data dimensionality is principal component analysis (PCA). This mathematical procedure is used to ensure that the reduction of large dataset variables into smaller sets does not affect the most useful information. The number of (possibly) correlated variables is transformed into a  number of uncorrelated variables, which are called principal components. PCA is designed to preserve the greatest variability of data and ensure that the data are later translated into new variables that are linear functions of the original dataset. These new variables have a consecutively maximized variance and are uncorrelated with each other . The useRecently, deep learning (DL) techniques have been successfully used in several fields. The learning of parameters in deeper architectures can be a challenging optimization task, similar to that found in a neural network with multiple hidden layers. Hinton et al.  suggest ALogP types extended connectivity fingerprint counts (LCFC), ALogP types daylight path-based fingerprint counts (LPFC), functional class extended connectivity fingerprints (FCFP), ALogP types daylight path-based fingerprints (LPLP), and ALogP extended connectivity fingerprint (LCFP) [An improved deep learning method for molecular similarity searching that utilizes feature selection.An introduced Stack of DBN method for features reweighting is proposed by emphasizing more weights to the important features.The proposed method showed promising results in terms of overall performances in comparison to the benchmark methods.The DBN’s deep-learning procedure is made up of two phases: layer-wise feature abstraction and reconstruction weight fine-tuning . In the t (LCFP) ,41,42. EThe core principle of feature selection is to choose a subset of all variables, so that a large number of features with little discriminative and predictive information is eliminated ,44. The Most of the current similarity measures consider all molecular features to be of equal value with the same importance, and all these features are used in calculating similarity measures, which is considered as a drawback. According to Vogt et al. , the recVarious text retrieval studies have shown that, even if the effectiveness of the provided algorithms is similar, the use of different models of retrieval or ranking algorithms will result in low overlap between identified relevant and non-relevant documents . Thus, TPCA has been used in chemo-informatics in several previous studies. Cao et al.  conducteA study of DBN techniques has been successfully conducted to perform feature abstraction and reconstruction of images ,63,64,65Most similarity techniques are based on the assumption that molecular fragments without links to any biological activity are similar in weight to the crucial fragments. Generally, chemists refer to elements in structural diagrams, such as functional groups and place importance on certain fragments over others. Researchers in this domain scrutinize all fragments in the chemical structure of compounds and assign additional weights to the more important fragments. Thus, a match involving two molecules with highly weighted features would be of more significance to the overall similarity than a match involving molecules with low weighted features ,69. In aThis section presents the methods used in the research, which involve the use of the Stack of deep belief networks method (SDBN) model for molecular similarity searching. We first describe the concept of the DBN method for molecular feature reweighting and then present the proposed SDBN method for molecular similarity searching.As shown in  ith RBM . The outo layers ,73,74, io layers , RBM is o layers ,77. The Generally, RBM has the structure of an energy based model that can be used for the visible and hidden layers <v; h> possessing the weights matrix W, that is linked to the connection between v and h . An RBM’The probability for every unit within the hidden layer Similarly, we can calculate the conditional probability of We can obtain the network’s learning rules in the log-likelihood-based training data using alternating Gibbs sampling ,78. In GThe angle brackets denote the expectations given within the distribution, specified by the succeeding subscript. This creates a very simple learning rule that can be used to conduct the stochastic steepest ascent for the training data’s log probability:Since no direct connections exist between the hidden units in an RBM, these hidden units can be considered as independent of the visible units ,80. Usinectively .This section presents the proposed model, which utilizes the stacked DBN based multi-descriptors method for molecular similarity searching. Eight 2D fingerprints were generated by SciTegic Pipeline Pilot and PaDEL descriptor software . These w(1)All stages combine two descriptors , , ,....)(2)All stages combine three descriptors , , ….)(3)All stages combine four descriptors , , )(4)Then, combine five descriptors .The key element of the method is the representation used to translate a chemical structure into mathematical variables. Some descriptors and molecular representations are complementary to other descriptors, and thus could yield better results when used in combination. This means that different descriptors may produce different results for molecular similarity searching. It incorporates several molecular representations in merging and combining the features from multi-descriptors that can improve the performance of the similarity search. In this study, we trained five of these descriptors  and all the combined probability stages have been undertaken as follows:After training all these stages, we found that the best results were obtained with the combination of three descriptors: ECFC4, ECFP4, EPFP4. The results of this combination were then used as a new descriptor for the similarity search. The design for combining multi descriptors with DBN for reconstruction of feature weights is shown in detail in The training of DBN was conducted in both stages (pre-training and fine tune). In this study, DBN was trained with different architectures and different numbers of RBMs. The DBN trained with two RBMs, three RBMs, Four RBMs, and five RBMs successively and with different learning rates , and different epochs . The weights were randomly initialized between 0 and 1. The configuration that obtained the best results was that using five RBMs , 70 epochs and learning rate =0.05. The size of the input layer in the first RBM was 1024; it was similar to the size of the dataset vector with reference to all the molecular features, while the size of the output layer in the first RBM was 2000. After completing the training of the first RBM, the output from the first RBM became the input layer to the second RBM and the size of the output layer was 1800. The third RBM input layer was the output from the second RBM, with size 1800, and the size of the output layer was 1300. Similarly, the output of this RBM was used as the input to the fourth RBM where the output layer size was 800. Finally, the last RBM input layer size was 800 and the output layer was 300. During the training of the RBMs, we used epoch = 70, Gibbs steps = 50, batch size = 128, and learning rate = 0.05. After training in the first stage, backpropagation was used to fine-tune the weights obtained from the first stage “pre-training stage”. The output from this training was pretrained as the new vector with the new weight. For testing, we calculated the reconstruction features’ weight by comparing them with the original features’ weight. Once all the RBMs had been trained and the weights saved for all RBMs, the DBN pre-training was complete. DBN then performed a backpropagation to fine-tune the weights. A reconstruction error was calculated for each input.The next step after training of the DBN is complete, and all new weights for all molecules have been stored, is to apply PCA. PCA has been used to decrease the molecules dimensionality (features) and filter features according to the percentage of reconstruction feature error, as shown in PCA remains among the fastest algorithmic methods used in the implementation of non-linear dimensional reductions, to decrease the dimensionality of each feature vector while gaining more efficient features . The aimnd so on .In this proposed method, three different types of data sets were trained for the dimension reductions which are MDDR-DS1, MDDR-DS2, and MDDR-DS3. Each data set contained 102516 molecules (samples), and each molecule was represented by 1024 features (variables). We used X to represent the training data set as a training matrix, N is the number of samples or molecules and M is the number of dimensions for each molecule (sample).Each vector in the training matrix X represents one molecule with M features, so that each molecule has 1024 features.Each column in the training matrix X represents a feature for N molecules or samples, where N is equal to 102,516 molecules.Three different structures of the DBN method were used in this study for reweighting molecular features, as described in , where Prior to implementing PCA, the reconstructed feature error matrix (E) is transformed as follows: k principal components (PCs) and we used k = 3 to obtain PCA1, PCA2, and PCA3 and used the 3D coordinates to draw the features based on the three values, as shown in The PCA space represents the direction of the maximum variance of the given data. This space consists of Following this, the distances between all features and original points were calculated through D = In this study, we selected 300 features from each fingerprint, using those features which had lower error rates after filtering all the features according to the percentage of reconstruction feature error and applying the threshold. To evaluate the performance of the proposed method, we conducted a series of experiments to fulfill the aims of this research, which are (1) What is the performance of the proposed DBN for similarity searching in virtual screening? (2) What is the performance when using stacked DBNs with multi-descriptor features reweighting and selection? (3) How does the proposed SDBN improve the performance of the similarity searches?MDL Drug Data Report (MDDR) datasets were used to validate the effectiveness of the SDBN method with multi-descriptors, using reweighting of molecular features and feature selection for molecular similarity searching.MDDR collection of datasets  remains i (search similarity method) is given the rank j (activity class), where there are in total n objects and m judges. Then, the total rank given to the object i isOne of the important measures using a quantitative approach is the Significance Test which is used to measure the performance of the similarity approach. The Kendall W test of concordance is used  in this The sum of squared deviations, S, is defined asThe Kendall W test shows whether a set of judges can make comparable judgments about the ranking of a set of objects. In the experiments conducted as part of this paper, the activity classes of each of the data sets were considered as the judges, and the recall rates of the various search models were considered as the objects. The results gave the value of the Kendall coefficient and associated significance levels, which indicates whether the value of the coefficient could have occurred by chance. If the value was significant, , then it was possible to give an overall ranking to the object. The similarity methods based on the reweighted fragment were also compared to standard methods such as BIN , TAN 8888, ASMTPSQB: This is a molecular similarity method that utilizes a quantum mechanics approach. The method specifically relies on the complex pure Hilbert space of molecules for improving the model’s performance.ASMTP: This is a similarity measure based on ligand-based virtual screening. The method was designed to utilize a textual database, p, for processing chemical structure databases.TAN : This meBIN : This seThis section presents various existing methods that serve as the basis for performance evaluation for the proposed model. These include:Experimental simulations of virtual screenings using MDDR datasets demonstrated that this proposed technique allows various means of improving the efficiency of ligand-based virtual screenings, particularly for more diverse datasets. The MDDR benchmark datasets  are three different types of datasets chosen from the MDDR database. The MDDR-DS1 includes eleven activity classes, some actives of which are structurally homogeneous while others are structurally heterogeneous. The MDDR-DS2 dataset includes ten homogeneous activity classes, while the MDDR-DS3 data set includes ten heterogeneous activity classes.From MDDR-DS1, MDDR-DS2, and MDDR-DS3, ten active molecules were randomly selected from each activity class, which are called as reference structures. The similarities between each reference structure and all the molecules in each database were calculated. The results of this similarity were then ranked in decreasing order and only 1% and 5% were selected for each reference structure. The results obtained for each reference structure were investigated to see how many active molecules belonged to the same activity group, referred to as true positive values of the retrieval results. These values were calculated for the ten reference structures, and the average of these values, known as the recall value for the activity class was calculated for the 1% and 5% cutoffs. This procedure was repeated for all datasets. Tables 4, 6, and 8 show the dataset activity classes in the first column, while the other columns show the average of recall values for all activity classes at the cut-off 1%. Tables 5, 7, and 9 show the dataset activity classes in the first column, while the other columns show the average of recall values for all activity classes at the cut-off of 5%. The end of each column shows the overall average recall results of all classes. The best average recall for each class is highlighted. At the bottom of each column, there is a row of shaded cells that corresponds to the total number of shaded cells for all the similarity methods that achieved best results.Some molecular representations and descriptors are complementary to the others; hence, their combination can yield good results. This indicated that use of different descriptors could yield differing similarity searching results since they incorporate various molecular representations. Based on these reasons, the SDBN model has shown an improvement for the molecular similarity searching based on reweighing and combining different molecular features. The key idea of using SDBN in this work is to learn a rich representation using unsupervised learning to provide a better similarity metric for ligand-based virtual screening. The reconstruction weights for all molecules’ features were obtained, along with PCA to reduce the dimensionality of these molecular features based on selecting the features that have a low error reconstruction error rate and removing the features’ outliers. SDBN was implemented with the important features selected from all the descriptors that were combined to improve the molecular similarity search process.The results obtained by SDBN are shown in Tables 4–9. The proposed SDBN method is compared with four different benchmark methods that have been used recently for similarity searching, which are BIN, TAN, ASMTP, and SQB.The MDDR-DS2 dataset includes ten homogeneous activity classes. The molecules in this dataset are more alike and have low diversity. Nevertheless, SDBN with the top 5% of recall results still showed a good performance in this case and performed better compared with BIN with gains of 2.68% for overall average results and 15.86 % with TAN.The MDDR-DS3 dataset includes ten heterogeneous activity classes. The molecules in this dataset are highly diverse. The best SDNB results were obtained with the MDDR-DS3 dataset compared with the other two datasets. p < 0.05; only then is it possible to give an overall ranking to the similarity methods.The performance of the similarity methods of the MDDR datasets was ranked by applying Kendall’s W test of concordance. Here, the judge ranking (raters) of the similarity methods (ranked objects) is considered based on the recall values for all activity classes . The Kendall coefficient (W) and the significance level  are the outputs of this test, where the p value is considered as significant if In This study has emphasized the usefulness of deep learning methods for exploring ways to enhance similarity searches in virtual screenings. In addition, the use of deep belief networks associated with the concept of data fusion has been investigated in this study. The aim of this research was to ensure that reliable reconstruction weights for all molecular features were obtainable in several molecular descriptors to reweight molecular features and selected only the important features, i.e., those have more weight with lower error rates and to remove the feature outliers. The feature outliers are those with large reconstruction errors, which can be identified by analyzing the distribution of the reconstruction errors. The experimental results showed that the SDBN with multi-descriptors enhanced the effectiveness of ligand-based virtual screening in chemical databases, establishing that the SDBN can be implemented successfully to enhance the performance of similarity searches. The experiments were conducted based on the MDDR benchmark dataset and revealed the ligand-based virtual screening of chemical databases, to be more effective than the other methods considered. Generally, the screening and evaluation results indicated that the proposed model provides an improvement over other similarity procedures, such as TAN, the SQB method, BIN, and ASMTP. The evaluation of the results achieved from conducting the screening showed that the performance obtained by the proposed measure was improved and, particularly, that the performance of SDBN with the structurally heterogeneous data sets (MDDR -DS1 and MDDR -DS3) achieved superior results compared with the other methods which have been used in previous studies to enhance molecular similarity searching.


A crop monitoring system was developed for the supervision of organic fertilization status on tomato plants at early stages. An automatic and nondestructive approach was used to analyze tomato plants with different levels of water-soluble organic fertilizer (3 + 5 NK) and vermicompost. The evaluation system was composed by a multispectral camera with five lenses: green (550 nm), red (660 nm), red edge (735 nm), near infrared (790 nm), RGB, and a computational image processing system. The water-soluble fertilizer was applied weekly in four different treatments:  and the vermicomposting was added in Weeks 1 and 5. The trial was conducted in a greenhouse and 192 images were taken with each lens. A plant segmentation algorithm was developed and several vegetation indices were calculated. On top of calculating indices, multiple morphological features were obtained through image processing techniques. The morphological features were revealed to be more feasible to distinguish between the control and the organic fertilized plants than the vegetation indices. The system was developed in order to be assembled in a precision organic fertilization robotic platform. These moil type .Deficient uses of organic fertilization can cause “abiotic diseases” on plants, decrease the suppressive effect of the soil, and the readiness of plants to defend themselves against attacks of plant pathogens . NutrienMeta-analysis studies indicate that the optimization of the use of nitrogen fertilization in tomatoes can decrease costs and environmental impact maintaining the same yield levels . Focusinhttps://projects.au.dk/coreorganiccofund/core-organic-cofund-projects/sureveg/) Project seeks to develop and implement innovations for intensive cropping systems using strip-cropping and fertility strategies. These farming systems consists of inter-row cropping with different families and species in the same plot. The increase in biodiversity enhances the resilience of the field against soil-borne disease, and pest attacks, which at the same time increase the soil protection against erosion and nutrient depletion.The SUREVEG  and computer system that control selectively foliar spray applications .Other systems based on real-time analysis were developed in order to deliver site-specific herbicide applications in arable crops. Besides the multispectral images and the vegetation indices, these systems use a bicameral system, binarization techniques, and morphological algorithms to differentiate between the crop and spontaneous plant species ,25,26,27Both types of real-time precision spraying systems are commercially available or are in process of being commercialized, and have shown a positive reduction in the use of chemical compounds with a reduction of costs and an increase in yield levels (in the case of precision fertilization) compared to traditional sprayers .Image processing techniques with multispectral cameras from visible to near-infrared spectrum are also being used to provide non-destructive plant phenotype image datasets. These approaches have allowed more precise and real-time, high throughput, and high-resolution data for the modelling and prediction of plant growth and morphological development in different conditions, with recent applications in plant health analysis ,29,30.More advanced prototypes were made in order to insert these types of sensors in autonomous vehicles. Several terrestrial robotic platforms were developed using multispectral cameras seeking for a more automatic, low-energy cost and accurate analysis of the crop parameters compared to conventional equipped tractors and unnamed aerial vehicles ,31,32.Some of these autonomous agricultural devices are capable of identifying weeds and control them using low-dose spraying, mechanical control or thermal control . Other pBesides the advances in the robotic agricultural platforms, most of them were developed for conventional orchards and arable crops, presenting a lack of technologies in the context of organic horticulture. For this reason, the SUREVEG team has developed the robotic prototype shown in This prototype is a cart with a manipulator robot and multiple sensors and actuators. The proposed sensors are laser scanners to build three dimensional models of the plants, which will be used to study their growth, and a multispectral camera, which will be used to study the state and health of the crops. Moreover, a system to apply treatments to the soil will be installed in the prototype, consisting of a tank to store the liquid treatment and a sprayer to apply it. Finally, the manipulator robot will be available to accurately place sensors and actuators at the target points.Having in mind this long-term goal of automatically monitoring crops and applying treatments, this work is focused on the collection of information about the plants by using multispectral imagery. Specifically, the paper describes the acquisition of multispectral images of tomato plants with different levels of organic fertilization, as well as the estimation of their nutritional states in the early stages through multiple vegetation indices and morphological features by using computer vision techniques.The experiment was carried out on a greenhouse in order to obtain the different spectral responses of plants. This greenhouse is in the facilities of the School of Agronomic, Food and Biosystems Engineering of the Technical University of Madrid, which are located in Madrid, Spain (40°26′19.9″ N 3°44′15.7″ W). The tomato (Mina F1 cv.) seedlings were initially cultivated in 3-l pots with height and diameter of 15 cm, filled with a mix of peat substrate (50%) and coconut fiber (50%). After 5 weeks of experiment, the plants were transplanted to 8-l pots with the same mixture.Fertilizante Huerto y Frutales). The initial volume of irrigation and fertilization was 300 mL of solution. One week after transplant, plants were assigned to the different treatments according to the fertilizer label . The plants also received two supplements of vermicomposting: one in the first transplant  and another after the second transplant . Images of the plants were acquired with 7-days intervals and processed to compute vegetation indices and analyze them.The experiment was designed as randomized blocks, with four treatments to be applied to four groups of plants . The trial design is described by The general system was composed of images acquired from the greenhouse experiment with different levels of fertilization. The evaluation system was composed of a bracket support stands clamp supporting the sensor and the power supply (battery). The platform has the possibility of height adjustment for the sensor. The distance between the sensor and the bottom of the pot was set at 0.7 m during the first 5 weeks and moved to 1.4 m after Week 5 , since tThe camera used for image acquisition was a multispectral camera  which was originally designed for use in agricultural Unmanned Aerial Vehicles (UAV’s). Its internal sensor is composed of four spectral bands which register the reflected light coming from the vegetation and can be used to distinguish plant vigor based on reflectance levels . These bThe Sequoia Camera has also a sunshine sensor that is used to calibrate the images depending on the sunlight. This makes it possible to compare photos over time, despite variations in light during photo shoots. The sunshine sensor is attached on the upper part to the system, facing the sky and correcting the signal and it also contains a GPS/GNSS module, a magnetometer and inertial measurement system .The images were taken with and without a black background placed to help the segmentation. The developed algorithms showed different efficiency in segmentation with or without the background. The pots were tagged with a coloured stamp in order to take the images in the same position during the experiment b. The seThe computational system used for the image analysis is composed of four major steps: pre-processing, calculation of vegetation indices, image segmentation, and morphological analysis.x- or y-direction (or both) was used.The Sequoia sensor produces tagged image file format (TIFF) images with a size of 1280 × 960 pixels. The camera was designed to take images from a minimum distance of 30 m to the target . When thThe parameters used to define the number of pixels and the direction that each image should be shifted were obtained using a sample image with a referential point and then applied subsequently to all collected data. The red image was arbitrarily selected as the reference, and then the green, near-infrared (NIR) and red-edge images were shifted in order to be in the same position as the red image. The parameters used for the distances (0.7–1.4 m) are shown in The central area of the images was defined as area of interest (AOI) and the consequent processes were used in this area in order to reduce the computational process and optimize the plant recognition and binarization. The images were clipped with AOI-x: 200:900 and AOI-y 200:1000, reducing the size of the image from 960 × 1280 to 700 × 800 squared pixels .Vegetation indices (VIs) computed from multispectral images are quite simple and effective parameters for the qualitative and quantitative evaluation of vegetation vigour, cover and growth dynamics among other applications. There is a vast implementation of these indices using different airborne and satellite platforms with recent advances using unmanned aerial vehicles (UAV) and tractors. Due to the complexity of the instrumentation platforms, light spectra combinations and resolutions used, there is no unified mathematical expression that attends all applications of VIs .According to the same authors , customiEleven VIs retrieved from the literature were adopted to find correlations between the fertilization status of the tomato plants and the spectral response. One of the vegetation indices were also used as a filter parameter for binarization of the plant leaves despite the background (image segmentation).The VIs were chosen based on the literature and in the available bands present at the sensor. The list of indices, abbreviations, formula and traditional application of the VIs can be observed in Digital image processing and computer vision approaches are powerful tools for plant analysis because they allow plant physiological and physical features to be measured non-destructively with high temporal and quantitative resolution . After tOther systems were developed using a fixed threshold and image standardization method . Normalibwarefilt function of MATLAB was applied. This function allows extracting all connected components from the binary image, where the area of the objects is in the specified range producing a new binary image containing only the objects that meet the criterion. In this case, the criterion was to extract the larger object of the image (with more connected components). This allowed the system to eliminate noises and unconnected pixels. The resulting binary image (mask) . After this step, the e (mask)  was multe (mask) .Nutrient deficiency in tomato plants can cause several symptoms besides the change in reflectance of leaves. It is common to observe changes in the plant growing behavior, specially related with morphological aspects of the leaves and shoots.Nitrogen deficiency can cause restricted shoot growth and small erect leaflets. Phosphorous deficiency can cause restricted shoot growth and small stiffed curved leaves. Deficiency of potassium can cause scorched and curled symptoms in old leaves. Zinc and Iron deficiencies can cause stunting. Boron and Calcium deficiencies can cause changes in the leaflets making them curved and deformed. Copper deficiency symptoms can be observed in the margins of leaflets and younger leaves that curl into a tube shape, the terminal leaves can become very small, stiff and contorted and the stem growth become somewhat stunted .From the binary plant datasets, it is possible to measure plant size, shape, area and other features in an automated way and correlate plant phenotypes with experimental treatments ,43.Several other morphological properties can be extracted from binary images using the regionprops function of MATLAB. This function uses the distribution of the pixels to calculate shape parameters and has been recently used for image analysis in medical studies , industrThese parameters can provide reliable information about the changes in the plants’ area, perimeter, shape and growth behaviour.Not all parameters resulting from the regionprops function were used, and the description of the morphological properties used for the tomatoes plants analysis can be observed in The experiment used randomized blocks, with four treatments and six repetitions. Each plot was composed of six pots. The statistical analysis carried out through the ANOVA and Tukey’s HSD test with 99% of confidence for average comparisons using the statistical and machine learning toolbox from MATLAB software. ANOVA and Tukey’s HSD test were also performed with 95% confidence, but since there it was no difference in terms of significant factors, the 99% was chosen for this study. For the significant predictor factors, a functional boxplot and a regression analysis (using the week means) was performed in the same software.Images were collected weekly (one image per plant) during the course of eight weeks. The segmentation of the plant tissue from the background was needed to quantify the spectral response from each plant and to calculate the mean values of the evaluated parameters.This process also enabled the morphological analysis of the plant growth behavior. Multiple segmentation codes were tested to remove the background, and the code based on the normalized difference vegetation index (NDVI) was the most efficient.All the other bands  alone were not able to differentiate the plant tissue from the background. The vegetation indices results were stored by date and treatment in image format seeking a visual confirmation of the segmentation process and a visual representation of the spectral responses .The images were taken in the same position during the weeks in order to observe the growth behavior of the plants in the different treatments. Both aspects could be observed during this time: morphological traits and vegetation indices .p-values are displayed below  was performed to find which parameters are more robust to use in an automatic and nondestructive fertilization level analysis model.The graphical representation of the boxplots and Tukey’s comparison test can be seen in The results presented in The regression models were created using the MATLAB function curvefit, the mean values of the control (T0), and the recommended dose (T3), for the first five weeks and excluding outliers. The selected parameters were: area, filled area, convex area, perimeter, equivalent diameter, major axis length and minor axis length .Seeking for a more robust, simple and reliable prediction model, functional boxplots were created using the mean values of each plant T0 and T3 for the first four weeks and plotted together. The functional boxplots represent the mean data of each week without the outliers with an interval of confidence of 95% .The system used for image acquisition showed to be effective for tomato plants at early stages. The same system could be adjusted for tomato plants at late stages, as well as for other plant species and seedlings in different stages (depending on the height adjustments). The pre-processing technique showed to be adequate for the overlay of the Parrot Sequoia images in the different heights tested 0.70 m and 1.40 m.Plant segmentation is the cornerstone of the image analysis at plant level. Diverse authors used multispectral images to access the nutritional level of the plants. These works are usually focused in small zones or areas, or in multiple plants in a parcel as the minimum management unit. In order to perform precise fertilization at the plant level, the segmentation and extraction of the plant of interest must be also precise. The algorithm developed using the normalized difference index was able to distinguish plant tissue from soil, pot, floor and metal platform extracting the whole plant apart of the background . The useThe vegetation indices used in this study did not show significant differences between the treatments during the initial stages of the plant development, but there were significant differences in the 6th week after the transplant between T0 and T2 and T0 and T1. Among all the studied vegetation indices, the NDVI, GRVI, OSAVI, SR, and MSR were sensible to these differences. Different results were found by , that usAccording to the authors , the SPAThe use of NDVI and the SPAD measurements as a nitrogen level parameter allowed to reduce significantly the use of nitrogen fertilization maintaining the same yield levels and fruit quality compared to the reference plot . On the The first factor is the type of fertilizer, in all the previous studies, mineral nitrogen fertilization was used. This type of fertilizer is more soluble and less stable and can be absorbed and translocated faster than the organic amendments. The crop stage during the measurements can also be a factor. In the previous studies, the measurements started after the 29th day after transplant where the plants were developed more, and the deficiencies symptoms were more present. The type of sensor is also an important factor because this was the first study to use the Parrot Sequoia sensor at plant level to monitor nutrient deficiency in tomatoes. , showed The most interesting results were obtained through the analysis of the morphological traits of the tomato plants. Seven of the twelve aspects analyzed showed significant difference between the control and the fertilized treatments. Only three weeks after the transplant, the plants started to exhibit a significant difference in the plant area, convex area, filled area, minor axis length and equivalent diameter discriminating T3 and T2 of T1 and the control. In the same period, the perimeter was significantly different between the recommended dose T3 and the control, without difference between T2 and T1. The length of the major axis could discriminate between the control against the treatments but not between the treatments already in the third week. In the following weeks, from the fourth week to the end of the study, all the seven morphological traits  showed significant differences between the control and the treatments, except for the major axis length in the seventh week after transplant.These results are in agreement with the results described in the literature. Mooy et al.  showed tThe use of functional boxplots as a predicting model is interesting because they can exclude outliers and can be used as a threshold parameter . As can The computer vision methodology developed in this article was suitable for monitoring the development of tomato var. Mina F1 at early stages with different levels of organic fertilization in a protected environment. The methodology involves consecutive steps, and using an automatic and non-destructive approach provides several morphological and spectral aspects of the plants that can be used in a robotic platform.After the pre-processing, the NDVI threshold showed to be crucial and effective for plant extraction. Among the several spectral (vegetation indices) responses observed, the NDVI, OSAVI, simple ratio, GRVI, and modified simple ratio were the only ones that showed a small difference between treatments in the sixth and seventh weeks after transplant, but without practical applications.Studying the evolution of a crop by using multiples images allowed to extract several morphological features. The automatic morphological analysis was able to distinguish between the control and the fertilized treatments from the third week after transplant until the end of the experiment with 99% of confidence according to the Tukey’s honest significant differences test.Among the studied morphological parameters, area, convex area, filled area, perimeter, major axis length, minor axis length and equivalent diameter showed to be useful to distinguish between the control and fertilized treatments. Convex area was the parameter that presented the highest difference between control and the fertilized plants according to the functional boxplot analysis.This was the first study to relate the use of morphological features extracted automatically with the organic fertilization level in vegetable plants. The image analysis system was able to distinguish plant tissue from soil, floor, and metal structures. Therefore, most of the image processing steps can be extrapolated to other vegetable cultures, although future analysis is necessary to evaluate its robustness. Moreover, the Parrot Sequoia camera fulfilled the requirements to be used in the robotic platform and besides be originally developed for aerial images this study proved the capacity of its use at short distances.Additionally, the same methodology can be used for analyzing other aspects related with morphology features in tomato plants, such as disease severity, hydric and salinity stress or any environmental modification that can cause aboveground changes in the morphology or spectral responses of vegetable plants. This technique has potential to be used in many branches of plant health studies and provides an innovative and automatic way to measure the severity of abiotic disorders.In a subsequent investigation, we are developing an extended version of the methodology to be applied using the robotic platform at the field level. Trials are being conducted in an experimental line with different vegetable crops to study the best image acquisition interval in a scenario with movement. Other trials using the robotic platform are being conducted in an experimental field area with different crops such as onion, pumpkins, tomatoes, and zucchini with different levels of fertilization. In these trials, the multispectral camera is being integrated with laser sensors (Lidar) for a multiscale analysis of the crops.