The millimeter-wave radar has been widely used in traffic applications. However, little research has been done to install the millimeter-wave radar on the top of a road for detecting road traffic flow at a downward looking direction. In this paper, the vehicle parameters, including the distance, angle and radar cross-section energy, are collected by practical experiments in the aforementioned application scenario. The data features are analyzed from the dimensions of single parameter sampling characteristics and multi-parameter relationships. Further, the correlations of different parameter series are given using the grey correlation analysis method. For millimeter-wave radar used in the traffic flow detection, our work can definitely provide significant support for further intelligent transportation applications, such as vehicle trajectory tracking, traffic flow estimation and traffic event identification. Real-time road traffic parameter collection and accurate road congestion evaluation are the prerequisites for applying better traffic congestion avoidance strategies to improve the traffic flow ,2. The tAmong the methods mentioned above, the radar based method has certain advantages. For example, it can avoid ruining roads when deploying the devices and it is affected little by environment, therefore it has been widely used in the field. In the radar detector, the target vehicle is identified by the differences between the transmitting signal and the echo signal and the vehicle speed is calculated based on the Doppler effect. As a maturely applied traffic flow sensor, the radar has been used in urban traffic and expressway management, such as the traffic volume collection and over speed detection. In , the radCurrently, the commonly used radars mainly work at K frequency and X frequency bands ,17. ThesAs the advantages mentioned above, the echo signal of the FMCW millimeter-wave radar contains two kinds of objects, the moving object with a certain Doppler speed and the stationary object with strong radar cross-section (RCS) energy. The stationary objects are not only the vehicles but also all kinds of surrounding road information, such as road, guardrails, lamp posts and other information, which are collectively known as the background noise. Since the target materials and the reflecting surfaces of the objects are different, the received RCS energy values are also distinct. This feature can be used for distinguishing vehicles and background interferences by setting certain RCS energy thresholds. Currently, the millimeter-wave radar can detect a wide range of intersection and it is also capable for tracking low speed vehicles (0–25 km/h), overcoming the aforementioned shortcomings of the traditional microwave radar. Nowadays, the millimeter-wave radar is mainly used for driving environment perceiving in the advanced driver assistance systems (ADAS) and UAV. The main work related to such radars can be classified into two directions. (1) progress towards hardware and software based on driving environments, such as the antenna optimization and improvement of the target resolution capacity as a lower layer and (2) data mining research for forward obstacle detection subjected to actual applications such as the target identification and trajectory tracking as higher layer.In , authorsFor target identification and trajectory tracking research,  proposesAs elaborated above, current research on radar for traffic applications can be generalized as follows: On one hand, in the traffic flow detection applications, the applied radars are mainly the 24 GHz microwave radar and they are generally installed on the roadside, detecting vehicle targets from the view orthogonally to the movements of vehicles. On the other hand, the millimeter-wave radar is mainly used for the road environment perceiving in ADAS and UAV applications. The radar is installed onboard for the forward obstacle detection. Little research has been carried regarding installation of the millimeter-wave radar on top of the road for detecting road traffic at a downward direction towards traffic flow. In this paper, we make an attempt that the FMCW millimeter-wave radar is used to detect traffic flow characteristics and analyze the data features. Differing from the onboard installation method for ADAS applications and the orthogonally to traffic flow installation method of traditional radars, we install the radar device above the road, for example, at the cross arm of electronic police devices. In this case, the radar detects the vehicles at a downward looking direction. By practical experiments, we collect the vehicle parameters and analyze the data features, including the sampling characteristics of the distance, angle, RCS energy and the multi-parameter relationships. Further, the correlations of different parameters are given using the grey correlation analysis method. For millimeter-wave radar used in the traffic flow detection field, our work can therefore provide useful support for further intelligent transportation applications, such as vehicle trajectory tracking, traffic flow estimation and traffic event identification.The rest of the paper is organized as follows. In this research, the non-scanning multi-target millimeter-wave radar having FMCW mechanism is used. The core-configuration of the radar are as follows:The radar signal processing chip: NXP-MPC5775K, a 32-bit heterogeneous multi-core microcontroller with two independent e200z7260 cores and an e200z420 core.The radio frequency front-end: NXP-MR2001, a high performance multi-channel 77 GHz radar transceiver chipset. This contributes to better system-level radar solutions with the MPC5775K chip.The Antenna system: The self-developed planar micro strip array antenna, which contains two groups of transmitting antennas and three groups of receiving antennas.The detailed technical parameters of the radar device are introduced in To collect traffic data for further analysis, we have conducted the experiment at Yangzijiang Road, Yangzhou City of China, as the sample case study. We have installed the device on a platform bridge across the experimental road, as is shown in As is shown in The millimeter-wave radar used in this research is a non-scanning multi target radar with the FMCW mechanism. The detailed technical parameters of the radar device have been introduced in In millimeter-wave radar, the distance of the target is acquired based on the time delay effect. The radar sends millimeter-wave signal, receives the echo signal and the processing system calculates the target distance further according to the frequency difference and the delay time between the two kinds of signals. The distance measurement algorithm can be expressed by Equation (1).When the radar is installed above the road, the distance of the vehicles can be detected as shown in In Based on And the theoretical farthest distance is:In V m/s, where the actual moving speed for a single vehicle is not represented as hypothetical velocity but rather general traffic flow rate in a short term which is very close to the actual scenario.The hypothesis for further macroscopic data sampling characteristics analysis is described as follows. Considering that a vehicle moves at a constant velocity Considering the measuring distance as independent variable, the derivative of Equation (5) can be shown in Equation (6).In Equation (6), In our experiment, the pitch angle of the radar device In the effective detecting region, when In ineffective detection region, the detecting vehicles are beyond the theoretical effective range of the radar. This problem is mainly caused by the side lobe interference. The data in these regions should be rejected for further vehicle analysis.In this work, the radar calculates the target angle using the phase difference method based on the linearity of the radio wave propagation and the directivity of the antenna beam. Different antennas receive the same echo signal and the processing system analyzes the phase differences for vehicle angle calculation, as is shown in The target angle is calculated by Equation (7).In the traffic flow detection scenario, the angle measurement can be shown in In In our experiment, the max angle measurement range is In Similarly, suppose that the vehicle moves at a constant velocity When The probability distribution for angle data sampling after rejecting the abnormal data is shown in In this paper, the RCS energy is calculated by the signal level of the actual echo signal wave which depends on the RCS characteristics of the target itself. When obtaining the analog-to-digital converter (ADC) sampling points of the echo signal, a data conversion is taken for the signal level to RCS energy value, which represents the signal strength reflected by the target. In the experimental road, there are many types of motor vehicles, such as buses, cars, trucks, agitating lorries and others. In order to further analyze the RCS energy characteristics for different type of vehicles, the sampling distribution for the big vehicles (mainly the buses) and the small vehicles (mainly the cars) are shown in Based on From The curve shown in In actual experiment, there is certain relationship between two different measuring parameters. In this section, the data characteristics between the RCS energy and distance, angle error and distance and angle error and RCS energy are discussed.y-axis implies the average RCS energy value for small vehicles detected at a certain distance.The statistical results of the RCS energy under different distances are presented in As shown in y-axis implies the standard deviation of the angle error values which is calculated based on the assumption that the vehicle moves along the center line of the lane and the theoretical angle is 0.Taking the middle lane as an example, the detecting angle error under different distances and the results are shown in From Taking the middle lane as an example, the detecting angle error under different RCS energy values are shown in As shown in In The grey correlation analysis is an important research area in grey system theory. It uses the linear interpolation method to translate the discrete observed value of behavior series to a piecewise continuous broken line and further estimates the correlation of different series according to the geometric curve shapes of the corresponding broken lines.Suppose the series of the characteristic behaviors are:The series of relevant factors are:The displacement difference of series Further, The correlation degree for In Equation (13), Let And,The resolution ration As elaborated in , the aboIn our experiment of millimeter-wave radar for traffic flow detection, the vehicle travels in one direction only, hence the target distance parameter is absolutely monotonic. As introduced in According to the analysis described in In Equation (19), The weight distribution is shown in Finally, the average of correlation coefficients of all data in the parameter series is taken as the index for describing the grey-relation degree, as shown in Equation (22).In Equation (22), As described above, we choose the target measuring distance as the characteristic behavior series, the angle and RCS energy as the relevant factors to build the GRA model. From the analysis results shown in The variations of the correlation coefficients are shown in From the results obtained, as the target vehicles move further away from the radar, the measuring distance increases and the correlation coefficients of both the angle error and the RCS energy show an overall downward trend. This indicates that the radar data detected at close range is highly reliable. In addition, by Equation (22), the grey relational degree of RCS energy is 0.72 and the value of angle error is 0.54. Compared with the angle measurement, the RCS energy distribution presents a much steady trend as the distance increases. The angle accuracy is affected by the antenna performance. For the non-scanning radar which we used in this study, the angle measurement exists relatively high and having random error, presenting a weaker correlation behavior.Based on the application scenario, the effective detecting region of the millimeter-wave radar is calculated and further the data sampling characteristics are analyzed in the region. At a certain close range, the sampling is dense and contains more useful information for further research.Subject to the antenna performance, there are obvious errors for angle measurement and the errors show certain random characteristics. Hence, the angle information cannot be used alone for further applications, such as lane recognition. In addition, when the RCS energy is high, the angle error is relatively low, which provides a method to combine the RCS energy and the angle information together for corresponding applications.The RCS energy presents a downward trend as the vehicle moves away from the radar. For different types of vehicles, the RCS energy distributions have certain differences, which can be used for identifying small or big vehicle targets. In this paper, a simple method for target type identification is also proposed.Combining with the second item mentioned above, from the point of view of the overall sampling series, the RCS energy presents a much higher correlation degree compared to the angle as the distance increases.In this paper, the data features are analyzed for the non-scanning multi target millimeter-wave radar used in traffic flow detection, with single parameter sampling, multiple parameters and parameter series correlation analysis. The main conclusions are summarized as follows:The data feature analysis for stationary vehicles using the stationary target detection function of the FMCW millimeter-wave radar.Further traffic status identification research based on the data features acquired, for example, multi-target trajectories tracking, traffic volume calculation and parking event judgment.These conclusions can provide effective reference for the further applications where millimeter-wave radar is used in the traffic flow detection. Based on the analysis, the future work mainly focuses on the following aspects:


Drivers’ behaviors and decision making on the road directly affect the safety of themselves, other drivers, and pedestrians. However, as distinct entities, people cannot predict the motions of surrounding vehicles and they have difficulty in performing safe reactionary driving maneuvers in a short time period. To overcome the limitations of making an immediate prediction, in this work, we propose a two-stage data-driven approach: classifying driving patterns of on-road surrounding vehicles using the Gaussian mixture models (GMM); and predicting vehicles’ short-term lateral motions  based on real-world vehicle mobility data, provided by the U.S. Department of Transportation, with different ensemble decision trees. We considered several important kinetic features and higher order kinematic variables. The research results of our proposed approach demonstrate the effectiveness of pattern classification and on-road lateral motion prediction. This methodology framework has the potential to be incorporated into current data-driven collision warning systems, to enable more practical on-road preprocessing in intelligent vehicles, and to be applied in autopilot-driving scenarios. According to the latest survey about causes of motor vehicle crashes in the United States , nearly With the rapid development of data-driven intelligent transportation system , there iTherefore, we propose a two-stage, data-driven approach to predict the lateral motions of the preceding vehicles. The Gaussian Mixture Model (GMM) is used to classify the driving behavior into erratic and consistent. This result is then utilized to bias the lateral motion predictions made by two types of ensemble decision trees for more safety. To implement the proposed methodology, we utilize real-world vehicle mobility data acquired from Ann Arbor, Michigan, provided by the U.S. Department of Transportation (USDOT) . The proThis paper is organized as follows: In Currently, there are three types of models for vehicle motion prediction: physics-based motion, maneuver-based motion, and interaction-aware motion predictions . The phyNonetheless, most of these predictions have taken driver behavior as inputs, and many predictions are made based on either mathematical modeling or an isolated lab created scenario instead of using real-world context. While mathematical models can provide the most complete data, they have the limitation of modeling. Regardless of the sophistication of the model created, the real-world or real-human behavior is more complex. Because of the inevitable assumptions made there will be some information lost during the model construction. Lab experiments overcome this as they collect data of real human drivers. However, they are limited due to simulation tools and an isolated environment. The designated vehicles and lack of diversity of the drivers can also easily bias the data acquired. Therefore, data collected in a real-world context from a diverse vehicle and driver set are more representative for driving behavior studies.A maneuver is a procedure or method of working that involves expert physical movement, and it is assumed that a vehicle’s future motion will match the intended maneuver of a driver . TherefoWe are particularly interested in the real-time on-road vehicle lateral motion prediction. Although significant efforts have been made by building mathematical models and conducting laboratory experiments ,15,16, tAmong the proposed statistical methods, a few data-driven approaches have been applied in classifying driving patterns (e.g. car-following behaviors ) and preLiebner et al., in 2013, developed a prediction approach for lateral motion  at urban intersections with and without the presence of preceding vehicles . The stuOverall, in The dataset utilized in this paper is generated based on the safety pilot model deployment (SPMD) program . As a paThe SPMD dataset was composed of four parts: driving dataset, basic safety message (BSM) dataset, roadside equipment (RSE) dataset, and contextual data dataset. The driving dataset contained the vehicles’ mobility data collected by two different sets of data acquisition devices developed by the University of Michigan Transportation Research Institute (UMTRI) and the Virginia Tech Transportation Institute (VTTI), respectively. These mobility data were time-stamped and included the acquisition device ID, which was unique for each vehicle. The trips of the monitored vehicles were also identified by the status of ignition, meaning that one trip started when the ignition was on and ended when it was off. The system recorded the vehicle’s motion data, such as GPS location (longitude and latitude), speed, acceleration, and body angle change; hardware status data, such as brake status and turn signals statuses; road environment data, such as distances to the lane lines, distances and speed to surrounding objectives, and whether the vehicle had crossed the lane lines or not.In this paper, we selected the dataset collected by the VTTI devices due to the consistency of the vehicles and their detailed spatial-temporal records. The dataset was composed of two major parts: the vehicle primary mobility data and the radar data. The primary mobility data contained motion and on-vehicle devices status records of the concerned vehicles, including 64 devices/vehicles and 14,315 trips in 61 days. The detailed data features can be found in [To process instances of lateral motion, the time series data have to be converted to features and labels to be acceptable to the regular data mining methods. The conversion is not meant to bring any value change to the original data. The operations involved are basically reshaping and new feature creation based on the original data. In order to do this, we found all the time tags where a lane change started to occur. We then calculated the lateral motion duration and considered a minimum time between lateral motion events in order to filter out potential mislabeling of the same lateral motion event as multiple events. For each event’s features, five-second windows (50 times tamps) were used before the turn. Negative labels were then created by chunking the rest of the data into five-second windows. Since we aimed to predict the lateral motions of the vehicle ahead and assist the driver in making a correct judgment, there should be some reaction buffer time left for the driver or the autonomous driving system. Two buffer times, Given the nature of the driving, this dataset is heavily skewed toward non-turning events. Based on our definition of the lateral motion events, the ratio for left turns in the whole dataset is only 1.54% and that for the right turns is 1.49%. Even though the appearances of both events are rare, the rates are close, which makes common sense. In the following methodology section, multiple technics will be discussed about eliminating the negative influence of this dataset unbalance between a large number of no turns and a dramatically small quantity of turning events as compared with no turns.Summary statistics are widely utilized in existing research ,6 for anSince there are The fast Fourier transform (FFT) can transfer the data sequence from the time domain to the frequency domain in order to reveal the rate of recurrence characteristics of the sequence. Zhang et al., in 2010, utilized the FFT data to recognize the driving patterns of the drivers in , and setWhen a driver starts to make the attempt to turn or shift lane, they tend to manipulate the vehicle more. This makes the vehicle’s motion status and devices’ statuses vary more frequently than during normal driving . Using the FFT method, we can analyze the temporal data sequence of each element in the data chunk, find out the frequency with the highest FFT coefficient, and observe the variation of said frequency which gives some indication of changing driving status. These frequencies of the elements are calculated and concatenated to the reshaped data vector as new features are created. Both groups of the created features, summary statistics, and FFT frequencies can be quickly calculated. The limited computing resources needed will allow data processing while driving.In order to make the application easy to be deployed in vehicles and increase the raw data availability, we focused on kinetic measurements, such as vehicle acceleration, and excluded several features that are either irrelevant, such as the number of GPS satellites and the headlight status, or directly indicated the turning action, such as crossing the left lane track and crossing the right lane track. The selected features are described in To illustrate the kinetic characteristics of the observations, we also included the first and the second derivatives of the “longitudinal acceleration” with respect to time as well as the speed and the acceleration of the vehicle to the left and right markers are also included. Take the jerk  as an example. Using Thus, there are 15 basic features. We included summary statistics, as well as FFT analysis of these features as other features. Here we used 5 s as the total time window for each lateral motion event. With The importance of prediction features was evaluated with random forest (RF)  and its Taking the feature vector and concatenating it back into the matrix form with one feature across time per column, we can average the error increase in each column to determine which measurements are the most important for lateral motion. As shown in A two-stage data-driven method was designed to predict the lateral motion of the preceding vehicle based on the processed dataset in the previous section. The steps are shown in To be more specific, in this paper, we used one-day data of a two-month dataset to be the training dataset. Our training set contained mobility data of 51 unique vehicles within 587,812 time instances (16.33 h of driving)  and one Considering the skewness nature of the dataset, we conducted multiple experiments based on different classification methods (three times for each method), including the supporting vector machine (SVM), basic decision tree (DT), and several ensemble decision trees (EDT). The bagging decision trees and the RUS boosted decision trees demonstrated better performance on the cross-validations.The term “bagging” is an acronym of “bootstrap aggregating” . The leaWithin the bagging decision trees algorithm, bootstrap samples The adaptive synthetic (ADASYN) sampling approach  was utilIn this table, false negative rates are more than 43% for Turn Left and around 59% for Turn Right which was due to the high disparity between classes. On the other hand, the bagging trees ensemble tended to bias more conservatively towards a no turn prediction.The RUS boosted decision trees were utilized to resolve skewness of the training data. The idea was to down-sample the training dataset using RUS so that the amount of the minority class data was comparable to that of the majority class data. The ratio of the quantities of different classes was predefined to make the observations of the three classes equal to each other. The weak learners we used were normal decision trees. For each iteration, the under-sampling was conducted based on the predefined ratio, and the weight for each instance of the sampled training dataset was calculated. The sampled dataset and the corresponding weights were used to train one weak learner. According to the error, the weight for the current weak learner was computed and the weights of each instance were adjusted based on it. After predefined times of iteration, multiple weak learners were generated based on training datasets that had more class-balanced data. For each testing instance, the weighted voting of each weak learner classification results was the prediction. The RUS boosted trees ten-fold cross-validation results are shown in The ensemble decision trees algorithms expressed two opposite biases on the prediction results. We observed such biases in several different learning schemes. To secure safer trips, we utilized these biases with predictions on more erratic drivers given that they have more of a tendency to change current driving status. Since the bagged trees tended to provide more conservative predictions, i.e., predicted a lack of lateral motion when it did occur , it reduced risk if we chose to cast less trust on its results when facing more erratic drivers. At the same time, since the RUS boosted trees tended to provide more aggressive predictions, i.e., falsely predicting turns , it reduced risk if we choose to cast more trust on its results when facing front drivers driving with more erratic manners.Multiple well-developed studies have been conducted to cluster the behaviors of drivers based on their aggressiveness. Kanarachos et al., in 2018, summarized the features used to implement the clustering, including acceleration and smoothness (variance of acceleration) . The harAs observed in the training dataset, the mobility data showed a fairly clear boundary between the two groups of drivers. We took data of two features, the mean of longitudinal acceleration and the mean of the jerk, as an example and showed the two-dimensional plot of them in In order to combine the two algorithms’ ascendancy on the algorithm, we utilized GMM to cluster the driver into one of the two clusters. The clusters of GMM are represented by two different Gaussian distributions characterized by distinct expectations and variances. Here we use The clustering results based on the training samples are listed in The algorithm can provide two probabilities, i.e., The flow chart of the two-stage method is shown in The remaining 60-day observation data in the aforementioned two-month dataset was used as test data for the demonstration of the effectiveness of the proposed prediction algorithm. The dataset collected on April 11, 2013 and used as training data in the previous methodology was excluded. After the training process, 40% of the data were used for the model test. The prediction results of the trained bagging trees are shown in Using the proposed GMM clustering integrated algorithm, the prediction results are shown in As demonstrated in For comparison, the observations with the 0.1 s prediction time were also generated. The results of the implementation based on this dataset are listed in the following tables. The prediction results of the trained bagging trees are shown in Using the proposed GMM clustering integrated algorithm, the prediction results are shown in In addition, we observed skewness with the 0.5 s prediction time. It seems that the prediction on the motion towards the right is more accurate than the others in terms of the false-negative rate. While with the corresponding false-positive cases considered, the trained decisions trees tend to provide more “turn right” predictions. The reason could be that in the training dataset, there are more “turn right” events. Overall, the high accuracy of the prediction results indicates that the proposed algorithm is effective and the idea of predicting the motion of other vehicles based on their accessible mobility data is realistic. Though some of the accuracy parameters can reach a level of more than 95%, the absolute numbers of the misclassifications are still to be decreased. This problem could be resolved with larger and more diverse training dataset, more detailed clusters, and even shorter prediction time. Because the functionality can be used in the autonomous driving vehicles, the computing capability of the vehicle-installed computers and processing capability of the corresponding software systems can tolerate much shorter reaction time than 0.1 s. The deployment of the proposed algorithm on such vehicles can downgrade much on-road uncertainty and provide more safety for the passengers. In this paper, we proposed an algorithm to predict the preceding vehicles’ lateral motions with their real-time mobility data considering the classification of drivers’ behaviors. The lateral motion prediction functionality can serve as part of the autonomous vehicle driving systems or other driver assistance software. The data-drive prediction technique includes two parts: drivers’ behaviors clustering and ensembled decision trees (bagging and RUS boosted) prediction. We validated the technique with the real-world vehicle mobility data from the SPMD database and the random forest feature selection. We also used summary statistics and the FFT to process the time series data. The prediction results on the real-world vehicle data demonstrate the effectiveness of the algorithm and its potential to be utilized in on-road vehicles. With more available complete and diverse real-world vehicle mobility data, the implementation of the prediction system can have higher accuracy and thus provide the driving assistant systems with more useful driving information. In the context of the emerging autonomous driving technique, it is expected to experience a time period when both autonomous driving systems and human drivers coexist. The application of the proposed prediction technique can decrease the uncertainty brought by human drivers in preceding cars, which can improve the safety level of drivers and passengers in vehicles as well as pedestrians. The designed approach also contributes to the development of highly automated transportation systems as well as building intelligent and smart cities.


The loss of a hand is a traumatic experience that substantially compromises an individual’s capability to interact with his environment. The myoelectric prostheses are state-of-the-art (SoA) functional replacements for the lost limbs. Their overall mechanical design and dexterity have improved over the last few decades, but the users have not been able to fully exploit these advances because of the lack of effective and intuitive control. Bimanual tasks are particularly challenging for an amputee since prosthesis control needs to be coordinated with the movement of the sound limb. So far, the bimanual activities have been often neglected by the prosthetic research community.We present a novel method to prosthesis control, which uses a semi-autonomous approach in order to simplify bimanual interactions. The approach supplements the commercial SoA two-channel myoelectric control with two additional sensors. Two inertial measurement units were attached to the prosthesis and the sound hand to detect the movement of both limbs. Once a bimanual interaction is detected, the system mimics the coordination strategies of able-bodied subjects to automatically adjust the prosthesis wrist rotation  and grip type  to assist the sound hand during a bimanual task. The system has been evaluated in eight able-bodied subjects performing functional uni- and bi-manual tasks using the novel method and SoA two-channel myocontrol. The outcome measures were time to accomplish the task, semi-autonomous system misclassification rate, subjective rating of intuitiveness, and perceived workload (NASA TLX).The results demonstrated that the novel control interface substantially outperformed the SoA myoelectric control. While using the semi-autonomous control the time to accomplish the task and the perceived workload decreased for 25 and 27%, respectively, while the subjects rated the system as more intuitive then SoA myocontrol.The novel system uses minimal additional hardware  and simple processing and it is therefore convenient for practical implementation. By using the proposed control scheme, the prosthesis assists the user’s sound hand in performing bimanual interactions while decreasing cognitive burden. The human hands are essential tools for many activities of daily living (ADL). They are capable of dexterous yet reliable manipulation, firm grasping, and are instrumental for haptic exploration of the environment and social communication. Unfortunately, hand amputations are estimated to occur 18,496 times each year and an estimated total of 541,000 humans are affected by upper limb amputations in the United States alone [Modern active prostheses are typically controlled using electromyography (EMG) signals . The surMost studies in prosthesis control, either manual or semi-autonomous, focus on the unilateral tasks, where a prosthesis is used to grasp and manipulate an object, without participation of the sound hand. However, many important activities of daily life are normally employed bimanually, for example, picking up a larger object or transferring an object from hand to hand. Able-bodied subjects accomplish these tasks routinely by exploiting the dexterity available in both limbs. Both hands actively collaborate while performing the task. The amputees, on the other side, chose to compensate the bodily deficit by over-employing the unaffected side , 24. In Importantly, when performing bimanual interactions with an object, in many cases the two hands move in a coordinated and stereotyped manner Fig. . For exaThe fact that the hands move in consistent patterns during bimanual tasks makes this an ideal context for implementing semi-automatic assistance to prosthesis control. In this study, we present a novel system for semi-autonomous bimanual prosthesis control (BPC). The system to automatically adjusts prosthesis wrist rotation  and grasp type  by classifying hand interactions either as unilateral, synchronous-bimanual or asynchronous-bimanual. The BPC was evaluated experimentally using functionally relevant tasks, and the results demonstrate that it substantially enhances the execution performance of bimanual interactions while using a myoelectric prosthesis.The BPC prototype consists of the following components Fig. a: (1) twMichelangelo prosthesis is an advanced, commercially available prosthesis that can perform two grip types with relatively high force (70–100 N) and actively rotate the wrist . MoreoveThe BPC system consists of 1) an autonomous controller  the system has recognized UNI interaction, 2) the system has recognized BI-ASYNC or 3) the BI-SYNC interaction and 4) the system is in the idle state – the system has detected that an interaction has ended (object released) and is ready to detect another one. These events are coded using single or double vibration burst as depicted in Fig. The BPC was compared to the conventional, two-channel myocontrol . Therefore, the myocontrol algorithm used for comparison to the BPC was identical to the one implemented within the BPC system. The experimental evaluation addressed four different interaction scenarios: unimanual interactions using the sound hand, unimanual interactions using the prosthesis and two variations of bimanual interactions (synchronous and asynchronous).The study was approved by the Ethics committee of the University of Göttingen . All experiments were conducted in accordance with the declaration of Helsinki, and all subjects signed a written informed consent prior to participation in the experiments. Eight able-bodied subjects were recruited for the study . Two out of eight subjects were naïve to myoelectric prosthesis control. Each subject performed the experiment in two conditions: once using the SoA myoelectric control and once using the BPC.The subject performed the experimental task standing in front of a tall closet with four equidistantly placed shelves Fig. a. SubjecThe overall task was to grasp and move objects, one by one, from one shelf to another. Four objects were used in the experiment Fig. b: a box,One experimental trial consisted of seven interactions Table  with fouThe two experimental conditions  were evaluated on separate days (sessions) in a randomized order. A session was divided in three phases: 1) introduction and system setup, 2) running the experiment and 3) filling out the questionnaires. Upon performing both experimental conditions, we performed an additional pilot test on three study participants in which they executed the experimental tasks without the prosthesis, using their own hands. This test was performed to obtain a reference performance for task execution in able-bodied subjects.A session started with a general introduction to the prosthesis operation and overall experimental task. Since both experimental conditions rely on myoelectric control, the myocontrol scheme was explained at the beginning. In the BPC condition, the semi-autonomous control and feedback were explained in the context of the two types of bilateral interactions. The objects were presented and experimental task explained Table . Next, tBefore starting the experiments, the subjects were instructed that the overall goal of each trial is to perform the object interactions Table  as fast At the end of each session, the subject answered the NASA task load index (TLX) questionnaire , which mp < 0.05. The outcome of each performance metric was presented as the median (interquartile range).In order to avoid influence of the learning curve on the performance metrics, only the last three trials of each block were used for data analysis, hence 18 trials per each control condition . The data was analysed by calculating the average performance of a subject across all included trials. Since the data did not pass the normality test (Lilliefors test), the Wilcoxon signed-rank test was applied to assess significant differences between the conditions. The statistical significance threshold was set to The following outcome measures were used to compare the two control systems: 1) the average time the subject needed to complete an experimental trial; 2) the number of dropped objects; 3) perceived workload; 4) perceived intuitiveness of the control system. In addition, the accuracy of movement classification was evaluated for the BPC system. The experimenter monitored the task execution during the experiment and noted each occurrence when the BPC made a classification error, i.e., the system decision did not match the movement class indicated in Table The subjects were significantly faster when performing the experimental task Fig. a with thThe BPC system correctly classified the limb movements in 95% of the 1008 interactions. The total misclassification rate  s which was five and four time faster than in SoA and BPC condition, respectively.We have developed a semi-autonomous system for bimanual prosthesis control (BPC) and compared it to the commercial SoA myocontrol in eight able-bodied subjects performing a variety of unilateral and bilateral object interactions. The results demonstrated that BPC substantially improved the interactions between the sound hand and a prosthesis by significantly decreasing the time to accomplish the task and perceived workload. In addition, the use of BPC improved the performance consistency across subjects, as evidenced by the lower variability of the time to accomplish the task Fig. . It is kThe subjects also experienced the system as more intuitive than the SoA myocontrol. This is likely because the BPC allowed the subjects to perform bimanual tasks smoothly, mimicking thereby natural movements. When using myoelectric control, the subjects were forced to make breaks to activate the prosthesis functions as required by the task (wrist rotation and grasp type). In BPC, the prosthesis parameters were configured automatically, and the subjects only needed to move and position the prosthesis. Importantly, the parameters were adjusted using natural strategies that characterize the coordination of sound hands during bimanual tasks of able-bodied subjects. Therefore, the prosthetic hand behaved as intuitively expected by the subjects.The simple method that has been used to detect and classify unimanual and bimanual movements showed to be very effective, resulting in a low rate of misclassifications while being tested in a range of functional tasks. This is important as the algorithm can be easily implemented in an embedded prosthesis controller. The method has to be calibrated individually for each subject, but this calibration was simple and fast (less than 20 min). Importantly, the system provided the subjects with vibrotactile feedback communicating the results of movement detection and classification. Therefore, the subjects were explicitly notified in case of misclassification, so that they could take over the control (semi-autonomous system) and prevent the unexpected behaviour of the hand. The subjects found this feedback useful but not necessarily required for the system operation. The latter is likely due to the high reliability of classification in the first place. If the classification has been less reliable, the feedback would be likely experienced as more critical by the subjects. Overall, the subjective estimations of the feedback utility were highly variable, indicating that the approach to feedback was subject specific.Although the presented experimental tasks could be considered simple from the able-bodied perspective their relevance is expressed in the very fact that they are still quite difficult to perform using classical (myoelectric) control schemes. Namely, using their own hands the subjects performed the task five and four times faster than in SoA and BPC condition, respectively. Therefore, similar to observations made in , 36 the The BPC uses only two IMU sensors but it is not self-contained, as one of the sensors needs to be placed on the sound hand. However, this is not an important drawback because wearable sensing is becoming more and more pervasive, and the contralateral IMU could be exchanged with a smart watch, fitness brace or a similar gadget. The system presented in this study supports some important types of bimanual tasks, but not all of them, and it is yet to be developed into a solution that can be used robustly in a clinical environment. For example, in some bimanual activities the spatial coupling between the hands will be neither symmetric nor anti-symmetric, as when grasping a jar by the side and a lid from the top in order to open/close the jar. Therefore, the BPC needs to be expanded with additional strategies and scenarios in order to support a greater variety of activities and movements . To this aim, a future system could be combined with machine learning  or with The BPC is one of the first systems that addressed the bimanual ADLs, which were until now largely neglected in literature. A prosthesis that actively supports bimanual activities could motivate the users to utilize the prosthesis in a larger set of tasks, thereby increasing the overall utility of the device and decreasing the rejection rates that are still very high , 28.During ADLs, the majority of manipulation tasks require collaboration between two hands in which both limbs should be coordinated in space and time , 29. How


Semantic segmentation and depth estimation are two important tasks in computer vision, and many methods have been developed to tackle them. Commonly these two tasks are addressed independently, but recently the idea of merging these two problems into a sole framework has been studied under the assumption that integrating two highly correlated tasks may benefit each other to improve the estimation accuracy. In this paper, depth estimation and semantic segmentation are jointly addressed using a single RGB input image under a unified convolutional neural network. We analyze two different architectures to evaluate which features are more relevant when shared by the two tasks and which features should be kept separated to achieve a mutual improvement. Likewise, our approaches are evaluated under two different scenarios designed to review our results versus single-task and multi-task methods. Qualitative and quantitative experiments demonstrate that the performance of our methodology outperforms the state of the art on single-task approaches, while obtaining competitive results compared with other multi-task methods. Semantic segmentation and depth information are intrinsically related, and both pieces of information need to be considered in an integrated manner to succeed in challenging applications, such as robotics  or autonOn the other hand, deep learning techniques have shown extraordinary success in both tasks  in recenIn this paper, we introduce a hybrid convolutional network that integrates depth estimation and semantic segmentation into a unified framework. We propose to build a model where the features extracted are suitable for both tasks, thus leading to an improved accuracy in the estimated information. One of the main advantages of the proposed approach is the straightforward manner semantic segmentation and depth map are estimated from a single image, providing a feasible solution to these problems.Depth estimation and semantic segmentation are two widely studied problems in the image processing community and recently have been tackled through deep learning techniques due to its successful results in terms of accuracy and efficiency. This section makes a review of the state of the art introducing, first, single-task approaches, and, afterwards, methods focused on solving multiple tasks.Semantic segmentation is a methodology which approaches the image segmentation problem by performing pixel-level classifications. Compared to the traditional image segmentation approaches, such as superpixel segmentation methods ,5, activBefore CNN-based techniques were applied to semantic segmentation; handcrafted features were usually employed to represent pixels when training the classifier . The emeApart from fully connected layers, one of the main problems of using CNNs for segmentation are the pooling layers. Pooling layers increase the field of view and are able to aggregate the context while discarding the ‘where’ information. However, semantic segmentation requires the exact alignment of class maps and thus, needs the ‘where’ information to be preserved. Two different classes of architectures evolved in the literature to tackle this issue.The first one is the encoder-decoder architecture. The encoder gradually reduces the spatial dimension with pooling layers and the decoder gradually recovers the object details and spatial dimension. There are usually shortcut connections from the encoder to the decoder to help the decoder recover the object details better. U-Net  is a popn times larger receptive field if the full resolution image is convolved with a filter ‘with holes’, in which the original filter is upsampled by a factor n, and zeros are introduced in between filter values. Although the effective filter size increases, it is only necessary to take into account the non-zero filter values, hence both the number of filter parameters and the number of operations per position stay constant.Architectures in the second class use what are known as dilated/atrous convolutions ,16,17. POne of the first works to tackle the depth estimation problem using CNNs is the one presented in . They usLikewise, a similar work based on  is preseApproaches addressing depth estimation and semantic segmentation with multi-task learning schemes are receiving large attention due to its potential of improving the performance of multiple tasks. The idea of merging tasks in one architecture is motivated by the fact that different correlated tasks commonly share some basic attributes in the parsing process. Approaching them together may be mutually beneficial. In practice, multi-task approaches in the state of the art seek to extract features suitable to perform diverse tasks at a time, which lead to an improvement in both estimated information and simplification of systems where multiple modalities are required, such as autonomous navigation , roboticIn , the autIn  a unifieA more recent multi-task approach is introduced in . The metAnother multi-task approach by Teichmann et al.  presentsA similar approach is Pixel-Level Encoding and Depth Layering (PLEDL) , this woMulti-task approaches aim to directly estimate the segmentation and depth maps from an input color image by unifying CNNs working for a single task into a sole hybrid convolutional neural network. Most of the state-of-the-art works unify tasks under a feature-extraction block whose output becomes the input of a group of decoders designed to carry out each task.In our preliminary work , we presIn this paper, we focus on comparing different hybrid network unifying strategies and investigating how those two tasks help each other. More specifically, we employ two unifying strategies, one from the hybrid architecture proposed in our previous work  and the The rest of the paper is organized as follows: in In this section, a general explanation of our hybrid convolutional neural network and its application to depth estimation and semantic segmentation is presented. To this end, a description about the two single-task architectures ,19 emplox and y direction computed from the depth map. Along with the global depth network and gradient network, the refining network is again trained on the downsampled depth maps in the training data.The depth estimation architecture , denotedThere are two main reasons to consider employing DepthNet as the depth estimation component in our approach: (1) DepthNet follows the state-of-the-art framework for depth estimation which is representative for a bunch of methods. (2) DepthNet has a modularized architecture, which allows us to analyze each of the components in it and better integrate DepthNet into a hybrid architecture.The semantic segmentation architecture  shown inDeepLab-ASPP is employed as the semantic segmentation component in our approach due to its outstanding performance in this task.Considering the functionality of each component in DepthNet and DeepLab-ASPP, we propose and compare two different hybrid architectures for the joint depth estimation and semantic segmentation task.Architecture 1: An intuitive way to unify two tasks in a sole architecture is to totally share the feature-extraction process for both tasks. It follows the idea from the most representative architectures in the state of the art [ the art ,23,24, i the art  as the fArchitecture 2: The motivation of building this architecture is to further clarify the common and specific attributes in the two tasks. Thus, we build the hybrid network by substituting the gradient network in DepthNet by a common feature-extraction network for the two tasks, while keeping the global depth estimation network only for the depth task. The advantages of this hybrid architecture are two-fold. On one hand, the strong power of extracting object information from a color image learned in the semantic segmentation task can also benefit depth layering when predicting a depth map, while the strong power of extracting rich depth boundaries from a color image learned in the depth estimation task is shared in the semantic segmentation task to improve segmentation accuracy on object boundaries. On the other hand, the global layout of the scene which is more relevant in depth estimation than in semantic segmentation is estimated independently by a global network in the depth estimation task. This avoids interfering the common feature extraction for both tasks. In practice, we keep the global network and refining network in DepthNet without changes, while replacing the structure of the gradient network with the VGG structure, in order to keep the structure consistent with the feature network in DeepLab-ASPP. We denote the architecture 2 as HybridNet A2 in the rest of this paper.Since the proposed architectures are assembled with basic components in the two single-task architectures, we explain the detail of the proposed architectures by describing the two single-task architectures in this section.As described in The gradient network aims at estimating depth gradient from an input color image. In practice, we employ the same architecture used in global depth estimation except for the fully connected layers.Finally, the refining network takes the concatenation of the output from the global depth network, the gradient network, and the color image as input and computes a refined depth map. The refining network improves the rough estimate from the global depth network, using gradients estimated by the gradient network and an input color image. In practice, the first convolutional layer processes the input color image, followed by a ReLU layer, an LRN layer, and a max-pooling layer, which produces the feature maps extracted from the color image. These feature maps are concatenated with outputs of the global depth network and the gradient network, then are fed to the remaining four convolutional layers. Each of them is followed by a ReLU layer. The output from the 5th convolutional layer in the refining network is treated as the output (a refined depth map with size 81 by 81).The feature network contains 5 groups of convolutional layers, forming a deep architecture. All these convolutional layers have the same kernel size On the other hand, the atrous upsampling network contains 4 parallel groups of three convolutional layers, to perform upsampling operation at different scales. Each branch upsamples the output from the feature network at the first convolutional layer with an atrous convolution. An atrous convolution employs a dilated convolution template, in which a convolution template is enlarged by filling zeros with respect to a defined rate. In this manner, we can explicitly control the resolution of the upsampling operation and enlarge the field of view of filters to incorporate larger context in the semantic segmentation task without introducing more parameters. In practice, we employ atrous convolutions with rates As explained in the In HybridNet A1, we initialize the feature network and the atrous upsampling network with the model provided by DeepLab  which waIn HybridNet A2, we initialize the feature network and upsampling network before the training process using again the model provided by DeepLab . AdditioOnce we have the initialization for our hybrid architecture, all of its components are trained simultaneously. Both hybrid architectures are trained for 100 K iterations with a learning rate The loss function used in both architectures is the same. For the semantic segmentation task, The hybrid loss function We quantify the performance of the proposed architectures on both semantic segmentation and depth estimation in different scenes using our Caffe implementation. We first evaluate the proposed architectures in road scenes which is of current practical interest for various autonomous driving related problems. Secondly, the proposed architectures are evaluated in indoor scenes which is of immediate interest to possible augmented reality (AR) applications.In this section, we present the evaluation of the proposed architectures in road scenes. Several road scene datasets are available for semantic parsing ,31,32. SIn practice, the training process of our approach was performed using the 2975 images of Cityscapes training set that provides a depth map and object labels of 19 classes for each RGB image. To evaluate the performance of the proposed architectures, we group the 500 images of the validation set and the 1525 images of the test set in the Cityscapes dataset into a single evaluation set of 2025 images. In the training phase, images in the training set are shuffled and randomly cropped to fit the input image size of the hybrid architecture. Training data augmentation is done by flipping and mirroring the original images, to enlarge the training set. In the testing phase, we crop the test image with the original size of Our first aim is to determine if the features obtained in the shared part of the proposed architectures solving the two tasks simultaneously provide better results than the ones that we would obtain using two identical networks trained separately. This is why, in addition to the results of the proposed architectures, we present the results obtained by the models that solve these two tasks separately for comparison. The models used to train semantic segmentation and depth estimation independently are denoted as DeepLab-ASPP  and DeptIn addition to qualitative results, we employ three commonly used metrics, to measure quantitatively the segmentation performance: the global accuracy (G), the class average accuracy (C) and mean intersection over union (mIoU). The global accuracy counts the percentage of pixels which are correctly labeled with respect to the ground truth labeling. The class average accuracy is the mean of the pixel accuracy in each class. The mean intersection over union measures the average Jaccard scores over all classes. The improvements obtained by our method against DeepLab-ASPP confirm the hypothesis that sharing the feature-extraction network between tasks leads to an improvement in terms of segmentation accuracy. The strategy of unifying two single-task architectures affects the segmentation performance of hybrid methods. HybridNet A2 where common and specific attributes between two different tasks are better clarified outperforms HybridNet A1 in which the feature-extraction process is totally shared for the two tasks. The improvement that HybridNet A1 obtains against DeepLab-ASPP is very limited (HybridNet A1 58.1% mIoU against DeepLab-ASPP 58.02% mIoU); however, Hybrid A2 improves the mIoU by around d method  in Tabled method . For addd method ,13,16,33For depth estimation evaluation, Additionally to qualitative analysis, we evaluate the performance of our methodology for depth estimation employing 6 commonly used metrics: Percentage of Pixel (PP), Mean Variance Normalized Percentage of Pixel (PP-MVN), Absolute Relative Difference (ARD), Square Relative Difference (SRD), Linear Root Mean Square Error (RMSE-linear), Log Root Mean Square Error (RMSE-log) and Scale-Invariant Error (SIE).d and N stands for the number of pixels with valid depth value in the ground truth depth map.In the quantitative experiment, we compare the proposed hybrid architectures and DepthNet. Road scene images have relatively limited variation in terms of the involved semantics and their spatial arrangements. They are usually captured by a camera fixed on a moving vehicle where the view direction of the camera is always parallel to the ground. This limits the variability of road scene images and makes it easier for the convolutional networks to learn to segment them robustly. In comparison, images of indoor scenes are more complex due to the free view point, the larger number of semantics in the scene, widely varying sizes of objects and their various spatial arrangements. On the other hand, although indoor scenes have smaller depth range than road scenes, they usually have more complex spatial layout, which provides challenges for depth estimation.In this section, we evaluate the proposed architectures on indoor scene data for both semantic segmentation and depth estimation. We employ RGB-D Scene Understanding Benchmark dataset  . To makIn this paper, we have introduced a methodology for depth estimation and semantic segmentation from a single image using a unified convolutional network. The main goal of the proposed method is to seek for a better hybrid architecture of CNNs that modularizes the feature-extraction process by separating it into distinct feature extraction for a specific task and common feature extraction for both tasks. In this manner, both tasks can benefit from the extracted common features without being affected by those features only relevant to one task, which leads to a better performance. We also prove that solving correlated tasks such as semantic segmentation and depth estimation together can improve the performance of methods tackling the tasks separately.The qualitative and quantitative results shown in Designing better loss functions for a multi-task learning scheme. The loss function employed in the state-of-the-art approaches is normally a balanced linear combination of losses for single tasks. However, these losses may have totally different physical meaning regarding the tasks , which makes it hard to combine them. Finding higher level evaluation metrics helps define the loss function for a multi-task learning system. For instance, evaluating on the prediction of the 3D oriented bounding box of objects requires using both semantic segmentation and depth estimation result, which naturally combines the loss function for both tasks.Applying to higher level tasks requiring 3D analysis. Since the proposed approach produces an object level segmentation and a depth map of an input image, applying the estimated result to applications requiring 3D analysis (such as traffic violation detection) will be of great interest.On the other hand, there are also some interesting problems pending for a future study:


We present a novel calibration method for a multi-view laser Doppler speed sensing (MLDSS) system. In contrast with the traditional method where only the laser geometry is independently calibrated, the proposed method simultaneously optimizes all the laser parameters and directly associates the parameters with a motion sensing model. By jointly considering the consistency among laser Doppler velocimetry, the laser geometry and a visual marker tracking system, the proposed calibration method further boosts the accuracy of MLDSS. We analyzed the factors influencing the precision, and quantitatively evaluated the efficiency of the proposed method on several data sets. Multiview laser Doppler speed sensing (MLDSS)  uses sevIt is important to mention that the precision of MLDSS strongly relies on the sensing accuracy of the laser Doppler velocimeter (LDV), the system-layout parameters, and calibration precision. Factors related to the LDV sensing, such as speckle noise ,10 and lA similar calibration principle can be commonly found in the literature for the calibration of galvanoscopic laser systems using various types of sensors ,14,15. IThe inspiration for our approach is derived from the calibration techniques of optical systems and computer vision, in which simultaneously calibrating all elements is rather beneficial in terms of both simplicity and accuracy ,17. It iIn this section, we briefly describe the basic kinematic formulation in MLDSS  and paraWe discuss a right-handed co-ordinate system with a fixed origin MLDSS utilizes a simple kinematic model to reconstruct the speed of a target rigid body. When a laser-beam ray hits the target, the one-dimensional velocity along this ray is measured by the Doppler effect. Let the By combining multiple measurements with the form in Equation , the 6-DA to denote the leftmost matrix, and A fully describes the system, but the dimension of the parameter space is smaller than the number of matrix elements. Consider the A, P consists of 24 elements.We use Equation . Matrix P consists of only the geometrical parameters of the lasers. We can directly calculate each laser’s spatial position, i.e., Note that We first briefly describe this geometric-only calibration method ,14,15. AA into a ground-truth matrix Apparently, this method has geometrical errors from sources such as the image processing error and lens distortion. Here, we go one step farther to see the error of motion reconstruction. Consider Equation . We can The intuition here, as we claimed in P that generate the most accurate 6-DOF speed estimation Hence, in this paper, the calibration of MLDSS is described as how to find the best set of parameters k. It is denoted by To simplify the discussion, we divided motion sensing into discrete frames. First, assume that we know the ground-truth position of the target object at each frame k and Note that long-term speed integration would inevitably introduce larger and time-dependent error, a short-time interval may result in undesired peaks in the estimated speed, and interframe subtraction helps to cancel systematic errors in a 3D tracking system. We formulate the problem with interframe motion increments. The pose change between frame ransform .A can be uniquely determined by parameter set P, the 6-DOF speed of the target measured by the MLDSS system at frame k could be calculated by Knowing that matrix We use P, which can be provided by the geometric-only method as introduced in We solve Equation  using thIt has to be mentioned that the description in The four cameras were calibrated ,24 beforWe address the nonideal factors in this practical calibration system as follows.Temporal misalignment. As we described above, the 3D tracking system runs slower than the MLDSS system so that they are temporally misaligned. Using f to denote the frame index of MLDSS, and Equation  is thus P that finally influence calibration precision. To solve it, we modeled the temporal misalignment as a fixed offset parameter Note that although the two systems are synchronized by hardware connection, unintended temporal misalignment might still exist due to the sampling time of the LDV and the exposure time of the cameras. This misalignment causes an error in Equation  is thus Mechanical velocity offset. A galvanometer scanner is used for controlling laser direction. While we assumed the scanner fully stopped when sampling the LDV measurement, the scanner could possibly slightly oscillate and cause error in the speed. Because the control sequence of the galvanometer is fixed and the optical path of each laser is different, this offset can be regarded as systematic to each laser. We thus refined speed reconstruction asWeighing error terms. In solving the multiobject optimization problem denoted by Equation , Gi=Equation , error iMore intuition is provided by Equation . TranslaTwo implications could be derived to help such a situation. First, contribution to the optimization problem from different samples should be balanced considering the corresponding gradient. This is done by reweighing error terms as we introduced in Based on the above discussion, we prepared the dataset for solving Equation  with a cWe validated the proposed method with a real dataset, consisting of 4500 sample frames for the calibration, and 1500 frames for the test. Both the calibration set and the test set consisted of six evenly distributed back-and-forward motion patterns, as shown in We compared the proposed method with the geometric-only calibration method , where tTo quantitatively demonstrate the results, RMS error between the motion-reconstruction result and the ground truth is shown in To show that the proposed method is also valid for the speed estimation of daily objects besides the calibration board, we briefly tested the new parameters on measuring a rotating globe as shown in In this paper, we described a new calibration method for MLDSS. In contrast with the conventional method where only laser geometry is independently calibrated from images, the proposed method simultaneously optimizes all laser parameters and directly associates them with the motion-sensing model. We discussed in detail the system model and factors related to calibration accuracy, and proposed a maximum-likelihood estimation-based method for solving a calibration problem. Finally, higher accuracy was confirmed by cross-validation from qualitative and quantitative evaluation on a 1500 sample test set compared with the state-of-art method.


R) and heart rate (HR). Then, the system was tested on two archers during four shooting sessions. This is the first study to monitor cardio-respiratory activity on archers during shooting. The good performance of the ST is supported by the low mean absolute percentage error for fR and HR estimation , calculated with respect to reference signals . Moreover, results showed the capability of the ST to estimate fR and HR during different phases of shooting action. The promising results motivate future investigations to speculate about the influence of fR and HR on archers’ performance.In precision sports, the control of breathing and heart rate is crucial to help the body to remain stable in the shooting position. To improve stability, archers try to adopt similar breathing patterns and to have a low heartbeat during each shot. We proposed an easy-to-use and unobtrusive smart textile (ST) which is able to detect chest wall excursions due to breathing and heart beating. The sensing part is based on two FBGs housed into a soft polymer matrix to optimize the adherence to the chest wall and the system robustness. The ST was assessed on volunteers to figure out its performance in the estimation of respiratory frequency (f Archery is a precision sport which requires consistency and stability of movements . A mismaIn precision sports, both breathing and heart rate (HR) influence the athlete’s performance ,6,7. TheIn archery, the shooting action can be separated into three main phases: set-up, aiming, and release. During the set-up phase, shoulders are brought in line with the target, hips are rotated forward, and the hand position on the bow grip is set-up. During the aiming phase, the focus is completely diverted to the target and the alignment of bow-sight-target is performed. The shooting of the arrow takes place in the release phase. It is highly recommended that archers start to inhale during the set-up phase of shooting, and to either exhale or hold their breath during the aiming phase. This allows for reducing the level of body rigidity and, simultaneously, preparing the body for the release phase ,10.As a result of training, master marksmen know exactly how to time the release of the arrow with the pattern of their breathing and cardiac cycle, thus minimizing the body jerk caused by the breathing process and the heart contraction . Expert Breathing and cardiac activities can be monitored by several solutions ,16. AmonThe aim of the present study is the feasibility assessment of a custom ST based on flexible FBG sensors for cardio-respiratory monitoring in archery. The proposed system was assessed on volunteers during quiet breathing and apnea, as well as on archers during shooting sessions.® fastener to put the two FBGs in contact to the chest in correspondence of the xiphoid process and the umbilicus. The use of anatomical landmarks allows the FBGs to be worn on the same measurement points. Each sensor consists of an FBG , previously housed in a flexible polymer packaging (90 mm × 24 mm × 1 mm) made of Dragon Skin® 20  as shown in Two flexible sensors were used to develop a ST consisting of two elastic bands  worn around the thorax and the abdomen, respectively. The fit of each band was adjusted by VELCROB). The FBG working principle is well described by the following equation:effη, is the effective refractive index of the fiber core and Λ, the grating period. The dependence of effη and Λ from temperature and strain makes FBGs an optimal solution for the development of measurement systems able to sense these two parameters [FBGs work as stop band filters of wavelength because they back-reflect a small portion of light traveling along the fiber at a specific wavelength  whose age and anthropometric characteristics are shown in i) a short apnea useful to synchronize the reference instruments  and the FBG outputs; ii) 16 quiet breaths; iii) a final apnea as long as each volunteer can. The study was approved by the local ethical committee (protocol number 27/18).Each participant was asked to perform a protocol consisting of three main phases: T for the band around the thorax and FBGA for the band around the abdomen, a box in b box) which worked at the sampling frequency of 250 Hz. The reference signal for the respiratory activity was collected by a commercial flow sensor  and HR (during apnea), and the obtained results.This paragraph will be grouped into two subsections according to the stages of the protocol . Each subsection describes the data analysis performed to estimate fi) the outputs of the FBGs and the flow sensor were synchronized by selecting the first minimum points  after the starting apnea (see ii) the quiet breathing stage was selected by cutting all the synchronized signals from the mentioned starting points and considering all the 16 breaths performed during the protocol (see iii) a filtering stage consisting of a second order pass-band filter (lower cut-off frequency of 0.05 Hz and higher cut-off frequency of 0.5 Hz) was applied on both FBGs and flow sensor outputs; iv) a custom algorithm was used to select the maximum peaks of each signal  theocol see ; iii) a gnal see .RA and TRT for the band around the abdomen and the thorax, respectively, and TRFLOW for the flow sensor) were calculated as the time interval between two consecutive maximum peaks. Then, the fR values estimated by both the FBGs  and the flow sensor  were calculated as the ratio between 60 and the related respiratory periods in order to express fR in acts per minute .The respiratory periods  for the calculation of the mean of difference (MOD) and the limits of agreement . The mentioned analysis was performed considering data provided by each flexible sensor. The mean absolute percentage error in the fR estimation  was used to compare mean fR values and was calculated as:R obtained by the proposed system and the reference one, respectively.The ST was assessed in terms of both breath-by-breath and mean fence one . This anR values measured by the proposed system and the reference one is confirmed by the high value of the correlation coefficient (R2) for both FBGT and FBGA and by the low value of both MOD and MAPEfR.Results of the breath-by-breath analysis are shown in i) for each trial, the first minimum points after the holding of breath were selected on the FBGs output to define the starting point of the apnea stage for the FBGs and the PPG sensors; ii) the same time interval  was chosen to estimate HR of all the volunteers during the apnea (see iii) a fourth-order Butterworth pass-band filter with a lower cut-off frequency of 0.6 Hz and a higher cut-off frequency of 20 Hz was applied on the signals. This band of frequency was chosen according to the frequency components of vibrations induced on the chest wall by the blood flow ejection into the vascular bed [iv) a custom algorithm was used to select minimum peaks (blue markers in C) from the FBG outputs was calculated considering the minimum peaks, as the time elapsed between two consecutive minimum peaks . Minimum peaks were chosen because they are easier to detect on the filtered FBG signals, automatically. Beat-by-beat cardiac periods were calculated considering the time interval between two consecutive maximum peaks on the filtered PPG signal (TCPPG). The HR values  were calculated as the ratio between 60 and TCT, TCA, and TCPPG, in this way HR is expressed in bpm.The HR estimation during the apnea was performed according to the following four main phases: pnea see ; iii) a ular bed ; iv) a cHR) was also calculated as follows:R values measured by the proposed system and the reference one is confirmed by the high value of the R2 for both FBGT and FBGA and by the low value of both MOD and MAPEHR.The ST capability of monitoring HR was assessed in terms of beat-by-beat and mean HR values. The Bland-Altman analysis was performed by comparing HR values estimated by the proposed system and the reference one, considering all the volunteers for a total number of 149 beats. The mean absolute percentage error . Their characteristics are listed in Archers were invited to perform two shooting sessions. Each session consists of six arrows to be shot in five minutes at designed 70 m targets. The first session is a practice round. In this round the arrows are shot at the beginning and do not count as part of the score. Instead, the second one is a scoring round with an awardable maximum score of 60 points (10 points per arrow).During each session of shooting, archers worn the flexible sensors on the same positions investigated during tests on volunteers see . The optT and FBGA output of the two practice shooting sessions are plotted in Changes of FBGT output because the respiratory acts and the shooting phases are clearly discernible while FBGA output has a more irregular trend  the six shooting actions were selected, as shown in ii) for each shooting action, the signal related to the aiming phase was filtered and its minimum peaks were detected to calculate the HR values (iii) the signal related to the breathing activity which precedes the shooting actions was filtered and its maximum peaks were detected to estimate the fR values. Results in terms of fR and HR for all the four trials are summarized in The analysis was performed by selecting FBGrend see . FBGT ouR values ; iii) thThis work is focused on cardio-respiratory monitoring in archery using a custom ST based on flexible FBGs.R and HR estimation, as shown in R estimation, while FBGT allows a more accurate HR monitoring than FBGA .In the literature, FBGs encapsulated into flexible materials for cardio-respiratory monitoring were mainly proposed for clinical applications . These technological solutions consist of FBG sensors encapsulated into polydimethylsiloxane, PDMS , polyvind HR. In , an FBG R during the breathing activity and HR during the aiming phase. In the literature, commercially available devices able to monitor only one of these parameters (fR and HR) were used on archers. Breathing activity and patterns were studied in [Focusing on the shooting sessions, results showed that the proposed ST can detect all the six shooting actions and, in turn, monitor fudied in  by usingudied in ,5,7 HR vR and HR are important since they influence archers’ performance. Therefore, the main improvement of the proposed system is the possibility to monitor both fR and HR with high accuracy during the different phases of the shooting action using the same sensing element. Since the sensing element can be connected to the optical interrogator by means of long, flexible and lightweight fiber optic, the proposed system does not impair the shooting action. This feature encourages further assessment of the proposed system in sports science applications .These studies showed that both fR and HR influence the shooting performance, and how HR can be estimated in the presence of breathing. These findings will aid the optimization of training strategies according to the experience of each archer and the maximization of their shooting action and scores.In future works, a high number of archers will be enrolled to investigate how f


The purpose of this study was to determine the clinical effects of a training robot that induced eccentric tibialis anterior muscle contraction by controlling the strength and speed. The speed and the strength are controlled simultaneously by introducing robot training with two different feedbacks: velocity feedback in the robot controller and force bio-feedback based on force visualization. By performing quantitative eccentric contraction training, it is expected that the fall risk reduces owing to the improved muscle function. Evaluation of 11 elderly participants with months training period was conducted through a cross-over comparison test. The results of timed up and go (TUG) tests and 5 m walking tests were compared. The intergroup comparison was done using the Kruskal-Wallis test. The results of cross-over test indicated no significant difference between the 5-m walking time measured after the training and control phases. However, there was a trend toward improvement, and a significant difference was observed between the training and control phases in all subjects. Due to the extension of the average life expectancy and the decline in the birth rate, population aging is progressing on a global scale . One of Maintaining physical function and activity frequency is important in increasing healthy life expectancy, and securing mobility ability is crucial for this purpose. Since walking is the most common form of mobility, preventing falls during walking is of great importance. Therapy for walking ability improvement to prevent falls typically included balance training, joint range of motion (ROM) training, and muscle training. Many studies have reported that when muscle training is included in physical therapy programs for the elderly, the result is improvement in walking ability . FurtherTibialis anterior muscle activity during gait has an amplitude equivalent to 45% according to the MMT  . In otheCurrently, there exist several methods of eccentric contraction training of the tibialis anterior muscle using devices. Roy et al. developed a novel ankle robot called “anklebot” . In addiThe training robot is equipped with two DC motors. The motors are controlled by a PC, based on angle information acquired from optical encoders attached to the motors. Force sensors attached to the footplate enable the acquisition of force data. The data obtained at the tip of the footplate are visualized by computer graphics, and the trainee performs the eccentric contraction exercise by adjusting the manifested force while referring to this force data. pK and dK denote the position and velocity gains, respectively. J, i, g, and n denote inertia of moment, motor current, cutoff frequency of pseudo derivative, and reduction ratio of the gear, respectively. Res, cmd and ref in the subscripts denote response, command and reference values, respectively. tk denote external torque and torque constant, respectively. This study considered only the dorsiflexion and plantar flexion of the ankle joint; therefore, the angle of the rear motor controlling the ankle height remains constant, and a wave input is provided to the front motor controlling the tip height. With such a controller, dorsiflexion and plantar flexion of the ankle joint can be achieved while maintaining a constant heel position.From control engineering studies, it is known that simultaneous position and force control cannot be achieved in robots . This caHowever, eccentric contraction training requires appropriate force along with sufficient velocity, which is determined based on position control. In general, it is difficult to accomplish a position/velocity command and a force command simultaneously. Compliance control is a common solution for the system with position and force commands, despite resulting in compromising the two commands. This study addresses the issue by introducing a control architecture shown in As shown in A total of four sets of training was performed with two sets for each foot. Here, one set consists of eight exercises of eccentric contraction with 3 s. One-minute breaks were taken between sets. The target force was 40N at the tip toe. The subjects in this study consisted of 11 elderly individuals with walking ability at a nursing care center. Subjects with lower limb pain and serious medical histories  were excluded in this study. The contents of the study were explained to the subjects and families prior to the experiment and informed consent was received from all subjects. The present study was carried out with the authorization of the Human Research Ethics Committee of Saitama University. The duration of the evaluation and training period took approximately two months. The training sessions were conducted four times within one month, in sets of eight, twice in one session. The interval time was 1 min between each set. In this study, we conducted a cross-over comparison test to determine the training effects of the robot as shown in The TUG and 5-m walking tests were conducted as methods for the evaluation of training effects with the use of the training equipment. Stopwatches were used for measurements in both tests. The time taker was blinded.The 5-m walking test is a test for the assessment of walking speed. In a previous study, the 5-m walk test was used for motor-performance measures . This teThe TUG test assesses functional dynamic capacity, and this method has also been shown to correlate with measures such as the Berg Balance Scale , gait spThe normality of data was assessed by the Bartlett’s test. In case any non-normal data was detected, appropriate non-parametric testing was used. The Kruskal-Wallis test was used for the intergroup comparison of walking speed and TUG time at different times . Furthermore, in this study, all the subjects were divided into a training phase and control phase regardless of a cross-over test, and a comparative investigation was performed. The Wilcoxon signed-rank test was used for comparisons between the two groups. The level of significance was set at 5%. Statistical procedures were performed using MATLAB 2016a.p < 0.05). However, there was no significant difference in the TUG time.As described above, we conducted a clinical study of a training device designed to assist in the eccentric contraction training of the tibialis anterior muscle. The purpose of this study was to investigate the effects of walking training with the use of the device. The results indicated that there was no significant difference in the walking speed during each period measured using the cross-over test. However, there was a trend toward improvement, and a significant difference was observed between the training and control phases in all subjects.An advantage of the present study is shown via the eccentric training focusing on muscle contraction mode, not muscle strengthening. Many previous studies recommend high-intensity and high-frequency training for muscle strengthening . The traThere was no significant difference in time in the TUG test. The results in  also shoIt is known that the cross-over comparative experimental design may indicate a training effect caused by the carryover effect. However, no improvement was found during the control phase in the group that initially performed the training. The fact suggests that there was no carryover effect and that the results obtained were due to the training itself. The scores for all subjects decreased 1 month after the intervention. The results indicate that the effect of the proposed training is limited to a short-term period and it is important to have continuous training. The limitations of this study include the fact that, since the training effect was investigated at only one performance level, we may not have fully investigated walking ability in detail. Improvement in walking ability might cause changes in overall physical dynamics, including the vicinity of the ankles, which are in direct contact with the tibialis anterior muscle. However, in order to investigate this in detail, a larger-scale experimental environment that includes the use of a three-dimensional motion analysis system would be required.This study discussed the possibility of using the training system with force bio-feedback to improve walking ability through eccentric contraction training of the tibialis anterior muscle. Evaluation of 11 elderly participants with months training period was conducted through a cross-over comparison test. There was a trend toward improvement on 5 m walking test, and a significant difference was observed between the training and control phases in all subjects. Since we investigated the effect of training in clinical settings, we were unable to perform a detailed analysis of the walking movement. We would like to conduct further studies on the reliability of this experimental device that also address these issues.