The modern-day vehicle is evolved in a cyber-physical system with internal networks (controller area network (CAN), Ethernet, etc.) connecting hundreds of micro-controllers. From the traditional core vehicle functions, such as vehicle controls, infotainment, and power-train management, to the latest developments, such as advanced driver assistance systems (ADAS) and automated driving features, each one of them uses CAN as their communication network backbone. Automated driving and ADAS features rely on data transferred over the CAN network from multiple sensors mounted on the vehicle. Verifying the integrity of the sensor data is essential for the safety and security of occupants and the proper functionality of these applications. Though the CAN interface ensures reliable data transfer, it lacks basic security features, including message authentication, which makes it vulnerable to a wide array of attacks, including spoofing, replay, DoS, etc. Using traditional cryptography-based methods to verify the integrity of data transmitted over CAN interfaces is expected to increase the computational complexity, latency, and overall cost of the system. In this paper, we propose a light-weight alternative to verify the sensor data’s integrity for vehicle applications that use CAN networks for data transfers. To this end, a framework for 2-dimensional quantization index modulation (2D QIM)-based data hiding is proposed to achieve this goal. Using a typical radar sensor data transmission scenario in an autonomous vehicle application, we analyzed the performance of the proposed framework regarding detecting and localizing the sensor data tampering. The effects of embedding-induced distortion on the applications using the radar data were studied through a sensor fusion algorithm. It was observed that the proposed framework offers the much-needed data integrity verification without compromising on the quality of sensor fusion data and is implemented with low overall design complexity. This proposed framework can also be used on any physical network interface other than CAN, and it offers traceability to in-vehicle data beyond the scope of the in-vehicle applications. In the last 20 years, the automotive industry has seen rapid growth in the usage of electronic control units (ECUs) to implement various technology features, such as dynamic vehicle control, infotainment, and ADAS safety features. These technologically-enhanced user experience and safety features are heavily dependent on the sensors and advanced communication networks integrated into the vehicles. An autonomous vehicle (AV), in particular, depends heavily on sensors and data transmission networks to implement the highly automated driving functions . AutonomAutonomous vehicle sensors are broadly divided into smart and raw categories. The raw sensors—specifically, the ones that have high data rates, such as the camera and LiDAR—use an Ethernet interface for raw data transmissions to the ADAS unit, whereas smart sensors that can detect and track objects internally send the tracked object list to the vehicle over limited bandwidth interfaces. The majority of sensor sets available for use in autonomous vehicles are sold as smart sensors. These sensor units can sense and track objects internally and send out the tracked data over a deterministic and fault-tolerant interface such as CAN/CAN-FD. The data from these smart sensors are fused by the vehicle ADAS unit to determine the final list of surrounding objects .Since most of the smart sensors use CAN or CAN-FD interfaces for data communication, a vulnerability assessment to cyber attacks is prudent for these network types. The CAN interface is widely used in the automotive industry due to robust and fault tolerant design; however, CAN lacks inherent security to protect against different network attacks. The shortcomings in CAN protocol such as the broadcast message format, clear data transfers, and lack of mechanisms to establish data authentication and confidentiality, expose CAN networks to masquerade attacks, replay attacks, and additional exploits . The numTo deal with such implementation level and practical shortcomings of the traditional data integrity verification methods, we introduce a new data hiding-based watermarking approach. This approach solves the problem of the data integrity verification in resource-constrained and real-time applications with simple algorithms that do not tax the system with high computational complexity and at the same time do not increase the bandwidth requirements of the interface, as no additional data are added to the payload. In this method, the watermark is embedded into the sensor using a light-weight software algorithm, which is easy to implement. The concept of using data hiding techniques for sensor data verification was introduced in  for raw In this digital age, with the growing trend of multimedia information exchange, the need to ensure data confidentiality and integrity is increasing. Data transfers over different networks are vulnerable to various attacks such as privacy infringement, content stealing, and tampering. The concept of the network information security deals with securing the data transmissions, storage, and processing from data leakage, theft, tampering, and deletion . To ensuIn comparison to the conventional cryptography, watermarking methods are computationally less complex and less resource hungry. Data modified by the watermarking algorithms can be directly used by the end application without having to modify or to clean it before usage. This comes as an advantage for applications that rely on real-time data processing. In Watermarking also provides the much-needed data traceability. The security offered by watermarking does not stop at the application level. Watermarked data provide security beyond the autonomous driving application since the data cannot be stripped of any additional payload. The embedded watermark stays with the data until the information is used by the consumer application and beyond, which is not the case with the traditional cryptography techniques that end at the transport layer. Applications acting as pass-through sections to the data, such as an onboard data recorder and secure logging mechanisms that push the sensor data to cloud for analytics, CAN benefit from the watermarking of the data. When data get exchanged from one entity to another and security keys are shared, end-to-end encryption does not help in identifying the leakage point, whereas watermarking can. These advantages make data hiding-based watermarking techniques a better choice over the cryptography in many edge computing applications such as sensor data integrity verification in autonomous vehicles.Automotive sensor networks have two constraints.Limited bandwidth: The communication interface from the sensor to the data processing unit is bandwidth limited in automotive applications. Most sensors use a traditional CAN interface with an 8-byte payload, that restricts the usage of traditional cryptographic methods, securing the sensor data [sor data . An enhaReal-time data integrity verification: Autonomous vehicle applications often require the sensor data to be processed in real-time. This constraint makes it difficult to use traditional data security methods based on cryptography as they require an additional step of decryption before data becomes useful.With these constraints and given the lack of cyber-security mechanisms in CAN/CAN-FD, the in-vehicle network can get hacked at the data interfaces or transport layer, as shown in Digital watermarking methods have a significant advantage over the traditional cryptography methods when it comes to the resource-constrained systems where the interface bandwidth or the computation resources are a bottleneck. State-of-the-art research in watermarking based techniques to secure resource-constrained systems leads us to wireless sensor networks (WSNs). WSNs have similar interface bandwidth constraints to those of legacy automotive networks. The dynamic nature of these networks makes them susceptible to adversary attacks, such as data tampering, forgery, selective forwarding, replay, and transfer delay. Hence, in this area, significant research has been done to secure data transfers using watermarking.Using watermarking for sensor integrity checks in WSN started in 2003 with the Feng et al.  proposalIn , KanchanIn the system model, we assume that a centralized ADAS unit makes autonomous driving decisions using the data fed by satellite sensors over the vehicle network, as shown in The dataset used to test the proposed framework was generated using the reference code provided by Mercedes Benz autonomous driving utility . The datv is the target velocity, The LiDAR measurements are represented as position coordinates np-dimensional vectors, n is signal length. The dimensionality of the signal depends on the application. Considering the example of position data from a radar sensor, which is represented by a two-dimensional vector p becomes 2. In data hiding-based methods, the host signal The watermarking model that fits the sensor data integrity verification problem is the side information based blind decoding method. In this model, the detector is unaware of the original host signal. Still, the auxiliary information such as the encoding message sequence or the key is used to accurately verify the data integrity. The host signal fidelity .m, the detector operates in the detection boundary. The detection region for a given message m and watermark key k is defined as the set of position vectors in the data-space that can be used to extract the embedded message. This region is defined by the threshold on a similarity measure, such as a linear correlation between the detector’s input and the original message m. During embedding, the message m is message mapped onto the pattern of the same type and dimensions of the host signal, and k. The detection algorithm computes a linear correlation between the received signal and the reference pattern Once the embedder creates this dataset of similar position vectors, based on the input message In our proposed sensor data integrity verification framework, we assume the following. First, the sensor and the ADAS processing unit share the watermark generation algorithm. Second, the ADAS processing unit collects detection lists from different sensors, in this case, LiDAR and radar, and generates a fused list of detection. The proposed framework, as shown in Watermark generation.Watermark embedding.Watermark decoding.The watermark generation is done in the sensor using the GPS timestamp information. This binary sequence of message Algorithm 1: Watermark sequence generator.A core component of our integrity verification network is our watermark generation technique. Our proposed model leverages timestamp from GPS sensor data to generate a binary sequence that will be embedded into subsequent radar data frames downstream. The diagram in x, generated from the seed value of the The sequence generator represented by Algorithm 1 will utilize the previously gathered information from the GPS time stamp to generate a deterministic sequence. This process involves taking the range limited random number We use a 2D QIM-based data hiding method for watermark embedding. QIM is a non-linear, data hiding-based, semi-fragile watermarking method that is widely used in digital forensics and steganography . In QIM It can observed from Equations  and 5)  5) that n-dimensional symbol, it results in a In QIM, the quantization operation uses a unique set of quantizers that result in a reconstruction grid . The dimn points . The resIn the proposed framework, a simple algorithm parses through the generated message symbols and applies the corresponding quantizer to the radar position data as per Equation . The senThe radar detection object list data received by the ADAS unit can be directly used by the sensor fusion algorithm. Tamper detection and localization algorithms can run in parallel. This is a significant advantage of the watermarking method over any other encryption-based methods. The decoding algorithm works similarly to the embedding. Each received position value is quantized using all the different quantizers used for embedding to generate different reconstruction points, which is a set of four in our case. These four reconstruction point values are compared with the received value, and the reconstruction point that returns least difference value, as shown in Equation  is consiSuccessful attacks are the ones that go undetected by the detection framework. In this section, we discuss various attack scenarios possible if an attacker gets access to the vehicle network and how the proposed framework can detect and localize these attacks. As a part of the attack model, we assume that the attacker has a good knowledge of the vehicle network protocols and automotive electrical system architecture. He has tools available to sniff the vehicle network and replay the modified messages on CAN/CAN-FD. With this knowledge, we identified three ways the attacker can modify the sensor data once he sniffs it from within the network. The attacks are be broadly classified asData addition;Data deletion;Data modification.For each of these attack scenarios in the following sections, we analyze how the proposed framework performs.n to N represents the number of encoding bits or dimensions, and Algorithm 2: Find added indices.Data addition is an attack scenario where the attacker modifies the radar reflections or tracklets with additional fake data elements either by copying the existing elements or by adding completely random data. A typical add attack scenario is represented in n points . It can element. dmin2>4·Nn to In this scenario, as shown in Algorithm 3: Find deleted indices.The tamper localization algorithm is shown in Algorithm 3. The decoded message sequence is compared with the expected message sequence in an Equation . It can Algorithm 4: Find modified indices.During a data modification attack, as shown in Though, the data hiding-based sensor integrity framework is computationally less complex than the traditional cryptography, one of the major concerns with the method is the embedding-induced distortion as the watermark is embedded directly into the data by altering it. A question arises whether the end application or the consumer of this data can handle this distortion. An answer to this question would help the automotive industry adopt the watermarking techniques for sensor data or other data integrity verification applications. In this study, we analyze the effects of embedding radar data using the proposed 2D QIM data hiding framework on a sensor fusion algorithm. The sensor fusion method we use for this study is EKF. In autonomous vehicles, Kalman filters are used to estimate the state of any dynamic system, such as position estimation of moving objects on the road. In doing so, the Kalman filter only needs current observations and previous predictions, hence Kalman filter is a light-weight fusion algorithm ,33,34. Tx and P at previous time step F is the state transition matrix. The uncertainty measure P is a stochastic process modeled as random noise that affects the prediction step. The state vector F is the state transition matrix that models state transitions from previous time step Q is the process co-variance matrix that models the stochastic part of the state transition function. A linear motion model with constant velocity is used to define the state transition matrix F. The position at next time step To predict the position of a target, a Kalman filter uses a motion or process model that estimates the future location of the target or object of interest. In this paper, we use a constant velocity motion model as a baseline for target motion estimation. The motion model, as depicted in In the update step, the sensor measurements are used to correct the predicted states and to obtain more accurate estimates. In the measurement function the vehicle only senses the pedestrian position and can be expressed asH, and covariance matrix R to correctly estimate the measurement vector z. The measurement matrix is required to transform the measurement vector z to the state vector as shown in Equation  . In EKF,y is the error value or the difference between the prediction and actual measurement at a given time step. The estimation error S is computed asHere K is computed asThe Kalman gain After the computation of the Kalman gain, the predictions are updated using the following equations and these steps are repeated for the entire drive cycle.R in Equation (Q in Equation (R, second the overall process noise Q, and third, the embedding step-size t and n is the length of data. The RMSE value is used to determine the accuracy of the prediction. Low RMSE value indicates that the sensor fusion algorithm predicted the tracked object position accurately throughout the track path. The RMSE values of position vector It can be observed that in an EKF, the uncertainty in both the process and measurements are taken into consideration. In general, the measurement uncertainty or the measurement noise covariance matrix Equation , is the Equation , is defiR represents the deviation of the sensor measured values form the true values. This deviation is estimated during the calibration phase by the sensor manufacturer. In the absence of the sensor manufacturer data, it can also be estimated using analytical methods [R value for a given sensor must be known or estimated to be used in Equation (The measurement input to EKF has a measurement noise component which is dependent on the intrinsic electronic characteristics of the sensor. This can be represented as an additive Gaussian noise Equation . The mea methods . To accuEquation .If we consider Equation , an EKF The RMSE results depicted in Equation . In thisEquation . Here thApart from the embedding-induced distortion analysis, two different experiments are conducted to measure the other performance parameters of the detection framework, such as the bit error rate and the false alarm rate.n is the size of the decoded message bitstream. When no additional noise is added to the radar data elements, the BER is close to In this experiment we analyze the errors in the decoded bit stream in the presence of channel noise. The decoder step in the proposed framework generates a binary message stream The false alarm rate analysis is another important performance indicator of the proposed framework. It measures the number of data-elements the framework determines as tampered when it is provided with clean or unmodified data. In this experiment, the framework is tested with a combination of clean and modified data elements and the false alarm rate Cyber-physical system, such as an autonomous vehicle, is susceptible to insider attacks targeting sensors and their transmission channels, making it necessary to verify the integrity of sensor data before acting on it. Traditional data integrity protection methods, such as cryptography, cannot be applied in their entirety to solve this problem due to their resource requirements and complexity. In this paper, we propose a pipe-line based on watermarking to detect and localize tampering of the sensor data in an autonomous vehicle. This pipeline is tested for the affects of embedding-induced distortion using simulated radar data on an EKF based sensor fusion algorithm. The experimental results conclude that that the 2D QIM method for watermarking has a little or no effect on the EKF predictions for small values of quantization step-size In the security world, often times a layered architecture is preferred wherein if an attack cannot be prevented; It can be detected to prevent the worst outcome. We believe that watermarking the sensor data adds another layer to the security scheme using some light-weight and yet efficient techniques. These techniques can be used either in a standalone mode or in conjunction with traditional cryptography methods where-ever necessary, to secure data transfers over any physical interface such as CAN/CAN-FD, Ethernet, etc. In future, we plan to extend the proposed framework to different sensor modalities, different watermark embedding methods and intend to study the affects of embedding-induced distortion on more complex process models and state vectors.


Traffic light recognition is an indispensable elemental technology for automated driving in urban areas. In this study, we propose an algorithm that recognizes traffic lights and arrow lights by image processing using the digital map and precise vehicle pose which is estimated by a localization module. The use of a digital map allows the determination of a region-of-interest in an image to reduce the computational cost and false detection. In addition, this study develops an algorithm to recognize arrow lights using relative positions of traffic lights, and the arrow light is used as prior spatial information. This allows for the recognition of distant arrow lights that are difficult for humans to see clearly. Experiments were conducted to evaluate the recognition performance of the proposed method and to verify if it matches the performance required for automated driving. Quantitative evaluations indicate that the proposed method achieved 91.8% and 56.7% of the average f-value for traffic lights and arrow lights, respectively. It was confirmed that the arrow-light detection could recognize small arrow objects even if their size was smaller than 10 pixels. The verification experiments indicate that the performance of the proposed method meets the necessary requirements for smooth acceleration or deceleration at intersections in automated driving. Automated vehicle technologies are considered to be the next generation transportation system. Many companies and research organizations are involved in the research and development of such technologies. Recent automated vehicle technologies focus more on urban driving, which is a mixed transportation environment with human drivers. Public road demonstration experiments have been carried out in the U.S., European, and Asian countries since the mid-2000s ,2,3. AutRanging sensors: light detection and ranging (LiDAR), millimeter-wave radar (MWR)Imaging sensor: cameraPositioning sensor: global navigation satellite system and inertial navigation system (GNSS/INS).The above mentioned sensors make it possible to observe surrounding transportation environments. In addition, recent automated driving technologies rely on high-definition maps (HD maps) which include the precise position of static road features such as lane boundaries, lane centerlines, traffic signs, and traffic lights. By referring to the predefined road features, it is possible to reduce false surrounding recognitions and implement accurate decision making, considering road structures. The following functions must be implemented in order to achieve automated driving on public roads.Self-localization: Estimating the position of the vehicle on the HD map in the accuracy of decimeter-levelSurrounding perception: Recognizing static/dynamic objects including traffic participants and road objects Motion planning: Designing the optimal collision-free trajectories following the traffic rulesMotion control: Determining adequate control signals such as steering, acceleration and braking.This study focuses on traffic light (TL) detection using HD maps. There are many studies on TL detection by using vision-sensors in intelligent transportation system. TL is one of the most important road features in order to decide an approaching the intersection. Although HD maps have the information of TL positions, the vehicle must recognize the current state of TLs in real time because it changes dynamically. For safety deceleration, it is necessary to recognize the current state of TLs at distances over 100 m. The required recognition distance can be estimated by calculating the braking distance from the vehicle to the stop line after smoothly recognizing the TL state. In studies on vehicle control ,5, the dOn the other hand, in automated driving using HD maps, the self-localization module precisely estimates the vehicle pose by map-matching using a range sensor or image sensor ,7,8. GenThe purpose of this study is to achieve small object recognition for both TLs and arrow TLs. Different types of candidate detectors are introduced to realize robust lighting area detection. In addition, in order to implement robust arrow recognition, the proposed method uses spatial information of the positional relationship of lights in TLs as prior information obtained from the HD map. The proposed method is specialized in the recognition of small-sized objects, and evaluate recognition performance for actual driving data. Moreover, verification experiments for automated driving were carried out by introducing the proposed method to investigate the validity of the method.The rest of this paper is composed as follows. Determine the search region: A region-of-interest (ROI) is extracted from the captured image by using the predefined map.Extract candidate objects: Circular lighting areas or rectangular objects are extracted from the search region as candidate TLs.Classify the state of the candidates: Simple color filtering or machine learning algorithms identify lighting colors and arrow light directions.Although different approaches have been developed, most of the methods involve extracting candidates according to their specific color spaces ,20 and cOn the other hand, assuming that our algorithm will be introduced into the automated vehicles, real-time proceeding is important for decision making. In addition, in order to reduce the delay in recognition, it is necessary to recognize TLs in an appropriate processing time in accordance with the velocity of the vehicle. For example, when traveling at a velocity of 50 km/h, a vehicle moves about 14 m per second. Then, it is important to estimate the required time in consideration of the responsive deceleration for practical development.In our previous study , the TL The performance evaluation is performed in challenging dataset including small objects of TLs and arrow TLs in the day and night.The effectiveness of prior information is evaluated with respect to the performance in recognition of distant TLs.Verification results are presented by introducing the proposed method in automated driving.Before describing the proposed algorithm, the problem of the TL recognition addressed in this study is explained. In the TL recognition, the task is to recognize the state of the TLs in the image that corresponds to the HD map. As shown in the 2-D TL positions attribute information of the TL directions attribute information of the type of the TL patterns see a.A highly-precise digital map is maintained by a growing number of professional mapping companies. Accurate positioning systems, combined with cameras and LiDAR sensors can be possible to generate a precise 3-D maps which contains latitude, longitude and altitude reflectivity. For the purpose of the TL detection, location data of each TL was recorded into the map information. The method described in this paper uses the following information as prior map information:Although the exact altitude of the TLs is not used as a prior information, they are installed at a height of approximately 5.0 m above the ground surface on the Japanese road environment. Therefore, the recognition process is performed by considering a height of 5.0 m as a reference height, and providing a margin that assumes a road gradient. Although the standard height of TLs is specified in Japan, the standard height should be different in other countries. It is necessary to set an appropriate height according to the target country or to set a wider recognition area in the image when the height information is unknown.Search target TLs and compute ROIGenerate a highlighted image as a feature image which emphasizes the lights of TLs in the ROI imageExtract candidates for TL lights in the generated highlighted image using three types of different methodsCompute the probability of existence area containing TLs using a time-series processingRecognize the arrow light, if the target TL has attribute information of arrow light.As described in Arrow detection requires recognition of directions, which is affected by unclear images. In order to improve such distant arrow recognition, we propose an arrow recognition method using prior information of the HD map. In addition, this study verifies the effects of using prior information in the arrow light recognition. The algorithms are described in detail in the following section.Among the TLs that appear in the frontal camera image, the TLs that are within a certain distance The ROI image is clipped for each target TL. The location of the ROI can be calculated based on the current pose and the map database. A global TL position Normalize the brightness value to emphasize the lighting.Update the saturation value to eliminate background noise.Weighting with respect to hue value, close to the lighting color of traffic signals.The lighting areas of the TLs have higher brightness and saturation compared to other objects. Therefore, the highlighted image can be generated by multiplying the saturation image by the brightness image. In order to extract lighting areas, RGB images are converted into HSVcolor space. The lighting area of the TL gets highlighted as shown in The first operation updates the image brightness. The brightness value is normalized using the following equation:The second operation updates the saturation values. The lighting area of the traffic signal generally has saturation values above a certain value. The pixels with saturations lower than this value are reduced using the following equation:The third operation multiplies the pixel values of the highlight image with weight, with respect to the hue values. After the generation of the highlighted image, a lighting area detection is applied to the obtained image. As mentioned in H, the types of the edges can be categorized by using The first method is the circle detector. The shape of the lighting area is generally circular shape in the image. A method based on the Hough transform has been adopted to extract circular candidates . HoweverThe second method is the blob detector. In the feature image, areas with higher pixel values are distributed near the lighting areas. These areas can be extracted by binarizing and segmenting the image as shown in The third method is the ML-detector. Because the detection is performed using camera images, the brightness of the whole image may be affected by the influence of the surroundings, such as sunlight and environmental light. In such cases, the lighting area of the TLs cannot be sufficiently emphasized in the highlighted image. In order to recognize such TLs with lower brightness, it is effective to focus on the whole shape of the TLs together. Generally, machine learning is a common approach to detect such shape of the TL. In the proposed method, the OpenCV cascade detector trained by AdaBoost  is used The lighting states are classified in the detected objects by these methods, and determined as final candidates. In order to eliminate false-positive detections, objects with comparatively smaller and larger radius are deleted based on the pre-calculated radius The lighting color of the candidate object needs to be classified from the hue and brightness distribution of the lighting area. In the proposed method, histograms of the highlighted image and hue image are created for the detected lighting area, and then the AdaBoost classifier is trained using the normalized histograms at the maximum frequency as a feature vector. The generated classifier recognizes the lighting state via four classes, namely Green, Yellow, Red, and Background.l and the probability p for the i-th target signal at time t.t, respectively. i-th TL. The log-odds can be updated by additional computation in BBF.i-th target signal. The initial value of The candidates detected in the ROI are potential objects of the target TL. In order to output a likely object from the obtained candidates, a time-series tracking process is performed by computing existence probability. In , multi-oIn addition to the TL detection, an arrow signal is recognized when the TL has the attribute of arrow lights in the HD map. In Japanese traffic environment, arrow lights are generally lit at red and yellow TLs. After detecting a yellow or red signal, an arrow detection ROI is determined as shown in By using the proposed method described above, the TL recognition is realized by detecting the candidate objects, classifying the lighting color, and computing the confidence using the existence probability. This work further improves the recognition performance, especially for distant arrow lights, by utilizing the prior information given in the digital map.In the TL recognition, when there are multiple candidate objects, it is possible to weight candidates according to the distance of the TLs in the probability updating procedure. It is expected to reduce false-positive detections in background.On the other hand, in arrow recognition, recognition can be improved by providing the pattern of the target TL from Experiments have been carried out to evaluate the effectiveness of the proposed method with actual driving data. Some driving data have been collected using the automated vehicle owned by our group. This vehicle had various functions necessary to enable automated driving in an urban area, and has actually been running in Japan for several years. In previous works, real-time localization algorithms have been developed using different types of sensors such as 3-D LiDARs, cameras or MWRs ,29,30. TIn evaluating the performance of the proposed method, it was important to objectively compare to the existing methods. However, it was difficult to directly compare the reported performance of other works due to different types of cameras, and sensors, and different experimental conditions such as driving area, and weather conditions. Therefore, in addition to the evaluation based on the recognition distance, the recognition performance for objects with the similar pixel size as other works was also evaluated. Analysis of the contribution to the recognition rate by using spatial prior information in the proposed methodComparison between the proposed method and a general object detection algorithm using DNN (YOLOv3 )Performance comparison of each candidate object detector  in the proposed methodComparison of processing time in recognition.The evaluations carried out in this work are summarize bellow:TP is the number of true-positives, FP is the number of false-positives, and FN is the number of false-negatives. The TL and the arrow TL were evaluated separately due to the difficulty in recognizing and the different number of data.YOLOv3 was adopted as one of the state-of-the-art methods because it is a widely used DNN in object detection. The ML detector and the arrow detector (AdaBoost) of the proposed method and YOLOv3 were generated using the training data in The relevant parameters of the proposed method were set as follows. Next, we objectively evaluate the recognition results of this study against the performance reported in existing works. de Charatte reported a precision of 95.38% and a recall of 98.41% (37.4 ms per frame) as a recognition performance of LARA, a French dataset . In addiIn order to detect candidate objects, the proposed method combined three kinds of methods: circular features by SURF, blob features by binarized images, and features of traffic signal shape by AdaBoost detector. To evaluate the contribution of each method, the obtained recognition rates were compared with the results obtained when each detection method was used alone. Although the recognition rate of the proposed method and SURF were almost identical, a detailed analysis was performed to verify the superiority of the proposed method. Finally, the processing time for each method has been evaluated. We showed the superiority of the proposed method by evaluating a recognition distance for TLs and arrow TLs. Next, additional verification was carried out to determine if the proposed method has the required performance in actual automated driving. In the verification experiment, automated driving was performed in the Odaiba area of Tokyo as shown in Driving data were measured on the following two types of travel routes.Route passing through intersections A to E with only TLs in Figure 21Route passing through intersections F to G with TLs and arrow TLs in In both routes, there was no preceding vehicle, because the automated vehicle should drive at the maximum velocity. Then it was verified whether the vehicle could drive smoothly while properly recognizing each TL. In this verification experiment, a camera different from In addition, in our group’s work, demonstration experiments of automated driving have been conducted in the area shown in This work has proposed a TL recognition algorithm for urban automated driving. We prepared the challenging dataset that includes objects with large and small pixel sizes of objects for traffic lights and arrow lights. The proposed method can be processed in real time by the CPU, and our work verified that the proposed method can recognize TLs within 150 m with an F-value of 91.8%. This f-value is the recognition rate in one frame. When approaching an intersection from a distance of 150 m, recognition process of about 100 frames is performed, then the state of the intersection can be estimated with high confidence. The evaluations verify the following as the contributions of the work.Robust recognition was achieved by integrating multiple types of detection methods to recognize TLs including the small size of objects with a few pixels.Arrow TL recognition using prior information obtained from the HD map was proposed, and it was shown that small arrow object can be detected even if their size is smaller than 10 pixel.It was verified that the proposed method satisfies the necessary requirements for smooth deceleration of approximately 0.2 G at intersection approaches in urban automated driving.In the arrow recognition by the proposed method, the arrangement pattern of the signal light and the arrow light given in


This paper presents two methods for the efficient evaluation of the power balance in circular metasurface (MTS) antennas implementing arbitrary modulated surface impedances on a grounded dielectric slab. Both methods assume the surface current in the homogenized MTS to be known. The first technique relies on the surface current expansion with Fourier-Bessel basis functions (FBBF) and proceeds by integration of the Poynting vector on a closed surface. The second method is based on the evaluation of the residue of the electric field spectrum at the surface-wave (SW) pole, and is demonstrated by using a current expansion in Gaussian ring basis functions (GRBF). The surface current expansions can be directly obtained either by analyzing the antenna with a Method of Moments (MoM) tool for homogenized MTSs based on FBBF or GRBF, or derived by a projection process. From there, the power contributions, namely the total power delivered by the feed, the radiated power, the SW power, and the Ohmic power losses in the dielectric are computed. Several efficiency metrics are presented and discussed: tapering efficiency, conversion efficiency, loss factor, and diffraction factor. Since the MTS apertures at hand are leaky-wave (LW) antennas, the designer must find a compromise between the aperture efficiency and the conversion efficiency. This requires accurate and fast computational techniques for the efficiency. The present paper demonstrates for the first time that the efficiency of MTS antenna devices can be accurately evaluated in a few minutes. The compromise that should be made during the design process between the tapering efficiency and the conversion efficiency is highlighted. The impact on the efficiency of isotropic versus anisotropic MTS, uniform versus non-uniform modulation index, is analyzed. An excellent agreement is obtained between both approaches, commercial software, and experimental data. The SW is in general launched by a transverse magnetic (TM) feed, such as a simple monopole placed at the center of the MTS. The theoretical premises of this technology go back to the work of Oliner and Hessel17, in which a rigorous treatment of SW propagation over sinusoidally modulated impenetrable reactive boundary conditions was provided. Over the past decade, this work has inspired many researchers in the antenna community and has led to practical devices with high performance18. For example, the introduction of anisotropy in the surface reactance in combination with a variation of the modulation depth and the local period have been used to synthesize a wide range of aperture fields with excellent beampolarization purity and tapering efficiency20. In addition, very general reactance modulation  based on the solution of the electric field integral equation (EFIE) has been proposed recently to improve the beam shaping capability21 as well as multibeam operation23. After the synthesis of the required impedance modulation, the surface impedance is usually implemented using sub-wavelength patches, printed on a grounded dielectric slab24. A progressive change of the parameters describing the patches  allows one to mimic the required impedance on a Cartesian regular lattice. This implementation can be carried out quite accurately as can be seen from the comparison of the surface reactance and the physical MTS structure simulations26. This means that the antenna performance can be accurately predicted at the homogenized (surface impedance) level, which is much more computationally efficient, as demonstrated by the EFIE techniques in27. Those techniques provide an excellent computational performance through the discretization of the EFIE involving the grounded substrate Green’s function28 kernel, and using appropriate entire-domain basis functions. However, the usage of the substrate Green’s function assumes the substrate to be infinite, and therefore does not properly deal with possible reflection/diffraction effects that stem from the substrate truncation, when the SW is not sufficiently attenuated before reaching the substrate boundary. Diffraction and reflection occurring at the substrate’s edges may produce sidelobes which are not predicted by the simulation models based on the infinite substrate Green’s function when a significant power level reaches the MTS rim. Ohmic power losses may also be non negligible at high frequencies and should be incorporated in the analysis. It is therefore relevant to have a clear understanding, during the design process, of the power balance in MTS antennas.Among different types of metasurfaces (MTSs)29, the efficiency of MTS antennas has been treated with the flat optics (FO) formalism30. However, this approach, although providing closed-form formulas, is limited to locally sinusoidal reactance modulation. The present paper aims at providing two efficient semi-analytical methods for the rapid and accurate evaluation of the efficiency of MTS antennas. The proposed methods are not necessarily limited to locally sinusoidal modulations. That means, the methods can handle arbitrary surface impedance profiles i.e. based on any synthesis method23. The derivation will rely on the mathematical formalisms introduced in27, where the MTS is modeled as a sheet (penetrable) impedance on top of the substrate. This modeling allows one to finely take into account the substrate dispersion, and therefore provides more accurate results in comparison with the opaque (impenetrable) impedance modeling25. In27, the current distribution has been expanded into Fourier-bessel basis functions (FBBFs), while26 uses Gaussian ring basis functions (GRBFs). The choice of those bases in the present paper, besides the fact they provide a fast MoM solution, is justified by their complementary properties. Along the radial coordinate, while FBBFs exhibit selectivity in the spectral domain, GRBFs are selective in space domain. Therefore, they are complementary in describing the fields features. In this paper, FBBFs are used to compute the total power delivered by the feed on the basis of the Poynting theorem. GRBFs current are exploited to compute the SW power flowing beyond the MTS using the residue theorem. Therefore, developing both approaches provides a good cross-validation methodology.InLet us consider a circular-domain MTS printed on a grounded dielectric slab, as depicted in Fig. Two different power descriptions are adopted depending on whether the substrate is truncated Fig. a or infientclass1pt{minima29.When the grounded substrate is infinite, it is convenient to use a surface of integration like the one in Fig. The first goal of the present paper consists in evaluating the power contributions, i.e., estimating 27 or Gaussian ring basis2627 for the analysis of the directivity of MTS antennas printed on an infinite grounded slab. One should note that both families of functions are defined in closed-form in spatial domain and admit closed-form Fourier transforms.In the following, it is assumed that each component of the surface current 26, Expression (42)] or FBBFs .The power radiated by the MTS is derived by integrating the Poynting vector over an infinitely extended disk just above the MTS layer (x-y plane). The spectral domain electric field is denoted as a is the radius of the MTS, this can be divided into two contributions: the radiated field and the SW field. The latter is given by the contribution of the residue at the SW pole of the bare grounded slab, h is the slab thickness, 26, Expression (3)].This contribution is relevant to the power of the SW that proceeds beyond the MTS. In the infinite-slab model this power is trapped in the slab, and does not contribute to radiation. However, in practical realizations it is reflected or diffracted at the slab truncation, and therefore contributes to the radiation pattern in an uncontrolled manner. For this reason, it is important to minimize this trapped SW power, during the antenna design. In the following, a closed-form expression is derived to evaluate this contribution as a function of previously computed currents. To this end, suppose that the z-component of the spectrum of the aperture electric field is a regular function that can be written as a Fourier series versus spectral angle out from  if GRBFst, Re stand respectively for the tangential part, the complex conjugate operator, and the real part. V is the volume enclosed by the surface Referring to the Poynting theorem, the total power delivered by a matched feeding source in absence of losses can be computed after integrating the Poynting vector along a closed surface comprising the source. We propose to use the closed surface represented in dotted red lines in Fig. 27. Since FBBFs are orthogonal, the bases coefficients K is a normalization factor . The aperture tangential magnetic field tested with FBBFs is computed as27.The MTS contribution is explicitly rewritten as:Similarly to the electric field, the magnetic field coefficients into FBBFs are obtained from the tested fields through renormalization. The aperture field is therefore written in the FBBF basis asressions , and usiz have been omitted. Expression . Details regarding the fields expansion procedure into Fourier Harmonics can be found in the supplementary material as well as in Ref.31, Section 7.3.2.To efficiently compute azimuthally the rim contributions, the key issue consists of expressing the azimuthal variation of the fields in  through 29. Considering a substrate with electric conductivity a, over a height corresponding to the substrate thickness h. In order to efficiently compute the electric field in the substrate, we have defined a substrate interaction matrix z) of radius a. The interaction matrix 27. Therefore, the total electric field at a given height z in the substrate, tested with the complex conjugate of the FBBFs This section is devoted to the Ohmic power losses evaluation in the MTS substrate. It is assumed that the losses in the metallization are negligible compared to the substrate loss, which is a good assumption at frequencies lower than 30 GHz term in , where t loss in  can be rFrom the previous sections, the total power delivered by the feed can be calculated in two different ways i.e29, the conversion efficiency is defined on the basis of the total radiated power, i.e. the feed efficiency is not separately treated here.The conversion efficiency The tapering (aperture) efficiency The loss factor The diffraction factor There is no universal figure of merit to define the efficiency of an antenna, since the compromise between the previous metrics will depend on the application of interest. In this paper, we define a “compound efficiency” as the product of the conversion efficiency with the tapering efficiency:This section provides numerical and experimental validations of the efficiency computation methods presented in the previous sections.26p is defined as M is the modulation index (depth). The MTS radius is fixed to First, we consider a broadside beam MTS antenna operating at 17 GHz, modeled as an impenetrable scalar impedance given byM, computed with the FBBF and the GRBF based methods. The conversion efficiency obtained with the commercial software FEKO32 is reported in the same figure. An excellent agreement is observed between the three approaches, despite being based on completely different basis functions (FEKO uses Rao-Wilton-Glisson (RWG)33 basis functions) and efficiency computation methods. At M because the tapering efficiency takes into account the space waves from the feeder. Indeed, for small modulation depths, the radiation contribution is dominated by those space waves, which are not present at broadside because the feeder is a TM vertical dipole. Increasing the modulation depth, the conversion efficiency improves, but the tapering efficiency reaches a maximum at approximately M, the tapering efficiency starts to decrease because the uniformity of the tapering illumination rapidly degrades.Figure 27. Anyway, a trend similar to that of the isotropic MTS is observed. However, the tapering efficiency is better in comparison with the isotropic MTS case owing to a better polarization purity11. The overall (compound) efficiency is maximum approximately at 7; in that case, it provides a rapid way to rigorously check the power balance obtained with holographic theory.A broadside beam anisotropic MTS antenna implementing a sheet impedance modulation is now considered.Now, losses are introduced in the substrate through a dissipation factor 18. This means that the modulation index increases with respect to the radial coordinate so as to maintain a quasi-uniform aperture field illumination  a conversion efficiency of about The previous example shows that one can reach see Fig. a. Such a7, or the more accurate Flat optics19, or the direct inversion method21. Whenever the antenna is synthesized, the two formulations presented here provide a rapid algorithm to rigorously check the power balance provided by any of the above mentioned methods. The first formulation is based on the integration of the Poynting vector along a surface enclosing the feed while the second approach relies on the residue evaluation of the electric field at the substrate SW pole. Those algorithms can cope with arbitrary anisotropic surface impedance modulation and provide accurate results in a few minutes. Although based on completely different formalisms and basis functions, those methods provide approximately the same efficiency results and those results have been confirmed with commercial software and experimental data. The method based on the residue theorem is more analytical and, hence, faster than the Poynting vector based method. Conversely, the Poynting vector based method may be applied to a more general class of problems. Progressive improvement of the antenna compound efficiency is illustrated from isotropic MTS with uniform modulation index  as with FBBF and GRBFs. The FEKO simulations have been performed using the impedance boundary condition (IBC) available in this software (SK card and user defined surface impedance). Hence, all the results have been obtained using the same IBC and integral equation, for a fair comparison. Figure Supplementary material 1


Aimed at improving upon the disadvantages of the single centralized Kalman filter for integrated navigation, including its fragile robustness and low solution accuracy, a nonlinear double model based on the improved decentralized federated extended Kalman filter (EKF) for integrated navigation is proposed. The multisensor error model is established and simplified in this paper according to the near-ground short distance navigation applications of small unmanned aerial vehicles (UAVs). In order to overcome the centralized Kalman filter that is used in the linear Gaussian system, the improved federated EKF is designed for multisensor-integrated navigation. Subsequently, because of the navigation requirements of UAVs, especially for the attitude solution accuracy, this paper presents a nonlinear double model that consists of the nonlinear attitude heading reference system (AHRS) model and nonlinear strapdown inertial navigation system (SINS)/GPS-integrated navigation model. Moreover, the common state parameters of the nonlinear double model are optimized by the federated filter to obtain a better attitude. The proposed algorithm is compared with multisensor complementary filtering (MSCF) and multisensor EKF (MSEKF) using collected flight sensors data. The simulation and experimental tests demonstrate that the proposed algorithm has a good robustness and state estimation solution accuracy. Unmanned aerial vehicles (UAV) have taken a prominent role in a variety of applications, mostly military but also civilian. There are many significant advantages to using UAVs, such as for aerial surveillance , three-dIn the research of UAV navigation technology, the inertial navigation system (INS) and global satellite navigation system (GNSS) are indispensable. The INS ,14, whicNowadays, the Kalman filter ,21, whicEverything described above refers to centralized nonlinear navigation filters, which have several drawbacks. These include the heavy computational burden, the high state dimension, and the uncertain filter divergence. Compared with centralized filters, decentralized filters  use the In this paper, with a focus on model nonlinearity and state solution accuracy, a nonlinear double-model multisensor-integrated navigation method is proposed based on the federated extended Kalman filter fusion framework. A nonlinear double-model system consists of the nonlinear attitude heading reference system (AHRS) model and the nonlinear SINS/GPS model. The nonlinear attitude heading reference system model was used to perform the first local EKF sensor fusion to estimate the attitude error Equation . The nonEquation . Then, tEquation ) of the 1. The multisensor error mathematical model, which includes the gyroscope error model, the accelerometer error model, and the SINS error model, are described in this paper. Furthermore, these error models are reasonably simplified to apply to small-UAV near-ground short distance navigation;2. A federated EKF approach is proposed combined with the federated filter and EKF, which not only solves the model nonlinearity, but also retains excellent filter precision;3. During UAV flight, the requirement of attitude solution accuracy is of great importance. Thus, a nonlinear double model is proposed, including the nonlinear AHRS model and the nonlinear SINS/GPS model. The common attitude solutions of the nonlinear double model are fused in the main filter of the federated filter.The rest of this paper is organized as follows. In The zero offset bias is the main influence factor of the gyroscope output precision when the UAV is in a static situation. Therefore, eliminating the zero offset error can improve the angular velocity accuracy of the gyroscope measurement. However, the random drift error of the gyroscope will change when the UAV is in flight, so the gyroscope error needs to be modeled  and estiEquation .(2)ε˙r=Equation .(3)ε˙0=Therefore, the constant error can be taken into account in the random drift error. The gyroscope error model  can be dTypically, the error of the accelerometer includes the constant error and random drift error. Constant error can be eliminated using sensor calibration . HoweverEquation .6)▽˙r=▽a=▽0+▽r+▽˙r=▽a=▽0The attitude of the SINS error model  can be wThe velocity of the SINS error model  can be rThe position of the SINS error model  can be dIn this paper, MEMS IMU sensors are used, which have a large noise compared to high-precision inertial devices. At the same time, the error state of the system only changes slightly under filter conditions, when the state of the SINS error equation selects the error quantity. Moreover, the SINS error model is quite complex in terms of computational complexity for the study of the embedded MEMS sensor fusion algorithm. Thus, there are some reasonable simplifications for the SINS error model in order to decrease the computational complexity and improve the update frequency of the navigation algorithm.There is the pretty low height and relatively slow flight speed when a small UAV is in flight, and the update frequency of the proposed algorithm is high when the airborne CPU is running. Therefore, the rotation of navigation system caused by the change in the position of the UAV can be ignored during the algorithm running period. In addition, the measurement accuracy of the earth rotation angular rate is small relative to the gyroscope, and so the earth rotation angular rate is considered to be 0.The UAV coordinate system is shown in b is the body system, n is the navigation system, and The nonlinear error model of Equations –10) is  is 10) iThe Kalman filter problem generally is assumed to be linear in the mathematical model. However, the model in practical applications is a nonlinear system. In this paper, the designed method relies on the extended Kalman filter  for the In Equation , the jacThere are generally two optimal combining methods for a multisensor-integrated navigation system: centralized KF and decentralized KF, respectively. Theoretically, centralized KF, which uses single KF to centrally process all the subsystem information of navigation, can give an optimal estimation of the state. As a result of the high state estimation dimension in practice, the computation is too large to run in real time for the algorithm. In addition, the centralized KF has a poor fault tolerance and is not conductive to fault diagnoses. However, the decentralized KF can use multiple filters to make estimations for the different subsystems of navigation so as to reduce error coupling. There are many decentralized KF algorithms currently. For example, an approach of decentralized algorithms named the federated filter was proposed by Carlson ,33. The The federated filter is often seen as a two-stage filter structure, as is shown in ith state estimation and covariance of the local filter, respectively.If there exists n local state estimations timation  can be eThe EKF has a good processing ability for nonlinear problems, and it is also easier to implement in practical applications. In addition, the federated filter has good fault tolerance compared to centralized KF. Moreover, the fault can be easily detected and isolated when the subsystem fails, then rest of the systems can be reconstructed. Thus, the proposed algorithm for filter precision is large and the computation of the fusion algorithm from the local filter to global filter is small, which is beneficial to execute in real time.i is the ith local EKF, Aimed at the application of multisensor fusion, the improved federated EKF algorithm is proposed in this paper, which uses EKF to optimize the nonlinear model of the local filter, and then globally optimizes for the common state of the local EKF in order to meet the requirements of autonomous position navigation. Additionally, it can improve flight stability and reliability. Equation  is the ii is the ith nonlinear model, Typically, this should be mathematically modeled when the actual system with nonlinear characteristics is described. Therefore, a unified nonlinear Gaussian state space model system is established for the multimodel of the federated filter based on the nonlinear theory .(17)XkiThe AHRS can affect UAV flight stability, which is regulated by fusing the gyroscope, accelerometer, and magnetometer. According to the multisensor error mathematical model in The measurement function of the AHRS can be written by Equation .(19)Zk1This paper uses the low-cost IMU that causes the state estimation to diverge over long running times. Thus, the GPS is selected as another subsystem of the multisensor-integrated navigation system of the UAV. The GPS, which has a good long-term stability, calculates the velocity and position of the carrier by utilizing the received satellite navigation signals. It can be seen that the SINS/GPS is the current mainstream choice because of the complementary performance of SINS and GPS. However, it is necessary to establish a nonlinear mathematical model that accurately reflects the system in order to better analyze the actual SINS/GPS system.The measurement function of the SINS/GPS can be established by Equation .(21)Zk2The common state parameters of the nonlinear double model, which are the error of attitude quations  and 23)δφn and gThe error of attitude and the gyroscope bias of Equations  and 23)23) are tThe unique state parameter of the nonlinear SINS/GPS-integrated navigation model is used as part of the main filter state output in the federated filter framework.The local EKF and federated filter were introduced in The establishment of the federated double-model EKF algorithm can make state estimation more accurate, because the UAV requires higher accuracy for attitude calculations compared to ground robots. This problem is effectively solved by the nonlinear double model in this paper. The decentralized multimodel can globally optimize the common state (error of attitude and gyroscope bias) of the two nonlinear models, which is suitable for the rapid maneuvering of the UAV. The traditional SINS/GPS-integrated navigation diverges when the GPS signal is interfered with or invalid; however, the algorithm in this paper reconstructs the rest of the sensor information in the case of GPS failure, which can guarantee that the normal attitude is provided to the UAV. The federated double-model EKF fusion diagram is shown in The nonlinear AHRS local EKF system includes three kinds of sensors: gyroscope, accelerometer, and magnetometer. As shown in The nonlinear SINS/GPS model system includes three kinds of sensors: the gyroscope, accelerometer, and GPS. As shown in Equation  to obtaiEquation  to obtaiThis paper used the experimental platform shown in The flight sensors data were collected with the GPS of the UAV using the experimental platform shown in As can be seen in the previous figures, the attitude error of MSCF is the largest compared with the other two algorithms in Equation  (i=2), bIt can be seen that the attitude of MSCF is influenced by the sensor noise from the diagram of the UAV attitude in It can be seen that the FDMEKF can better track the velocity of the truth compared with the velocity of MSCF and the MSEKF in It can be found that the FDMEKF can better track the position of truth compared with the position of MSCF and the MSEKF in The flight sensors data without GPS of the UAV were collected by the flight platform as shown in The velocity and position of the state estimation parameters are inaccurate when GPS signal is disturbed in city buildings and jungle environment. At this time, the reliable attitude solution is very important when the UAV is in flight. It can be seen that the attitude curves of the FDMEKF are closest to the truth compared with the MSCF and MSEKF, as shown in A nonlinear double model approach based on the decentralized federated extended Kalman filter (EKF) is proposed in this paper for the multisensor fusion algorithm of small UAVs. The contributions of this paper include the following: (1) The multisensor error mathematical model is established and simplified to a reasonable extent; (2) the federated EKF is further developed to enhance the filter accuracy and robustness; (3) a novel nonlinear double model is designed to obtain a more accurate attitude solution for cases in which there is GPS and there is not. Simulations and experimental test results demonstrate that the proposed filter algorithm can effectively provide the state estimation in order to meet the flight requirements of a UAV, as well as having a much higher solution accuracy.


Gait phase detection, which detects foot-contact and foot-off states during walking, is important for various applications, such as synchronous robotic assistance and health monitoring. Gait phase detection systems have been proposed with various wearable devices, sensing inertial, electromyography, or force myography information. In this paper, we present a novel gait phase detection system with static standing-based calibration using muscle deformation information. The gait phase detection algorithm can be calibrated within a short time using muscle deformation data by standing in several postures; it is not necessary to collect data while walking for calibration. A logistic regression algorithm is used as the machine learning algorithm, and the probability output is adjusted based on the angular velocity of the sensor. An experiment is performed with 10 subjects, and the detection accuracy of foot-contact and foot-off states is evaluated using video data for each subject. The median accuracy is approximately 90% during walking based on calibration for 60 s, which shows the feasibility of the static standing-based calibration method using muscle deformation information for foot-contact and foot-off state detection. Chronic diseases caused by ageing are associated with a decline in walking ability ,2. EarlyGait can be classified into two phases: stance and swing. As shown in ® system [Gait phase detection methods have been investigated with sensing techniques based on foot contact force or lower limb kinematics for decades. One of the representative sensors for gait phase detection is the force-sensing resistor (FSR), which is a thin film force sensor that can be used as a foot switch. Classical methods set a threshold using FSRs ,10,11, t® system , or inso® system . To ensu® system ,15. Usin® system ,17. Howe® system ,18.In addition to FSRs, different types of sensors have been used to provide kinematic information for gait phase detection while avoiding foot-contact impact to the sensors. For example, inertial measurement units (IMUs) have been widely used with HMM ,20,21, sSeveral approaches are available for gait phase detection without obtaining training data. An auto-calibration method using a planar structure in the angular space of leg joints was proposed  to detecIn this study, we focused on muscle deformation during walking to construct a novel method of gait phase detection. The objective of this study was to establish a gait phase detection system for foot-contact/foot-off state detection based on muscle deformation using a machine learning algorithm with fast and easy semi-automatic calibration and does not require manual labeling using gait data. Our hypothesis was that static standing-based calibration performed for a short time, just before walking, enables the user to use the calibrated sensor system for gait phase detection while avoiding labeling work. Our previous research demonstrated the potential of muscle deformation information sensed by only one sensor for gait phase detection with calibration using static standing posture data . The hypCalf muscle deformation data were monitored because calf muscles, which are related to ankle movement, have a strong relationship with foot-contact information . We inveEstablishing an easy and fast semi-automatic calibration method for gait phase detection using discrete static standing data with muscle deformation information;Enabling the use of a gait phase detection system with only one sensor.Conducting experiments with the novel gait phase detection system on humans.The contributions of this work are as follows:In this section, we present a detection system for foot-contact/foot-off based on LR, the probability of which is adjusted using the leg kinematic information with static standing-based calibration using muscle deformation information. FirstVR   was usedOur system is based on a machine learning algorithm with static standing-based calibration for the binary classification of gait phases. Calf muscle deformation data were obtained while standing with several postures of gait motion, which trained the machine learning algorithm. Muscle deformation is caused by muscle contraction when a joint is moved or weight is applied. Because the calf muscle acts in conjunction with foot motion, training standing data corresponding to foot position were selected. Here, 7 postures were considered as training data options in terms of foot position (backside or front-side) and foot-contact/foot-off states, as shown in LR is used for the binary classification algorithm of the gait cycle with static standing-based calibration in this study. Because the algorithm should be trained quickly to ensure usability, and the amount of input data is limited, a deep learning method that is adaptable to the complicated structures of huge inputs and requires time to be trained is not appropriate. SVM can be considered as a high-ability binary classification algorithm. However, because the classification ability of SVM relies on hyperparameters related to the penalty weight of the error and the scale of the feature space, which can differ between users, classifying gait muscle deformation data would be difficult using SVM in a static standing-based calibration method, where cross-validation (hyperparameter tuning) cannot be performed for each individual. However, LR does not adjust the scale and penalty weight as hyperparameters. Furthermore, LR can output the probability of respective classes. Even though the muscle deformation state during standing might be similar to the muscle deformation state in some phases during walking, it is challenging for the decision boundary to be perfectly consistent between standing and walking. Therefore, we determined that probabilistic consideration was required. Deriving probability from muscle deformation information enables the system to reflect another type of sensor information for the adjustment of probability. In this study, we used the simple information of leg movement obtained with FirstVR for adjustment of probability.w and the input vector x, which is derived for solving a linear regression problem. The LR algorithm calculates the regression value by assigning this inner product to the sigmoid function, which is obtained from 0 to 1 as probability P as:LR is an algorithm that classifies input vectors into two classes using probability. The probability is calculated based on the inner product value of the weight vector P is 0.5; that is, P was more than 0.5 and as the stance phase if P was less than 0.5.The decision boundary is a region of values where Because the decision boundary might slightly differ between standing and walking data, the gyroscope and magnetometer values from FirstVR were used for adjusting the decision boundary. The probability of the swing phase is higher when the leg moves in the anterior direction of the body. Therefore, the size of the internal product based on  as:(2)ϕθP was derived as:a is the adjustment value, a, Next, the angular velocity in the spatial coordination system was calculated from the angular velocity We conducted an experiment to evaluate the detection ability of the proposed gait phase detection system based on static standing-based calibration. The gait data were classified into two labels: the stance (foot-contact state) and swing (foot-off state) phases. We recruited 10 subjects, compared the estimated labels and ground-truth label obtained from video, and evaluated the effect of the standing data combination, algorithm, and data amount on the detection accuracy.Ten healthy adults  where the Euclidean distance was the smallest for each posture was counted in each gait cycle. The percentages of the counted closest labels were derived by dividing the counted numbers of stance and swing phases by the total number of gait cycles in each posture. The strength of the relationship between the phase and posture data increased with the percentage of labels in the posture data.N denote the number of correctly classified samples and the total number of evaluation samples, respectively. We manually annotated the ground-truth labels  by viewing the video. Frames in which the heel of the right foot contacted the treadmill and the toe of the right foot left the treadmill were extracted by visual observation. The frames from the toe-off to the heel-contact and from the heel-contact to the toe-off were labeled as the swing phase and stance phase, respectively. Among the 14 channels of FirstVR, 11 were used as model inputs because the other 3 channels did not work. The model was trained with the standing data  were used as the detection algorithm. These algorithms were implemented in Python using a scikit-learn library. The LR solver was newton-cg and the other hyperparameters were the default. The best Equation  was consFirst, the training data combination required for gait phase detection was investigated. Next, the effect of algorithms of LR, SVM, and LR-a on the detection accuracy was evaluated. To consider the duration that people should stand for calibration, the detection results for data numbers of 500, 1000, 1500, and 2000 for each posture were compared. The Friedman test in the MATLAB toolbox, which is a non-parametric version of analysis of variance, was performed to evaluate the effect of algorithms and data amount. The significance level was set to 0.05.The detection results for each phase were evaluated using experimental conditions for accuracy. True positive rates for the swing and stance phases were derived. The true positive rate is the proportion of frames that were correctly identified as positive, which is calculated as:All conditions included the postures of the back and front heel contacts because the muscle deformation vectors of these postures were close to those of the stance or swing phase, respectively. We assumed that these two postures were key factors for identifying foot-contact and foot-off states during walking. The postures of the back heel-off and front foot-off, whose muscle deformation vectors were farthest from those of the swing or stance phase, could be factors that reduce the detection accuracy. However, the conditions including the posture of back foot-off, such as condition 9, tended to result in higher accuracy, except for condition 6. Because of the arbitrary nature of human posture, differences may exist in the posture between subjects. Even if the position of the foot of a certain posture is slightly different, the resulting change in muscle deformation status may be sufficiently large to affect the accuracy of gait phase detection. For several subjects, the accuracy tended to be lower when only the muscle deformation data of the back foot-off posture was used as the training data for the swing phase. We assumed that foot position during standing in the back foot-off of these subjects was different from that of other subjects. This might reduce the number of postures required for calibration if the arbitrary posture can be defined in more detail. However, three to five postures were required as training data to compensate for the arbitrary values in this work. Because the median accuracy in condition 9 was the highest, we compared the effect of the differences in the algorithm and training data number using condition 9.This study is the first to examine a gait phase detection method with static standing-based calibration using muscle deformation information. The static standing-based calibration enabled the system to be calibrated without walking. As shown in However, our proposed system has several advantages. The proposed system requires only one sensor, which enables the users to easily attach the sensor to their legs. Because the amount of data did not significantly affect the accuracy, the required amount of muscle deformation data for each posture was 10 s in this study. The calibration could be performed by simply standing in six postures. As the time to put on of the sensor was less than 10 s, the total preparation time for using gait phase detection was less than 70 s. Therefore, the proposed method is valid for daily use for gait phase detection.The classification algorithm is a computational method for drawing a decision boundary in the data space. SVM applies the concept of margin maximization to determine the decision boundary. The margin is the distance between the decision boundary and the data, and margin maximization involves finding the boundary that maximizes the distance between the boundary and the data point closest to the boundary, called the support vector. The decision boundary can be obtained so that the classification performance can be improved to some extent while allowing for classification errors. Thus, the classification is robust against slight differences between standing and walking data. However, it is necessary to adjust the penalty weight to the error and the scale of the data as hyperparameters. Because the values of the hyperparameters varied among the subjects, it is difficult to achieve the same level of accuracy in this experiment in an actual-use environment where cross-validation is impossible. Logistics regression, which does not require hyperparameter adjustment, had significantly lower detection accuracy than SVM. However, the accuracy of LR-a, with the addition of probability adjustment based on gyroscope data, is equal to or better than that of SVM. The adjustment probability could reduce the error in the decision boundaries between the postures and the stance/swing phases. In this study, we applied simple information about the direction of foot movement for probability adjustment. Using the IMU-based rules in ,33 for pWe observed individual differences in the detection accuracy. In particular, the accuracy for the female subjects tended to be lower, as shown in The main feature of the system is its static standing-based calibration using a muscle deformation sensor. The calf muscle deformation information is useful for obtaining the training dataset without walking. We assumed that the foot-contact/foot-off states were detected because the calf muscle deformation changed depending on whether body weight was applied to the foot or not. Therefore, the proposed system might be used in various walking environments, such as steps and uneven terrain. We think that standing-based calibration can also be applied with force myography or ultrasound. However, we think that EMG cannot be applied for this calibration method because of the inconsistency between measurement results based on force myography and EMG . The maiOne of the limitations of this study is that the current sensor usability relies on individual muscle mass. Muscle deformation information tended to be difficult to obtain in three female subjects who were considered to have low muscle mass. An increase in the number of photo reflectors might be beneficial for application in diverse subjects. In addition, the arbitrary nature of postures is another limitation. Understanding the change in muscle deformation caused by slight differences in foot position during standing in the same posture is beneficial. Furthermore, the low sample frequency of the current system limited detection accuracy. To achieve higher detection accuracy, a system should be constructed with shorter processing time and faster communication speed. In this experiment, the model did not need to include a directional extraction calculation because the anterior direction of the body (east direction) was constant and known. In practice, because people walk in any direction, this calculation will be necessary in the future.In this study, we constructed a novel gait phase detection system with static standing-based calibration using muscle deformation information. A logistic regression algorithm was used as the machine learning algorithm, and the probability output was adjusted based on the angular velocity of the sensor. An experiment was performed with 10 subjects, and the detection accuracy of foot-contact and foot-off states was evaluated using video data for each subject. The median accuracy was 90% during walking based on calibration for 60 s, which demonstrated the feasibility of the static standing-based calibration method using muscle deformation information as an alternative and fast calibration method using walking data for foot-contact and foot-off state detection. There were individual differences in detection accuracy. In particular, the accuracy for the female subjects tended to be lower. Conversely, for six male subjects whose muscle state was easily observed by the sensor, the gait phase could be classified into the stance and swing phases with higher accuracy (approximately 0.05 s error) using only muscle deformation data. When the muscle deformation information was obtained clearly by the sensor, the proposed system produced similar detection accuracy as some previous studies where the calibration was performed using gait data as teaching data.An increase in the number of photo reflectors would be beneficial for application in diverse subjects in the future. In addition, understanding the change in muscle deformation caused by slight differences in foot positioning during standing in the same posture wold be beneficial. Furthermore, the low sample frequency of the current system is limiting detection accuracy. To increase detection accuracy, a system should be constructed with shorter processing time and faster communication speed. In this experiment, the model did not need to include a directional extraction calculation because the anterior direction of the body (east direction) was constant and known. In practice, because people walk in any direction, this calculation will be necessary in the future.


A real-time classification method has been implemented for online classification of explored surfaces. In contrast to previous techniques, which mainly rely on visual modality, the proposed approach based on optical fibers might be more suitable for operation in extreme environments (such as nuclear facilities) where radiation may damage electronic components of commonly employed sensing devices, such as standard force sensors based on strain gauges and video cameras.This paper demonstrates how tactile and proximity sensing can be used to perform automatic mechanical fractures detection (surface cracks). For this purpose, a custom-designed integrated tactile and proximity sensor has been implemented. With the help of fiber optics, the sensor measures the deformation of its body, when interacting with the physical environment, and the distance to the environment's objects. This sensor slides across different surfaces and records data which are then analyzed to detect and classify fractures and other mechanical features. The proposed method implements machine learning techniques . An average crack detection accuracy of ~94% and width classification accuracy of ~80% is achieved. Kruskal-Wallis results ( An important task often performed in remote hazardous environments is the detection of mechanical fractures on the objects, such as containers, tanks, pipes, and other technical systems used for keeping chemical and radioactive waste. A crack may be caused by physical damage or material degradation over time or environment changes . The effects of non-detected fractures may lead to larger macro-scale catastrophic failures making the cracked surface mechanically weak to perform its function.Conventional automatic crack detection methods applied in industry to inspect large mechanical structures rely on acoustic methods  as shown in In this work, machine learning techniques are employed for crack detection and crack width classification. A set of 10 objects with different surfaces  were manufactured with PLA plastic using 3D printing technology . The wavy pattern consists of a repeated pattern of sine waves of 1mm amplitude and 5mm magnitude. The samples are shown in The data-set generated and used in this study is publicly available on the figshare repository “Automatic Fracture Database”The goal of the proposed algorithm is to detect and characterize mechanical fractures, such as cracks, based on the deformation and proximity data recorded from the sensors. The time history of the deformation and proximity data is recorded. Preprocessing step and features extraction are performed. The resultant output is used as an input for the classification algorithm.x, 1d, 2d, 3d, and p vectors of size m × 1 representing single trial recordings (time history) of the sensor's displacement, three deformation signals and proximity signal, correspondingly, and m the number of data points in a specific trial. The proximity data (p) of each measurement trial was used to extract the data points corresponding directly to a specific mechanical feature . This allowed to create a subset of data containing only the information specific to the mechanical feature, and to exclude the data points at the start and the end of the recording. This process was performed automatically, based on the analysis of the discrete-time derivative of proximity sensing for a given time window, and extracting the data for which the derivative exceeded a pre-defined threshold. The average discrete-time derivative for proximity measurement was computed aspi a local discrete derivative of ith proximity signal at measurement based on n data points, pi representing ith element of proximity measurement vector, and n = 10 data samples), is sufficient to achieve better good classification accuracy.The goal of preprocessing was to prepare datasets containing information for the mechanical features, such as crack, bump, and wavy pattern and exclude not relevant datapoints . The preprocessed labelled datasets were then used for training process and cross-validation test. The preprocessing was fully automatic and was performed on the data collected from the haptic manipulator (sensor displacement) and optical sensing elements integrated in the sensor . The sensor's position data (obtained from the Geomagic haptic device) and the sensing elements data  were synchronized and sampled at 400 Hz. Prior the prepossessing stage the measurements of each trial  were arranged in the following matrix:Feature extraction was performed on each successive 25 ms time window with an increment of 5 ms. The size of the time window was selected based on the sampling frequency. Feature extraction is executed on windows of 10 data points with a window shift of 2 data points. The window length was empirically chosen through a grid search analysis. Time-domain features, including Mean Absolute Value (MAV) and Root Mean Square (RMS), were computed. The advantage of time-domain features is that they are fast to calculate since they do not require any mathematical transformation, e.g., into the frequency domain. On the other hand, they are sensitive to noise. These feature demonstrated high performance in previous surface Electromyography (sEMG) works of  with five neighbors, and Quadratic Discriminant Analysis (QDA). Random Forest classifier Breiman,  can succp < 0.001) indicates that the null hypothesis of having all data samples from the same distribution is rejected. Thus, there are significant differences between the implemented features.The goal of the Crack Recognition experiment is to recognize the presence of a crack in the object. p < 0.001) indicate statistically significant differences among results obtained when analyzing only deformation signals, only proximity data and both deformation and proximity data. In this case, instead of using a classifier, a regressor may be more appropriate to use since having a discrete class may not be the best solution when predicting the width of a fracture.The scope of the crack width classification experiment is to classify the width in millimeters (mm) of the fracture of the explored object. To further test the result of the classifier an online application was developed for the crack recognition analysis. During this experiment, it was found that the fiber optic cables position and their twisting influence the sensor data. Thus, an additional acquisition was necessary to obtain a model to use for the real-time classification. Offline models of Random Forest Classifier, KNN, and QDA were generated implementing the newly acquired data. The models were later used to predict the class of the data acquired in real-time while sliding the sensor over different surfaces. The software marks the start position of the detected crack and end position in relation to the Geomagic position. The same analysis, as the previously described offline classifier, was applied. Three continuous sliding movements were performed on each of the crack type surfaces as shown on the first row of This work demonstrates how tactile and proximity sensing can be efficiently used to perform automatic crack detection. The proposed method uses machine learning techniques to detect the surface fractures and bumps of explored objects based on fiber optical proximity signals which are recorded during physical interaction between a custom-designed robotic finger and the remote environment. Experimental validation of the proposed method has shown that it is possible to achieve around 94% for crack detection and 80% for crack width classification accuracy. To achieve better results for the crack width classification, an alternative regressor may be more appropriate to use with respect to the implemented classifier. Real-time classification results, on three sliding movements, shows that it is possible to correctly characterize the surface of the investigated object. During this experiment, it was found that the fiber optic cables position and their twisting influences the sensor data. Thus, additional analysis will be required to compensate for the change of flow of the data. In contrast to previous techniques, which rely on visual modality, the proposed approach, based on optical fibers, which may be more suitable for operation in extreme environments (such as nuclear facilities) where radiation damages electronic components of video cameras.Future research will focus on integrating a multi-modal approach with visual patches and implementation of the proposed system on a teleoperated mobile manipulation system. We plan to demonstrate how automatic fracture characterization will be efficiently integrated with the mobile manipulator controller (Farkhatdinov and Ryu, https://figshare.com/s/14deb00d874400e34d67.The datasets generated for this study can be found in the figshare repository “Automatic Fracture Database” available at FP developed the classification algorithm, experimental methods, performed the experiments, and wrote the draft of the paper. IF developed the experimental methods, contributed to the data analysis, and provided the input to the paper. KA advised on the experimental methods and corrected the paper. JK and SP assisted with the paper writing. All authors contributed to the article and approved the submitted version.JK was employed by the company Ocado Technology. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.