As a new type of jamming, the interrupted sampling repeater jamming (ISRJ) derived from the digital radio frequency memory (DRFM) technology, can generate coherent multiple false targets after pulse compression. At present, the traditional interference suppression method and its improved methods have insufficient characteristics and poor detection performance under the condition of low signal-to-noise ratio (SNR). Aiming at addressing this defect, this paper proposes an interference suppression method for ISRJ based on singular spectrum entropy function (SSEF) from the aspects of singular value decomposition (SVD) and information entropy theories. In this method, firstly, considering the local fine characteristics and extraction efficiency, an adaptive multi-scale segmentation (AMS) method is proposed. The purpose of this processing is to extend the salient characteristics while to smooth the similar ones. In AMS, the segmentation criterion based on average energy of segments and the constraint of minimum segmentation is also proposed, then the improved delay embedded matrix is established from the improved trajectory matrix by AMS and delay embedded mapping. Secondly, the singular spectrum of the improved delay embedded matrix is extracted by SVD. Thirdly, because the recognition algorithms based on singular spectrum analysis (SSA), classical SSE and other characteristics fail at low SNR, this paper proposes a characteristic named as SSEF retrieved from the Shannon entropy model. The following proposed entropy-based threshold detection is carried out on the echo signal to realize the band-pass filtering and interference suppression. Finally, experiment results show that in comparison with other interference suppression approaches, SSEF can increase the probability of target detection and the peak-to-side-lobe ratio (PSR) after pulse compression, which validates its stability to noise and jamming especially in the condition of low SNRs. Electronic countermeasure (ECM) techniques such as active deception jamming aim at forming false targets to confuse the victim radar, for surrounding desired radar echoes with so many false targets that the target information cannot be extracted. ECM techniques are often enhanced by using digital radio frequency memory (DRFM) systems. In a DRFM system, the interrupted radar signal is down converted first and sampled with a high-speed analogue-to-digital converter (ADC). The samples stored in memory are manipulated in amplitude, frequency and phase to generate the deception jamming, and subsequently processed by a digital-to-analogue converter (DAC), up converted and transmitted back to the victim radar.As a classic jamming derived from DRFM, the interrupted sampling repeater jamming (ISRJ) is highly correlated to the radar signal which makes it prone to generate multiple false targets after pulse compression . MeanwhiIn comparison with jamming technology, the anti-jamming technique turns out to be a more arduous task due to the shortage of prior signal information and published researches, as well as military sensitivity and no related theoretical frameworks and mathematical models . DespiteIn summary, most interference suppression methods are based on the idea of characteristics extraction, difference identification and band-pass filter construction. These methods perform well under the high SNR and high JSR condition, but when comes to low SNR, their performances will degrade. Aiming at this defect, this paper proposed an interference suppression method based on SSEF for ISRJ from the aspect of entropy characteristics. Firstly, the improved trajectory matrix is established by adaptive multi-scale segmentation (AMS) to achieve fine characteristics extraction of the echo signal and improves time efficiency. In this step, the segmentation criterion based on average energy of segments and the constraint of minimum segmentation is also propose. Secondly, the improved trajectory matrix is delay embedded mapped to develop the improved delay embedded matrix for further SVD. Thirdly, the singular spectrum of the improved delay embedded matrix is extracted. Based on the SSEF retrieved from the Shannon entropy model, threshold detection is carried out on the echo signal to realize the band-pass filtering and the interference suppression. Finally, experiment results show that in comparison with other interference suppression approaches such as EFDBP, STFT, PF, STMF, the proposed method in this paper can increase the probability of target detection and the peak-to-side-lobe ratio (PSR) after pulse compression, which validates its stability to noise and jamming especially in the condition of low SNRs.The structure of this paper is as follows: in c is velocity of light. The pulse width of the interrupted sampling is In order to introduce jamming more clearly, we take the airborne jammer on the condition of self-screening jamming (SSJ) as an example. In general, the jammer is divided into receiving subsystem and jamming subsystem. The receiving subsystem includes a receive antenna, a receiver and a signal processer while the jamming subsystem includes a transmit antenna, a jamming generator and a power control unit. The jammer receives the radar transmitting signal through the receive antenna after detecting the rising edge of the radar transmitting signal and generates slices by interrupted sampling, storing. Then, after modulation, the ISRJ is transmitted through the transmit antenna to victim radar. By this way, the ISRJ based on DRFM generates main false target and several other false targets located symmetrically around the main false target. The schematic diagram of jammer against radar based on ISRJ is showed in A is the amplitude, T is the pulse width, K is the frequency modulation slope.The ISRJ derived from signal sampling, storage and repeater can generate coherent multiple false targets during the matched filtering of pulse compression. At present, the papers and researches in the field of interference suppression mainly focus on pulse Doppler (PD) radar and linear frequency modulation (LFM) radar. During the process of interference suppression, signal accumulation is needed. The PD signal performs pulse accumulation in the time dimension while LFM signal performs pulse compression, which is equivalent to accumulation. Considering that the bandwidth of LFM signal is relatively wide and the resolution is high, LFM signal has great advantages for target detection . Both waThe interrupted sampling signal is rectangular pulse train which can be written as:After interrupted sampling, the jamming signal can be expressed as:s(t′) represents the echo signal, j(t′) is the jamming signal and n(t′) is the Gaussian noise. The time sequence of the radar receiver has a delay relative to that of the jammer.After DAC, up converted and transmitted back to the victim radar, the echo signal received by the radar receiver can be indicated as:As demonstrated in i = 1, 2, …, n are the sample values of the radar signal with P is the interrupted sampling matrix. P is a diagonal matrix and the values of the diagonal elements are ‘one’ or ‘zero’ . In this paper, d is set as 2 to solve the compromise problem between the calculation burden and the singular value extraction. Then the improved delay embedded mapping can be expressed as:As to ensure that all elements participate in the operation, SVD is an important tool in matrix analysis which reflects the intrinsic nature of matrix vectors in statistics . The maiU, V to realize:According to the SVD theory , for eacThus, the singular spectrum of each improved delay embedded matrix is:The singular values in singular spectrum represent the relevant characteristics contained in segments of radar echo signal. In fact, after a certain singular value, the latter values are very small, which means that the important characteristics are former ones and the rest are secondary characteristics.i-th delay embedded matrix is:j-th singular value in the singular spectrum of the i-th improved delay embedded matrix. This probability represents the proportion of each singular value in singular spectrum. For the noise, since the relative variation of singular values in singular spectrum is small, the probability of each singular value is not much different. According to the principle of entropy, SSE is a large value. For the target signal, the difference between singular values begin to increase due to RCS, which make SSE decrease. But due to the influence of target location and attenuation in transmitting, SSE does not decrease much. And for ISRJ, the amplitude of ISRJ is usually much greater than target signal and noise, so in this case there are only a few large singular values in the entire singular spectrum, which causes the difference between the probability of these large singular values and that of the rest to be much greater. Therefore, the SSE becomes much smaller than that of noise and target.Information entropy is widely applied to measure the complexity of the signal which is proportional to the signal uncertainty . Therefoi,jp should be non-zero. The proof is shown in i-th improved delay embedded matrix:N delay embedded matrix only contains the noise without any components of the target signal and the jamming signal. Empirically, N values from 2 to 10. Here, N = 5.According to Equations (31) and (32), as to ensures that SSEF make sense, Finally, the signal after interference suppression The flow chart of the SSEF-based ISRJ interference suppression method can be summarized as R′. The echo signal sequence R′. The mathematical expression is rewritten here:Step 1: Computing the improved trajectory matrix DD.  is written as:f(•) is the delay embedded mapping defined in Equation (28).Step 2: Computing the improve delay embedded matrix Step 3: Computing the singular spectrum Step 4: Computing the SSEF R′ by entropy-based detection and band-pass filtering. The specific threshold detection algorithm is described below and entropy-based threshold γ is defined in Equation (34). After this step, the interference suppressed signal Step 5: Updating the element In this experiment, the transmitting radar signal is assumed to be LFM signal as shown in Equation (1). Based on the jamming model in We first carry out the simulation comparison experiment by extracting the average energy of segments with fixed-scale segmentation and AMS. In order to highlight the comparison results, the experiment is established under the condition of high SNR. That is because under this condition, both two methods are efficient and the advantage of AMS in fine characteristic can be highlighted successfully. Since the SSEF has stability to noise and jamming, we illustrate the advantages of the SSE between fixed-scale segmentation and AMS under low SNR. Based on the parameters in The echo signal of ISPRJ in time domain is demonstrated in As to compare the performance of different interference suppression methods, experiments are carried out between the proposed method and the STMF, the STFT, the FEDBP and the PF methods. The parameters in the experiments are the same as Results can be first made from the aspect of jamming pattern. For the same jamming pattern, the performances of five interference suppression methods increase with the SNR, which indicates the correctness of this method and the applicability of ISRJ and its improvement pattern. When the SNR is greater than 5 dB, the target detection probability of the five methods after interference suppression is greater than 0.9. This means the proposed method is comparable to the interference suppression ability of the other methods in the condition of high SNR.It can also be concluded from the aspect of SNR that for the same JSR, the detection probability of this method after interference suppression is greatly improved compared with other four methods when the SNR is less than 0 dB . This means that the proposed method has significant advantages of interference suppression and better noise stability under low SNR conditions.For the same SNR, when the JSR is larger than 10 dB, the detection probability after interference suppression increases with the JSR value and the difference between the jamming signal and target signal become larger as well. Meanwhile, it can be observed that in this case the detection probability and the interference suppression performance of the proposed method is higher than other four methods and there is a significant improvement of detection probability when the JSR is between 5 dB to 10 dB.Based on the study of ISRJ principle, this paper proposed an SSEF-based ISRJ interference suppression method aiming at the problem of multiple coherence false targets in low SNRs. This method first establishes the singular spectrum model by proposed AMS, delay embedded mapping and the following SVD. Then based on SSEF derived from the Shannon entropy, this method realizes the detection of SSE function and the band-pass filtering of the echo signal received at the radar antenna. The influence of AMS and delay embedded mapping on the interference suppression performance are analyzed during the model construction. The validity and the feasibility of this method is further proved by mathematical deduction. Through the comparison with other suppression methods, the superiority of this method is finally presented from different aspects. Experiments show that in condition of low SNR, the proposed method can achieve better noise and jamming stability. Moreover, this method gives a mathematical verification and theoretical basis for the overall design of anti-jamming in ECM. Further researches will be carried out on the application of this algorithm.


The demand for nondestructive testing has increased, especially in welding testing. In the current study, AA1060 aluminum plates were jointed using the friction stir welding (FSW) process. The fabricated joints were subjected to free vibration impact testing in order to investigate the dynamic properties of the welded joint. Damping capacity and dynamic modulus were used in the new prediction method to detect FSW defects. The data acquired were processed and analyzed using a dynamic pulse analyzer lab shop and ME’Scope’s post-processing software, respectively. A finite element analysis using ANSYS software was conducted on different types of designed defects to predict the natural frequency. The results revealed that defective welded joints significantly affect the specific damping capacity. As the damping ratio increased, so did the indication of opportunities to increase the presence of defects. The finite element simulation model was consistent with experimental work. It was therefore revealed that natural frequency was insufficient to predict smaller defects. Friction stir welding (FSW) and processing is an advanced technique used for the joining and fabrication of aluminum alloys. As a result of different processing parameters, such as welding speed and tool shape, discontinuities occur in welded joints, including channel cavities and porosities. Many researchers have studied weld defects and their classification ,5,6,7,8.Internal defects are an important type of defect that are difficult to examine using traditional methods . TraditiA free vibration method was used as a nondestructive test to determine the quality of the welding , which dMost of the previous investigations concerning the dynamic properties of welding focused on natural frequency, and very small defects were not detected. In the present work, damping capacity is used to more accurately detect and identify FSW defects using free vibration analysis. The new detection method depends on the variance in the values of the damping ratio and other dynamic properties. The results are verified using radiographic analysis. Finite element analysis using ANSYS software was carried out to simulate and predict the natural frequency of the defective joint. Through this study, we show that natural frequency alone is insufficient for detecting welding defects.3 +3 mL HCL + 190 mL distilled water) for the macroscopic examination.Aluminum alloy AA1060 plates were used as the base metal. The chemical composition of this commercial alloy is presented in Samples were prepared as a cantilever beam with one free end and fixed using a special clamp. The net fixed dimensions of the tested cantilever beam were 150 mm long, 21 mm wide, and 4 mm thick. The vibration signal was acquired using a piezoelectric CCLD accelerometer. The accelerometer was mounted using plastic clips. An impact hammer with a force transducer was used to excite the welded joint. Vibration analysis was performed using a LAN-XI . The data acquisition system was a 6-channel input module, with a frequency range of (0–51 KHz), FFT resolution up to 6400 lines. The main purpose of this investigation was to predict FSW defects using the dynamic properties of a welded joint. The vibration analysis was calculated and analyzed using commercial vibration analysis software (Pulse Labshop and ME’Scope). Empirical and theoretical equations were used to verify the estimated dynamic properties, such as the damping ratio and natural fundamental frequency. The damping capacity refers to the ability of a material to absorb energy. Materials with a high damping capacity often indicate weakness in the material, and from this principle , it is po, xn, and n are the amplitudes of the first and last cycles, and number of cycles, respectively. The damping ratio is calculated using Equation (2), where ζ is the damping ratio.In this method, which is also called the damped free vibration method, the specimen freely vibrates at its natural frequency, and is damped only by the internal friction of the material. The amplitudes of successive cycles are determined and used to calculate the logarithmic decrease according to Equation (1). The logarithmic decrease is one of the most common methods used to express the damping capacity, as presented in Equations (1)–(3). The logarithmic decrease represents the rate at which the amplitude of a free damped vibration decreases ,34,35. TThe time required to dampen the vibration was minimal when compared to other cases e,f. TherThe effect of FSW processing parameters, such as the welding speed rate and tool rotation speed, on the formation of defects was observed using the variance in the damping capacities in the tested welds. The specific damping capacity and the FSW tool rotation speed  were formulated using the regression analysis in Equation (4). As this represents the regression model, we formulated the relationship between the processing parameters (rotation speed) of friction stir welding and the corresponding damping ratio. At a lower rotation speed, the specific damping capacities have higher values than at high speeds, as shown in a and b are the constants of the regression equation, a = −0.00767, b = 5.89567 × 10−5; and ω is the rotation speed in rpm.Here, v, mm/min). From the experimental results, we observed that whenever the welding rate increased, the heat generated during the friction stirring process decreased. The specific damping capacity decreased nonlinearly with respect to the welding speed, as shown in In addition to the rotation speed, one of the main processing parameters that affects the friction stir welding process is the traverse speed (welding rate) . This method is used to predict the net heat generated during friction stir welding . Figure dE is the dynamic modulus, m is the mass of the bar, b is the width of the bar, L is the length of the bar, t is the thickness of the bar, f is the natural frequency of the bar in flexure, and T is the correction factor for fundamental flexural mode to account for the finite thickness of the bar.The dynamic modulus was calculated according to ASTM Standard E1875-08, using Equations (5) and (6) . Table 3The dynamic modulus was calculated using experimental results from the vibration analysis and the measurement of the sample mass and volume in order to obtain the density. According to ASTM Standard E1875-08 , errors The dynamic properties of the welded joint reflect the quality of the welds through three main factors: natural frequency, damping capacity, and dynamic modulus. The dynamic modulus depends on the natural frequency, mass, and volume of the welded sample. The effect of natural frequency cannot accurately detect defects in welding joints, and this is explained in the next section through the simulation model to predict the natural frequency of defective welds. The second parameter is the mass of the sample, where internal defects decrease the mass. Hence, the defect forms an empty cavity within the metal or welds. When the mass decreases in addition to the slight decrease in the natural frequency, the dynamic modulus also decreases. The experimental results revealed that lower dynamic modulus values were observed at higher damping capacity values.In this test, the welded joints were subjected to a nondestructive radiography scan , to inspect the internal defects of all the samples. The test was performed according to E 1032–95, the Standard Test Method for Radiographic Examination of Weldments. The main purpose of this test was to verify the current investigation using a traditional nondestructive test. The radiography scan uses an X-ray beam projected onto the welded joints. The depth of the defect in the direction of the radiation beam was obtained from the density profile of the defect. As shown in The welded samples were verified using a destructive sectioning method after determining the position of the cavities and defects in each sample using previous nondestructive radiography scans. Finite element analysis (FEA) was carried out using modal analysis and harmonic response modules. Different models of welding defects were designed in order to simulate the mode shapes of the models and calculate the first natural frequency of each design. The welded joint was modeled as a cantilever beam with the same dimensions as the experimentally tested samples using the free vibration impact test discussed previously. The defect shape was modeled to the nearest regular form for analysis purposes. Hence, the difference between the regular shape and the actual irregular shape (from the macrograph) was very close in the final results. The simulation process involved two steps: loading the modal analysis; and designing suitable parameters to calculate the natural frequency. The meshing type and sizing were adjusted based on tetrahedron methods in order to perform the simulation with minimal error. Furthermore, boundary conditions were applied to the end of a cantilever beam as a fixed support, and the force was provided at the tip of the beam. The second step (harmonic response) was conducted as a simulation process in order to predict the frequency response function (FRF).All the calculations and essential data were collected based on the previous processing step. The frequency range was adjusted to 0 to 150 Hz according to the natural frequency calculation in the first step. Six models were built and simulated using finite element software . In the present investigation there are six types of welding defect have been simulated using finite element analysis software Each model represented a welded joint under certain defects, except the first one, which was defect-free. The models simulate some of the common experimental defects generated during welding process. The models were designed to form defects varying from the smallest to the largest in size. Linear porous defect in section A-A represents the smaller defect. The other models represent: regular tunnel type section B-B, regular tunnel type section C-C, taper rectangle tunnel section E-E, and intermittent taper triangle tunnel section F-F. The shape of a defect affects the stiffness of the welded joints. When the sample mass decreases, the natural frequency does not decrease at the same rate. From the previous figures, natural frequency is not a factor that can be used to detect welding defects. The results of the finite element analysis were consistent with the experimental vibration test, from which the dynamic properties could be deduced.From previous results, we concluded that the FSW processing parameters affect the dynamic properties of the welded joint, according to the heat generated due to the stirring action and different processing conditions. The processing parameters that influence the dynamic characteristics are natural frequency, damping capacity, and dynamic modulus.Specific damping capacity decreased nonlinearly with increases in the tool rotation speed and welding rate. This parameter expresses the welding quality because the damping capacity depends on the internal friction of the material. Consequently, specific damping capacity is an excellent parameter to detect welding defects.The corresponding natural frequencies of the defective welding samples increased slightly with respect to the defect-free weld. Therefore, the experimental results of the FRF agree with the numerical method using the FEM results to show that the natural frequency of the defective joint has no significant effect on the detection of internal defects, especially if these defects are small.


This paper presents fault diagnosis logic and signal restoration algorithms for vehicle motion sensors. Because various sensors are equipped to realize automatic operation of the vehicle, defects in these sensors lead to severe safety issues. Therefore, an effective and reliable fault detection and recovery system should be developed. The primary idea of the proposed fault detection system is the conversion of measured wheel speeds into vehicle central axis information and the selection of a reference central axis speed based on this information. Thus, the obtained results are employed to estimate the speed for all wheel sides, which are compared with measured values to identify fault and recover the fault signal. For fault diagnosis logic, a conditional expression is derived with only two variables to distinguish between normal and fault; further, an analytical redundancy structure and a simple diagnostic logic structure are presented. Finally, an off-line test is conducted using test vehicle information to validate the proposed method; it demonstrates that the proposed fault detection and signal restoration algorithm can satisfy the control performance required for each sensor failure. The desire for convenient and safe passenger transportation has increased the need for automated and intelligent automobiles. Consequently, the function of autonomous navigation has been introduced for passenger safety and convenience, and various in-vehicle devices such as ultrasonic sensors, radars, cameras, and actuators have been installed to detect the surrounding environment, recognize information, and control the motions of the vehicle. In addition, speed sensors, steering angle sensors, gyroscopes, and acceleration sensors can be installed to measure the vehicle operation state and movement and to use the gathered information for control. If faults occur in these sensors or actuators during automatic running, the vehicle may deflect from its route or fail to conduct the precise operation required for control, which may lead to an accident. To prevent accidents caused by these faults, technologies applying the soft computing method in fault detection (FDI) and fault tolerant control (FTC) of vehicles are garnering attention in academia and industry. In conjunction with these developments, various ideas and techniques for FDI/FTC methods, including neural network and fuzzy approaches, are presented ,2,3. To Although studies on fault detection are particularly important for the aircraft sector ,11,12,13In these fault-related studies, when a fault is detected, the fault signal has been largely recovered using either direct or analytical redundancy . In pracTo prepare for unexpected sensor failure, this study develops a method for signal duplication using the analytical redundancy method based on the information provided by sensors installed in the vehicle and a mathematical vehicle model. Moreover, the analytical redundancy method is employed to realize fault detection and signal restoration.It is difficult to apply existing research results to other vehicles because existing study have a complex logic structure with different types and numbers of sensors applied to the target vehicle. In addition, existing studies have difficulty in practical application because they have a multi-variable structure in which the judgment logic of failure is complex. These results do not demonstrate the effect of the actual running on the failure of each sensor. Furthermore, their research results do not demonstrate the effectiveness of their performance by applying a restored signal for each fault.This study aims to detect main sensor faults in real time to guarantee occupants’ safety in a vehicle automatically following a designated route without a driver. A vehicle model-based fault detection and signal restoration method is proposed, and the information provided by the failed sensor is restored to create a time margin in which the vehicle control system can conduct normal safety control procedures. To judge the fault of each vehicle sensor, one first needs to identify a suitable comparison standard. Therefore, the velocity and direction components in the central vehicle axis are defined as the central information on vehicle motion. In the absence of abnormalities in each sensor, the central axis speeds estimated by each vehicle sensor should have the same value. Moreover, if the center speed is correct, it can be used to estimate individual wheel speeds, and the calculated values should match actual wheel speeds if each sensor is normal. A kinematic model is designed to convert the speed of each wheel into the central axis speed. In this process, relationships for estimating the central axis speed using a steering angle or a gyroscope are defined, and fault classification conditions obtained by applying these are derived.To validate the proposed method, defects are simulated independently for each normal sensor signal. The signal of the normal sensor is compared with the restored signal, and the resulting error is presented. Additionally, it is confirmed that when the fault signal is restored and applied to the position estimation of the autonomous vehicle, the results of path tracking error are within the valid range.By conducting studies on fault diagnosis and restoration based on the major motion sensors that are installed in most vehicles, the possibility that this study can be used in several vehicles has increased.A vehicle kinematic model based on the central axis of the vehicle, which is used to detect faults and restore fault signals using a structure in which the failure of each sensor does not affect each other, is proposed.In the logic for fault detection, finally, a simple diagnostic logic structure is presented; this structure helps discriminate between normal and fault and distinguishes a specific fault using a conditional expression of only two variables.In summary, the contribution of this study is as follows: The target vehicle has four speed sensors, one steering angle sensor, and one gyroscope for vehicle control  48]. It. It48]. The sensor mounted on the vehicle strongly influences the safety of automatic operation. First, the vehicle controller determines the vehicle position during driving in automatic mode and controls the speed and steering angle to reach the destination based on the fusion of various sensors. The positional information of the vehicle is important for driving control and can be estimated by fusing landmark information, GPS information, and vehicle motion sensor information. The vehicle control system selects a certain point in the vehicle and controls the speed and position of this point to match the desired reference. For example, when the central axis of rear wheels is used as the control reference point in a straight-running vehicle, the calculated speed becomes half of the actual speed if the simple average of the left and right rear wheel speeds is used and one of the corresponding sensors fails. Moreover, when this information is used for position estimation, the calculated position deviates from the actual position. If the gyroscope fails in a curved path, and the information provided by this device is used for position estimation, the vehicle running direction is not calculated correctly, and the resulting position error causes a deviation in route guidance control. Finally, depending on the failure situation, a fault of the steering angle sensor may cause a path-following error or deviation from the traveling path.The main idea behind sensor fault detection and signal restoration process is the conversion of the speed of each wheel to the central axis speed of the vehicle. If all sensors are normal, all calculated central axis speeds should be identical, whereas different values should be obtained in the case when any sensor fails. Thus, this difference can be used as a sensor fault indicator. Moreover, once a fault is found, the speed of each wheel can be estimated from the center axis speed determined as normal, and the faulty signal is recovered by replacing the speed of the defective wheel with the estimated speed. In this case, steering angle information is used to convert the speed of each wheel to the central axis speed. If there is an error in the steering angle sensor, the calculated value is also erroneous, which highlights the importance of knowing whether the steering angle sensor is normal or not. Therefore, two methods are used to calculate the central axis speed. Specifically, the input variables are divided into the cases of steering angle usage and usage of gyroscope-provided information on rotational angular speed. This strategy allows one to detect speed sensor, steering angle sensor, and gyroscope faults and to restore the affected signals even if either the steering angle sensor or the gyro sensor is abnormal. The employed procedure is illustrated in β is the side slip angle, fδ is the front wheel steering angle, flδ is the front left wheel steering angle, frδ is the front right wheel steering angle, fl is the distance from the front axle to the vehicle central axis, rl is distance from the rear axle to the central axis, xl is the distance from the front axle to the rear axle, yl is front or rear axle width, bl is the half-width of yl, νc is the speed at the central axis, ω is the yaw rate, R is the instant radius at central axis, flν, frν, rlν, and rrν are speeds of the front left, the front right, the rear left, and the rear right wheels, respectively.In the above figure, Formulas to convert the vehicle wheel speed to the central axis speed are proposed. The velocity and direction of the central axis are obtained based on the steering angle or gyroscope information.When the steering angle information and gyroscope information are fused together in the fault detection formula, it becomes difficult to distinguish between each of the corresponding faults. Therefore, to differentiate the steering angle fault from the gyroscope fault, the vehicle central axis speed is calculated from the wheel speed, the steering angle, and vehicle parameters excluding the gyroscope information. First, the side slip angle and curvature are obtained from Equations (1) and (2), respectively, using the front wheel steering angle and vehicle parameters.C is the curvature at the central axis, and c,ijν is the central axis speed calculated based on individual vehicle wheel data. In a straight section, the speed of each wheel is the same, whereas in a curved section, the speed inside the curvature radius is lower than that on the outside, as shown in The central axis speed can be obtained from Equation (3) to Equation (6) using individual wheel speeds and Equations (1) and (2).The central axis speed satisfies the condition of Equation (7) when all wheel speed and steering angle sensors are normal and hence afford almost identical speed values, as shown in Therefore, if the central axis speed error exceeds a predetermined threshold value, the sensor or another part are considered to be abnormal, i.e., such an error may be indicative of an abnormal pressure difference between wheel tires, wheel slippage, wheel encoder abnormality, steering angle error, or communication abnormality.The steering angle information is included in Equation (1) to Equation (6), i.e., if the steering angle information is defective, all calculated central axis speeds may be erroneous. Therefore, if the steering angle information is assumed to be normal and there is a defect in the speed sensor information, the following procedure is followed to select two judged-to-be-normal speeds from the four calculated central axis speeds. The two most closely matched central axis speeds are selected using Equations (8) and (9).The selected two central axis speeds are averaged (Equation (10)) asNext, the obtained average is used to estimate the speed of each wheel (Equation (11) to Equation (14)):The thus obtained estimated speed of each wheel should match the measured speed of the wheel if there is no fault in each sensor. For the purpose of fault detection, the error between the estimated and the measured wheel speed is calculated as in Equation (16), and the largest among these errors (defined as in Equation (15)) is used as a condition variable value for detecting sensor defects.ω is the rotational angular velocity measured by the gyroscope. The estimated front-center steering angles should be equal to each other as shown in Equation (21) if there are no faults in the speed sensor and gyroscope.To distinguish between steering angle and gyro faults, gyro information is used instead of steering angle information in the calculation of the central axis speed. For this purpose, the steering angle is estimated from the gyroscope information, wheel speed, and vehicle parameters, and the central steering angle is estimated using Equation (17) to Equation (20).If the gyroscope is fault-free, a fault of the speed sensor should result in a difference between some estimated steering values. Therefore, to select the steering angle with the smallest error due to the defect, two estimated steering angles with the smallest error among the estimated steering angles (The selected estimated steering angles are averaged (Equation (24)), and the obtained value is used as the estimated front wheel steering angle.If the front steering angle is estimated using the gyroscope information, a process identical to that used for central axis speed estimation, as described in the previous section, is applied. The speed at each vehicle center is calculated using the speed of each wheel and the estimated steering angle. Next, the side slip angle and curvature are obtained as in Equations (25) and (26), respectively, using the estimated steering angle and vehicle parameters.The central axis speed can be obtained using Equation (27) to Equation (30) by considering individual wheel speeds, Equations (25) and (26).If each wheel speed and the gyroscope are normal, the calculated central axis speeds should be equal to each other as shown in Equation (31) .(31)vc,As in the previous case, two speeds with the smallest error selected based on the error between the central axis speed of the two combinations are averaged (Equation (32)).The thus obtained value is used to estimate the speed of each wheel using Equation (33) to Equation (36).The thus estimated speeds of each wheel should match the measured values if there is no defect in each sensor. For the purpose of fault detection, the error between each estimated wheel speed and the measured wheel speed is obtained as in Equation (38), and the largest among these errors (defined as in Equation (37)) is used as a condition variable value for detecting sensor faults.asmax, agmax) and the corresponding limit values  are used to determine whether the sensor is faulty.To identify sensor faults, Equations (15) and (37), which describe the calculation of maximum estimated speed and the measured speed error, are used for judgment. The maximum values of each error  between the estimated speed and the measured speed is small, all sensors applied to the relational expression can be viewed as not defective. The maximum allowable error limit , which is the defect judgment boundary, is determined based on the observed results under the condition that all sensors are normal. Herein, limit values of aslimit = aglimit = 0.025 were selected for the running test of the test vehicle. Therefore, if the maximum error  is smaller than the above values, all sensors are normal, and if not, a fault is concluded to be present. This condition is expressed in Equation (39).If the maximum error (agmax) of the error calculated using gyroscope information is smaller than the limit value (aglimit), the gyroscope and speed sensor are both viewed as normal. If the maximum value (asmax) of the error calculated using steering angle information is larger than the limit value (aslimit), the steering angle sensor is viewed as defective, as there is no defect of the speed sensor under the preceding condition. This condition is expressed in Equation (40).If the maximum value (If a steering angle fault is identified according to the condition of Equation (40), the measured steering angle sensor information is replaced with the estimated steering angle information calculated using Equation (24).asmax) of the error calculated using steering angle information is smaller than the limit value (aslimit), both the steering angle sensor and the speed sensor are viewed as normal. In this case, if the maximum value (agmax) of the error calculated using the gyroscope information is larger than the limit value (aglimit), one can judge that only the gyroscope is defective, as it follows from the above that the speed sensor is not defective.If the maximum value (If a gyroscope fault is identified according to the condition of Equation (41), rotational angular velocity is estimated from Equation (42) by applying Equations (13) and (14), which describe the estimation of rear wheel speeds based on steering angle information.asmax) of the error calculated based on steering angle information is greater than the limit value (aslimit), and (ii) the maximum value (agmax) of the error calculated based on the gyroscope information is greater than the limit value (aglimit) [the error of the speed sensor affects both sides]. In this case, when the speed error is larger than the limit value, fault identification of each speed sensor can be performed as follows.Under the condition that the gyroscope and the steering angle sensor do not fail simultaneously, the speed sensor is viewed as defective if (i) the maximum value (If a speed sensor fault is identified according to the condition of Equation (43), the value provided by the defective sensor is replaced with the value estimated using Equation (11) to Equation (14).To verify the validity of the proposed algorithm, we performed an off-line test using the sensor data collected from the test vehicle . The vehOne of the four wheels simulated the fault of one rear right wheel speed sensor from 20 to 80 s. A zero was injected during this time instead of the normal signal to simulate a failure. Therefore, the actual speed was known. The faulted signal, the normal signal, and the signal estimated using Equation (13) are shown in Only the case where the steering sensor is faulty was considered, and the fault was chosen to occur between 20 and 80 s. During this period, the steering angle information was changed to zero, and the signal was estimated using the proposed algorithm and compared with the normal sensor signal . BecauseOnly the case where the gyroscope is faulty was considered, and the fault was chosen to occur between 20 and 80 s. During this period, the signal was estimated using the proposed algorithm and compared with the normal sensor signal. Notably, the proposed method allowed for good recovery of the normal signal. This test aimed to check the effect of sensor faults on automatic running and to determine whether the proposed fault detection and signal restoration method is valid. In the case of a running vehicle, the sensor was configured so that only one fault occurs at the same time.The estimated position and tracking control state of the vehicle were checked in the case where all sensors were normal, and the observed performance was compared to that in the case of sensor fault. The case when only the rear left wheel speed sensor of the vehicle is defective was considered. When the average of the left- and right wheel speed is used as the center speed, the calculated travel distance is half of the actual travel distance when one of the speed sensors fails during vehicle operation, which results in erroneous position calculation and deviation from the traveling path . As showWhen a gyroscope fault occurs during vehicle operation, an error is generated in the calculation of the running direction, which increases the error in the calculation of vehicle position. As a result, the vehicle deviates from the traveling path. As shown in In this paper, fault diagnosis logic and signal restoration algorithms for vehicle motion sensors are presented. To this end, a central axis kinematic model is applied to each wheel speed of the vehicle. Both steering angle and gyroscope information are considered to distinguish failure effects. For fault diagnosis logic, we derive a conditional expression with only two variables to distinguish between normal and fault, and an analytical redundancy structure and a simple diagnostic logic structure are presented to distinguish specific faults. This study further assumes that only one sensor can fail at any given instant, which may limit the current scope of application of the proposed fault diagnosis scheme.To verify the validity of this method, vehicle sensor data are collected under normal driving conditions, and the algorithm used in the actual vehicle is applied to estimate the vehicle position and orientation in an off-line test. The risk of automatic driving according to each failure is examined through the addition of faults to normal sensor information. It is shown that the autonomous vehicle can satisfy the valid normal driving conditions when the proposed fault detection and signal restoration method is applied under fault conditions. To improve vehicle safety, the authors plan to investigate diagnostic methods for multiple sensor faults.


Directed-assembly of nanowires on the dielectrics-covered parallel electrode structure is capable of producing uniformly-spaced nanowire array at the electrode gap due to dielectrophoretic nanowire attraction and electrostatic nanowire repulsion. Beyond uniformly-spaced nanowire array formation, the control of spacing in the array is beneficial in that it should be the experimental basis of the precise positioning of functional nanowires on a circuit. Here, we investigate the material parameters and bias conditions to modulate the nanowire spacing in the ordered array, where the nanowire array formation is readily attained due to the electrostatic nanowire interaction. A theoretical model for the force calculation and the simulation of the induced charge in the assembled nanowire verifies that the longer nanowires on thicker dielectric layer tend to be assembled with a larger pitch due to the stronger nanowire-nanowire electrostatic repulsion, which is consistent with the experimental results. It was claimed that the stronger dielectrophoretic force is likely to attract more nanowires that are suspended in solution at the electrode gap, causing them to be less-spaced. Thus, we propose a generic mechanism, competition of dielectrophoretic and electrostatic force, to determine the nanowire pitch in an ordered array. Furthermore, this spacing-controlled nanowire array offers a way to fabricate the high-density nanodevice array without nanowire registration. Nanowires made of a diverse range of chemical compositions, homogeneous or axially and radially heterogeneous, have been demonstrated through various synthetic means ,5,6,7,8.For the investigation of nanowire spacing on a substrate, the parallel electrode structure was patterned by a thin metal (Ti/Au = 20 nm/50 nm) following the liftoff process in an acetone dipping. A dielectric layer of polymethylglutarimide (PMGI) with a thickness of 300 nm is covered over the entire substrate so that it prevents the assembly of metallic nanowires from shorting out the biased electrodes. The fluidic cell was used to pull the nanowires in the solution down in the vicinity of the electrically active region. Rhodium (Rh) nanowires were used in this study due to their high elastic modulus, high mechanical strength, and the ease of batch-synthesis via electrodeposition.After the nanowire suspension is injected into the fluidic cell by using a micropipette, a sinusoidal bias with different voltages and frequencies being applied between the electrodes was applied to investigate the effects of bias conditions on the nanowire attraction to the electrode gap and the nanowire spacing in the array. COMSOL Multiphysics was employed to simulate the electric field distribution and its relevant dielectrophoretic force.The theoretical model accounting for nanowires’ repulsive behavior between two identically polarized nanowires appears in Due to the charge neutrality, the same amount of positive and negative charges are separated within a nanowire, and their capacitive-coupling to the biased electrodes centers the nanowires at the electrode gap, as illustrated in Theoretically, it can be suggested from An approximation suggests that the induced charge density is uniform on either half of the nanowire with an opposite sign of the charge density. This approximation, for the simple model, as drawn as a solid line in y, between the assembled nanowires can be written:Using this simple uniform charge density model, the total Coulombic interaction force, FThe first term in the equation refers to the repulsive force between two adjacent nanowires, whereas the second term refers to the attractive force. Since the electrostatic interaction between the nearest neighboring nanowires is dominant, this total force calculation can readily apply to all of the nanowires assembled at the electrode gap. Equation (5) suggests that the total force is always positive in any case and is inversely proportional to the distance between the adjacent nanowires when they are close. This condition results in strong repulsive force at small distances. y, (Equation (5)) as a function of the assembled nanowire distance (y) at various nanowires’ lengths (ℓ). This plot predicts that the repulsive force (FR) tends to become less strong as the inter-nanowires’ separation distances (D) increase. A further approximation from the previous model indicates that the uniform charge density of half of the assembled nanowires can easily clarify this tendency. In other words, the polarized nanowires can be a further approximation of dumbbell dipole structures having the opposite charges, positive and negative, at the ends of the dumbbells, as shown in the inset of Plotting Equation (5) provides more useful information to clarify the relationship between the repulsive nanowire interaction and nanowires’ material parameters, such as the inter-nanowire distance and the nanowire length, as shown in The uniformly-spaced nanowire array begins to form as the nanowire spacing comes close to a certain value for efficient electrostatic interaction. According to the plot in 4 nanowires/μL by adding isopropyl alcohol. Special attention maintained injection of 10 μL of nanowire suspension for all of the length scales until the excessive nanowire-nanowire chain configuration began to form. Nanowire assembly, accomplished with the same interdigitated electrode structure varied nanowire length . For consistency of the nanowire assembly, the concentration of the nanowire suspensions remained as low as ~1 × 10y distribution for 3 μm and 6 μm wide electrode gaps in y becomes less strong beyond the edge of a biased electrode, particularly in a narrow electrode gap. According to Gauss’s law , the electrostatic repulsive force can be determined by the value of the charge density (pression .(6) Q =e(K)]. The frequency of applied bias also affects the induced charges in the assembled nanowires, resulting in the frequency-dependent electrostatic force between them. In contrast to the dipole formation in semiconducting nanowires, the metal is of a very high permittivity material due to free electrons in it. Ideally, an infinite value makes Re(K) almost constant over a wide range of frequencies. Without doubt, the frequency response of free electrons in the metal is instant. Thus, the induced charges in the assembled nanowires as well as the dielectrophoretic attraction of them can be considered as frequency-independent factors for determining the nanowires’ spacing in the array. The average spacing of 7 μm-long rhodium nanowires, assembled at the electrode gap with a 3 μm width, is estimated to be about 5 μm in the range of 10 kHz to 10 MHz. As discussed, no significant change in the nanowire spacing was observed at relatively low frequency ranges, as compared to the relaxation time of free electrons in the metal.The frequency of the applied bias could influence nanowire alignment on a biased electrode. In other words, the frequency response of dipole moment in the polarized nanowires determines the dielectrophoretic nanowire attraction by the real part of the Clausius-Mossotti factor [Rpp). It was observed that the 7 μm-long rhodium nanowires form uniformly-spaced arrays on the electrode gap for all of the voltage ranges. In contrast to the theoretical expectation that is described in Equations (5) and (6), the plot , which is responsible for the number of nanowires to be assembled, and the electrostatic repulsion (FES) between the nanowires, responsible for the inter-nanowire spacing, is a reasonable indication for determining the inter-nanowire spacing for given conditions. The expression of the dielectrophoretic force is [ES) to the dielectrophoretic (FDEP) forces enables the estimation of the relative inter-nanowire spacing in the array according to the chosen specific bias conditions and materials’ parameters. In other words, their high ratios of electrostatic to dielectrophoretic forces at the given conditions of applied voltage provide an indication of the relatively stronger electrostatic force between the assembled nanowires compared to the dielectrophoretic force, leading to larger spacing between the nanowires.Comparison of the dielectrophoretic force (Fforce is .(7) FDEES) and dielectrophoretic (FDEP) forces in which Ey for FES and DEP are extracted from the PMGI surface above the electrode edge and the surface of the assembled nanowires in the simulations, respectively. The voltage varied from 1 Vpp to 35 Vpp. As anticipated from the equations, notably, from ES/FDEP increases by a small increment as the voltage increases. This further suggests that the inter-nanowire spacing at varying voltages has the same relationship. Apparently, this relationship is quite consistent with the plot of the experimental results that are shown in The thickness of dielectric layer, i.e., PMGI in the experiment’s assembly setup, also affects the electrostatic repulsion between the assembled nanowires, eventually establishing the average spacing between them. Understandably, the amount of induced charge on the surface of the nanowires depends on the thickness of the dielectric layer because of the capacitor-like structure of nanowire-PMGI-biased electrodes . Figure ES) and dielectrophoretic (FDEP) forces is necessary. Similarly, a plot of Ey/cosh−1(2H/D) for the inter-nanowire repulsion force and ES to FDEP increase nearly linearly as PMGI thickness increases, suggesting that the electrostatic force between the assembled nanowires becomes relatively stronger when compared to the dielectrophoretic nanowire attraction. Thus, the ratios of Ey/cosh−1(2H/D) to In order to interpret this inter-nanowire spacing dependence on the thickness of PMGI, performing a simulation to compare the ratio of the electrostatic (FThis study of control of the inter-nanowire spacing in the array demonstrated that the electrostatic repulsion between the nanowires plays a key role in determining the average distance between them, while the dielectrophoretic force also contributes to it by controlling the number of nanowires to be assembled. This electrostatic repulsion as well as the dielectrophoretic force is not only limited to the metallic entities and is also applicable to other semiconducting nanowires. Significance of this study lies on the fact that the average spacing between the nanowires can be readily controlled either by the bias conditions or by the materials parameters. In addition to the unique feature of our nanowire assembly, such as a high degree of alignment and orientation at well-defined locations and excellent end-to-end registration, this control of nanowires spacing offers a promising method without the need for the registration of nanowires to fabricate the individual devices by the metal deposition following the lithographical patterning. For instance, setting the width of narrow metal interconnects to the average distance of the nanowires produces a large number of single functional nanowire devices in a scalable and parallel fashion where the nanowire registration relative to the metal electrode is not required. Its versatility could be extended to the novel applications, such as the plasmonics of metallic nanopattern and nanoresonator devices, chemical and biological sensors, and flexible light-emitting diode, where the multiple nanowires with a tight control of its number are preferred for the uniform transistors array requiring high transconductance.


In this paper, a multi-frame based homography estimation method is proposed for video stitching in static camera environments. A homography that is robust against spatio-temporally induced noise can be estimated by intervals, using feature points extracted during a predetermined time interval. The feature point with the largest blob response in each quantized location bin, a representative feature point, is used for matching a pair of video sequences. After matching representative feature points from each camera, the homography for the interval is estimated by random sample consensus (RANSAC) on the matched representative feature points, with their chances of being sampled proportional to their numbers of occurrences in the interval. The performance of the proposed method is compared with that of the per-frame method by investigating alignment distortion and stitching scores for daytime and noisy video sequence pairs. It is shown that alignment distortion in overlapping regions is reduced and the stitching score is improved by the proposed method. The proposed method can be used for panoramic video stitching with static video cameras and for panoramic image stitching with less alignment distortion. With the development of information and communications technology (ICT), there is a growing demand in various fields, such as virtual reality (VR), image security, entertainment, aerial image, and mobile applications for panoramic images and videos with a wide angle of view ,2,3,4. ARecently, as the demand for the VR application of 360-degree video increases, existing research on image stitching is being extended to video stitching ,10,11. HConventional studies to improve the performance of video alignment for stitching can be classified according to the motion characteristics of cameras. Since homographies between cameras is fixed in static camera environments, the homography extracted from the first input frame pair can be used in all subsequent frames ,14,15. IIn all existing types of video stitching methods, a homography between the first frame pair is used for the subsequent frame pairs. In a static camera environment, an inaccurate initial homography can affect the stitching performance of the entire video sequence pair. In a dynamic camera environment, the homography of the following frame pair may be inaccurate because it is calculated using an inaccurate initial homography and an estimated camera motion. Thus, if the initial homography is inaccurate, the stitching performance of the entire video sequence pair can be degraded irrespective of the accuracy of the estimated camera motion. Therefore, in order to improve the video stitching performance, it is necessary to improve the accuracy of the matching between frame pairs, which is the basis of the initial homography estimation.Finding correspondences across cameras has been attempted through pixel-by-pixel mapping between two images  and by uFinding correspondence matching of multi-view video sequence pairs has been attempted ,26,27,28To improve the quality of stitched video sequences, it is necessary to find correct alignment parameters between cameras while suppressing the wrong feature point extraction from time-varying noise. In this paper, a novel multi-frame based homography estimation method is proposed for correct video alignment in static camera environments. A homography that is robust against spatio-temporally induced noise can be estimated by intervals, using feature points extracted during a predetermined time interval. To find spatially consistent representative feature points for matching of a pair of video sequences, representative feature points with the largest blob response are selected among feature points extracted in a predetermined time interval for quantized location bins. To find temporally consistent feature points over the predetermined time interval, SURF feature locations are quantized into finite two-dimensional histogram bins for the accumulation of their numbers of occurrences. After matching representative feature points from each camera, a homography for the interval is estimated by random sample consensus (RANSAC) using feature points in the histogram bins, with their chances being sampled in proportion to their histogram counts.For the evaluation of the accuracy of the estimated homography, the frames of the left camera are warped into the planes of the frames of the right camera using the estimated homography. The warped left frames and the right frames are average blended into stitched frames without a seam-line estimation. The alignment error of the video sequences stitched using the proposed method is compared with that of video sequences stitched using a per-frame method.The contribution of the proposed method is three-fold. First, the proposed method can reduce alignment error due to inaccurate estimation of initial homography in stitched video sequences. Second, the proposed method can be effectively applied to video sequence pairs, of which confidence for extracted feature points is not sufficiently high, due to noise or the plain texture of the video sequences. Finally, the proposed method does not require moving objects for homography estimation, and can be used for general video sequence pairs with or without moving objects.This paper is organized as follows: In In this section, a novel multi-frame based homography estimation algorithm, shown in In the first step, as shown in A square quantized location bin For a quantized location bin  in SURF . Sn reprRepresentative feature point selection is independently performed on each sequence of a video sequence pair in the homography estimation interval. These representative feature points are spatially consistent during the interval and are used for matching without the other feature points, which makes the matched feature points more reliable and the matching faster.In the second step, to find temporally consistent feature points, a histogram of SURF feature locations is generated. SURF feature locations are quantized into two-dimensional histogram bins, and the number of occurrences of feature points in each histogram bin is counted over the time interval. A square quantized histogram bin A set of feature points in the histogram bin In the interval The histogram generation is independently performed on each sequence of a video sequence pair in the homography estimation interval. The width of the histogram bin is set to a half pixel, i.e., In the third step, nearest neighbor matching is performed on the representative feature points from video sequence pairs in the homography estimation interval ,20,29, wThe sub-figures titled Frames 1, 2, and 3 in H between a pair of video sequences is estimated based on the simple outlier elimination technique, RANSAC [In the final step, the homography , RANSAC , with thM represent the The average blob response for all the feature points is calculated asfollows:The average histogram count for all the matched representative feature points is calculated as follows:The replication number of the For homography estimation using the proposed histogram weighted RANSAC algorithm, four feature correspondences are selected from the histogram weighted RANSAC dataset. The homography on (DLT) . This prIn order to evaluate the performance of the proposed multi-frame based homography estimation method, the video alignment results of the proposed method are compared with those of the per-frame stitching method ,20. The Experiments for performance evaluation consist of three parts. The first experiment compares the alignment distortion and stitching score  of the aTo evaluate the performance of the proposed algorithm, four daytime and two nighttime video sequence pairs, listed in seq1–4. As shown in After selecting representative feature points for the homography estimation interval, a matching experiment is performed using seq1–4, respectively. The worst ratios of incorrect matches in the per-frame method are 8.0%, 14.5%, 11.3%, and 10.2% for seq1–4 during the homography estimation interval, respectively. Because the proposed method uses representative feature points consistent in the spatio-temporal direction for matching, the ratios of incorrect matches are lower than those in the per-frame method.The matches were visually inspected for their correctness in descending order of their reprojection distances. Most of the matches with reprojection distances exceeding five pixels are considered as incorrect in our experiments. The ratios of incorrect matches with the proposed method are 5.9%, 9.6%, 2.3%, and 6.0% for The first experiment estimates homographies using the proposed and the per-frame methods for daytime video sequences and compares the alignment distortion and the stitching score for the video frames stitched by these methods.The proposed method estimates the homography by the homography estimation interval; all the frames in the interval are aligned with the same homography. The per-frame method aligns each frame pair using the homography estimated for the pair.Alignment distortion can be visually assessed when a frame warped by homography is superimposed on the other frame in the region where the frame pairs overlap. The first column in In the per-frame method, noise and illumination change over time, which may affect the feature extraction and matching processes. As a result, the estimated homography may change every frame, degrading the quality of the entire stitched video sequence with jitter. On the other hand, the proposed method shows unnoticeable alignment distortion in the frames in the homography estimation interval. This is because the homography is estimated by selecting from feature points existing in the interval matched representative feature points that are robust against noise or illumination variation. The proposed method effectively filters noise components distributed in spatio-temporal directions through two processing stages. In the first step, feature points of relatively low blob response, which may be temporally induced by noise, are removed by selecting one representative feature point having the highest response for each quantized location bin in video frames in the homography estimation interval. In the second step, feature points frequently extracted in the same locations in the homography estimation interval are differentiated from infrequent feature points using a two-dimensional histogram of feature point locations. In the RANSAC based homography estimation process, replicating both spatially and temporally consistent matched representative feature points increases their possibilities of being sampled for homography estimation.For quantitative evaluation of the video alignment results, stitching scores for stitched video frames, shown in We used average blending on the overlapped region to present misalignment in this region due to an incorrect homography estimation. The seam selection and blending algorithms of advanced video stitchers can efficiently maintain the spatial and temporal consistency and remove any color discontinuities in overlapped regions ,8,15,32.To evaluate the robustness against noise of the proposed method, the second experiment compares the alignment distortion and stitching scores of the proposed method with those of the per-frame method for daytime video sequences with Gaussian noise added and nighttime video sequences. As can be seen in seq2, seq3, and seq4. As the variance of noise increases, the alignment distortion increases because the estimated homography becomes less accurate, due to the increment in the number of false feature points and the decrement in the number of real feature points. In the per-frame method, there is noticeable alignment distortion in all stitched video sequences. In the case of seq4, when the noise variance was 1600, the per-frame method resulted in a complete misalignment in the stitched frames due to a large error in the estimated homography. On the other hand, the proposed method has little or unnoticeable alignment distortion for all noise powers. This is because the proposed method performs matching and homography estimation using noise-resistant feature points of high blob responses and frequent occurrence in the homography estimation interval.seq2 and seq4 were captured and these pairs were stitched with the same process used for daytime sequence pairs. For nighttime video sequence pairs, there may be noise amplification by night vision circuit or noise due to characteristics of illumination sources. In order to evaluate the influence of noise on homography estimation in the actual shooting environment, nighttime sequence pairs similar to seq1–4, seq2N, and seq4N; the average stitching scores for video sequence pairs are shown in In the third experiment, the effects of the homography estimation interval on the stitching score and the processing speed are evaluated. Homography estimation and stitching are performed with four different homography estimation intervals for As can be seen in Next, the average speed of homography estimation and stitching with respect to Although the proposed method experiences some initial latency during homography estimations, it can be used for the real-time stitching of video sequence pairs. In addition, the proposed method shows less misalignment distortion of the stitched frames because it is possible to estimate a noise-resistant homography in the spatio-temporal direction. The proposed method can be more economical than the per-frame method from the viewpoint of operational cost, because it can stitch all the frames in a video sequence pair with only an initial estimation of homography in static camera environments.In dynamic camera environments, an earlier real-time homography estimation method  also reqIn this paper, a multi-frame based homography estimation method is proposed for video stitching in static camera environments. Homography that is robust against spatio-temporally induced noise can be estimated by intervals, using feature points extracted during a predetermined time interval. Representative feature points with the largest blob response in each quantized location bin are used for matching a pair of video sequences. After matching representative feature points from each camera, the homography for the interval is estimated by RANSAC on the matched representative feature points, with their chances of being sampled proportional to their numbers of occurrences and their blob responses in the interval.Compared with the per-frame method, for daytime and noisy video sequences, the proposed method reduces alignment distortion in overlapped regions and improves stitching score. The increase in the average computational load by the proposed method in static camera applications is negligible if the initial homography estimated is used for an entire video sequence pair.Since the proposed method is based on video sequence pairs captured by static cameras, it cannot be directly applied to dynamic camera environments. However, if the pose or displacement information of cameras can be used, feature points robust against noise in spatio-temporal directions can be searched for each interval and used for per-frame homography estimation for the interval.


Two target words in each sentence were manipulated for lexical frequency (high vs. low frequency) and parafoveal preview of each target word . Eye movement data revealed visual parafoveal-on-foveal (PoF) effects, as well as foveal visual and orthographic preview effects and word frequency effects. Fixation-related potentials (FRPs) showed visual and orthographic PoF effects as well as foveal visual and orthographic preview effects. Our results replicated the early y effect  in the X The investigation of the time course of visual word recognition has been of special interest for researchers in the field of cognitive science. Indeed, it is of particular importance to understand how the brain represents and transforms information during reading, and the nature of processes that occur as different stages of processing unfold over time, from the early visual perception of a word form through to its full identification and linguistic interpretation.Although there has been a large amount of research conducted with both eye movements and event-related potentials (ERPs) to examine the timing of the processes underlying reading, the debate on the exact time course of these processes is still ongoing. As suggested by It is evident that the paradigm typically used in ERP research , in which one word at the time is presented in the center of the screen, is different from natural reading conditions in several ways. First, participants are required to fixate the middle of the screen and avoid making any eye movements . Second, the required central fixation and successive word-by-word presentation prevents some natural eye movement phenomena from occurring; for example, word skipping, natural refixations, and regressions. Third, each word is displayed on the screen for a set amount of time , preventing readers having control over how long they fixate each word. Fourth, the speed of reading is reduced, as the rate with which words are presented on the screen is typically slower than the natural reading pace. Lastly, parafoveal information is not available. This last point is of crucial importance. A distinctive feature of natural reading is that multiple words are presented in one, or more, horizontal lines of text and the words are processed (largely) successively, in the order in which they appear. This means that during natural reading both foveal  and parafoveal  information is available during any particular fixation. The important point to note here is that visual information about a word is encoded during two separate perceptual phases; first, visual degraded information about a word is accrued over time when the word is in the parafovea ; second, nondegraded visual information about the constituent letters of a word becomes rapidly available when the word is directly fixated . To be clear, then, the RSVP technique, by its very nature, cannot deliver visual information in a manner that accurately approximates the way visual information is delivered during natural reading. Thus, the ERP studies conducted to date using RSVP do not permit researchers to investigate parafoveal processing, nor to examine the time course of foveal processing of a word in relation to preceding parafoveal processing of that word. In short, RSVP does not permit the study of perceptual and linguistic processing as it occurs naturally during reading.Aiming to overcome some of the limitations associated with ERP investigations that adopt an RSVP methodology, two approaches have been developed: (a) the use of RSVP-with-flanker-word-presentation, and (b) the coregistration of eye movements and EEG signal. In the first approach, sentences or lists of words are presented one by one in the center of the screen, and are simultaneously flanked to the left and to the right of the central fixation with the preceding and following word(s) in the text . It is asaccadic reading; In the second approach, readers are free to move their eyes, and eye movements and EEG signal are simultaneously recorded while participants read pairs of words , lists oPoF effects refer to any influence that the characteristics of parafoveal words have on processing of the currently fixated foveal word. Evidence of the existence of these effects is controversial both in the eye movement and FRP literature . MoreoveSome eye movement studies have reported PoF effects produced by unusual letter combinations in the parafovea see . HoweverSimilarly, the results are mixed in the FRP literature. Three studies using coregistration of eye movements and EEG signal have investigated the neural correlates of orthographic PoF effects. Even more disputed are the lexical PoF effects of word frequency. These effects have been mainly found in corpus analysis studies e.g., . In suchTwo coregistration experiments have investigated the PoF effects of word frequency, and each has obtained different results. One of the most well-established findings in the eye movement literature on reading is the preview effect demonstrated using the boundary paradigm. In such an experiment, when readers receive a valid preview of the upcoming parafoveal word, the identification of that word during the subsequent fixation is facilitated compared with when readers receive an invalid preview. Eye movement studies across different languages have consistently reported that parafoveal previews that have some form of visual or orthographic overlap facilitate the speed of processing of the target word see . This expreview positivity and consists of an attenuation of the N1 component amplitude for identical compared with invalid previews, which is maximal over occipito-temporal sites. In addition, a late effect of preview has also been observed over the N400 component. Between 300 ms and 500 ms after fixation onset, identical previews yielded less negative amplitudes than invalid previews over midparietal electrodes. These effects have first been observed by blade-blade) to invalid previews , while participants read a list of five unrelated German words displayed horizontally across a presentation screen. The findings have been replicated in Chinese with the RSVP-with-flanker-word presentation approach The objectives of the current experiment were a) to investigate the neural correlates of visual, orthographic, and lexical PoF effects; (b) to identify the neural correlates of the preview effect in natural reading, and if successful, to further demonstrate that those neural correlates may be modulated by the degree to which the preview is related to the target word; and (c) to explore possible neural correlates of the foveal lexical frequency effect under natural reading conditions , then it is very visually dissimilar to a word, and it is likely that participants would detect the string in the parafovea . Thus, gAs is standard in boundary paradigm experiments such as this, for the effects at the target word, we predicted increased preview benefit in the identity condition, less preview benefit in the random letter string preview condition, and least preview benefit in the X-string preview condition. This is the pattern of effects that we would certainly predict for the eye movement measures alone . Note, though, as mentioned earlier, a letter string preview is more word-like than a X-string preview, and therefore it is possible that the FRP data at the target may reflect increased difficulty associated with the integration of the target with letter string previews than X-string previews, as well reduced difficulty caused by feature overlap between target and preview. Importantly, this is the first study to use coregistration methodology to explore differences in FRPs between previews comprised of strings of letters and strings of Xs, and therefore, this is the first opportunity to explore whether different preview types have a differential influence on the time course of foveal processing e.g., .The third objective of the study was to explore the neural correlates of the foveal lexical frequency effect under natural reading conditions. Based on previous research, we expected to replicate word frequency effects in the eye movement data. However, given that the neural correlates of word frequency have rarely been investigated during natural reading, we were less confident in our predictions for frequency effects in the FRP data. Recall that M = 85.38, SD = 17.05 according to the Edinburgh Handedness Inventory; M = 19.29, SD = 1.60). All participants provided written informed consent and received course credits or money for taking part.Forty-two participants  took part in the study. All participants were English native speakers, right-handed  that did not differ significantly in terms of plausibility  and predictability . The characteristics of the set of words used in the experiment are provided in We selected 696 five- or six-character words from the English Lexicon Project , 348 higWe used a 3  × 2 (Target Word Frequency: high vs. low frequency) experimental design, with 18 sentences and 36 target words per condition. Sentences were presented using the boundary paradigm . As illuParticipants were seated 70 cm from a 19-in. stimulus CRT display screen, with a resolution of 1,024 × 768 and a refresh rate of 140 Hz. The text was presented in 14-point Courier New sentence case font, in black ink on a gray background. Approximately 2.19 characters subtended one degree of visual angle.Viewing was binocular, but eye movements from the right eye only were recorded from a desktop EyeLink 1000 eye tracking system (SR Research), at a sampling rate of 1000 Hz. A 3-point calibration procedure was completed at the beginning of each block of sentences, and whenever needed during the experiment. Calibration was accepted when the average error was lower than 0.3° and the maximum error lower than 0.99°. Furthermore, a 2-point drift correction check was performed at the beginning of each trial.The EEG signal was recorded from 64 scalp electrodes  located according to the 10–20 International system. Four EOG channels were also used to record the EEG signal associated with eye movements. AFz was used as ground electrode, and the nose as the online reference. The EEG signal was recorded from DC SynAmpsRT amplifiers  with a sampling rate of 1000 Hz, and it was low-pass filtered online at 100 Hz (with an attenuation of 12dB/octave).The current experiment was approved by the University of Southampton Ethics Committee (study ID: 25066).Before starting the experiment, participants were required to complete the Edinburgh handedness inventory  in orderThe experimental session involved five blocks of sentences. The first block always comprised 10 practice trials to familiarize the participant with the procedure. The following four blocks were formed from one filler as the first trial and 27 experimental sentences, which were presented in a random order for each participant. Participants were asked to silently read each sentence and to answer comprehension questions for 24% of the trials, while their eye movements and EEG signal were simultaneously recorded.Each trial began with a cross on the left side of the screen. Participants were required to fixate the cross for 500 ms before a sentence was displayed, with the first letter of the sentence located at the same position of the cross. Once participants had silently read the sentence to understand it to the best of their ability, they were instructed to fixate a cross on the right side of the screen to terminate the current trial and initiate the following one.After the experiment, participants were asked to complete a short questionnaire to assess their display change awareness . The expM = 0.29, SD = 0.46 ms in absolute value).At the onset and offset of each trial the stimulus display computer (running SR Research Experiment Builder) sent a trigger to the computer recording the EEG signal and a message to the computer registering eye movements. This allowed for an accurate offline synchronization of eye movements and EEG signal via the EYE-EEG extension of EEGLAB toolbox , as confThe eye movement data were pruned via the “clean” function in DataViewer (SR Research) such that only fixations longer than 50 ms and shorter than 800 ms entered the analyses. Fixations on each interest area were excluded when the display change occurred early, during a fixation on the preboundary  word, and when the display change was late . We also removed fixations with hooks, wherein the display change was triggered early by a saccade that temporarily crossed the invisible boundary to finally end to the left of the boundary. Furthermore, we excluded from the analyses fixations on pretarget and target words in which participants made a blink and/or a skip. Additionally, only consecutive fixations, which landed first on the pretarget word and then proceeded onto the target word, entered the analyses. Lastly, only fixations that occurred during first-pass reading were analyzed.M = 2.98 components removed per participant, SD = 1.52) as oculomotor artifacts . Spherical interpolation of a channel was performed when that channel exceeded the threshold for more than 5% of all epochs, which was the case for nine of our participants. FRPs were then averaged within and then across participants for analyses. After eye movement and FRP preprocessing, we were left with a total number of 5,349 observations for the pretarget and target word.The EEG data were band-pass filtered offline with the EEGLAB 14_1_1  toolbox rtifacts . The artrtifacts , 2009. TEye movement data were analyzed with linear mixed effects models within the R environment for statistical computing . We usedp values were estimated using the “lmerTest” package, with the default Satterthwaites’s method for degrees of freedom and t-statistics  as a function of the visual and orthographic properties of the upcoming parafoveal word n + 1 (target word) during natural reading. With respect to PoF effects of lexical frequency, only one existing study has found evidence of these effects (FieldTrip (t-statistic for each sample (channel-time pair) for the difference between the conditions. If the t-statistic was smaller than a threshold of 0.05 and was temporally and spatially adjacentt value, we assigned this t value to a cluster. For each iteration, we then computed the maximum  positive and negative cluster-level t-statistic by calculating the sum of all the t values within a cluster and we created the null distribution. For the original (observed) data, we applied the same procedure except that we did not shuffle the trials between conditions. The observed t-statistic was tested against the null distribution of the permuted data. The observed cluster-level t-statistic was considered significant  when it was located beyond the determined threshold . This type of permutation test is a mass univariate approach that allows for a considerable number of univariate tests  to be performed to compare the electrical activity of different conditions at each sample, that is at each of the multiple scalp locations and at each of the multiple time points, while controlling for the multiple comparisons problem. Indeed, the assumption is that real effects will typically occur over multiple temporally and/or spatially adjacent time points and electrodes , showing that participants read and understood the sentences.The average accuracy for comprehension questions was 93% (SD = 21.01) occurred . Thirty-two of the aware participants perceived that one or more words in the sentence were replaced by string of Xs, while nine participants noticed that one or more words were replaced by jumbled letters. This finding is consistent with a number of studies in the literature . Participants estimated on average that 32 display changes .One of the goals of the current study was to investigate the neural correlates of visual and orthographic PoF effects during natural reading, as to date, these effects have only been studied in relation to paradigms presenting word pairs  or listsThe eye movement data showed that reading times on the pretarget word were longer when the parafoveal preview was formed from Xs compared with when the preview was the target word itself or comprised of a string of random letters see . The difThe eye movement results indicate that participants were quite sensitive to visually unusual strings in the parafovea , but they were less sensitive to visually word-like stimuli . These findings appear to be consistent with previous evidence that readers obtain at least visual information from words in the parafovea before fixating it e.g., , and repNext, let us consider the FRP data time-locked to the pretarget word. The topographies representing the FRP grand averages are shown in The left panels of Given that this study provides the opportunity to investigate whether different types of preview have a different influence on parafoveal processing, let us focus our attention on In sum the FRP results confirm our predictions that processing letter strings in the parafovea is far more similar to processing words in the parafovea than X-strings. Random letter strings appear word-like in the parafovea, thus, readers engage in visual and orthographic processing. However, some modest disruption occurs, likely during orthographic, or even morphological processing of a parafoveal letter string. In contrast, parafoveal previews comprised of Xs share very few features or letters (if any) with words. Readers are sensitive to this visual oddity in the parafovea, and upon detection of a string that is so unlike a word, they desist from continued  parafoveal processing of it. We believe that it is for this reason that we observe the greatest disruption to processing at the pretarget word for this condition.As we have noted, previous studies investigating lexical PoF effects have been inconsistent both in the eye movement and in the FRP literature. Analyses of eye movement measures on the pretarget word did not reveal any significant main effect of target word frequency, nor an interaction between parafoveal preview and target word frequency see . These rFinally, if readers process words in parallel, then we expected that they should be sensitive to orthographically illegal letter strings in the parafovea that do not have lexical status. As such, we would expect very large and significant differences in activation between the identity and letter string condition at the pretarget word see . Our faipreview positivity effect was obtained such that there was an attenuation (reduced negativity) of the N1 component, as well as a later effect such that there was an attenuation of the N400 component, when a word was preceded by an identical preview relative to several forms of invalid preview. Therefore, at the target word, we predicted differences in those component latency ranges between the identity and the X-string preview conditions, and between the identity and letter string preview conditions. In relation to the former comparison, we might suggest that X-string previews would delay any form of effective orthographic processing of the target (relative to processing in the identity condition) because these previews are so very visually and orthographically different from the target. In relation to the random letter string preview condition, we were more cautious, because to date there has been no work to examine how different forms of preview impact on neural correlates of lexical processing. Somewhat tentatively, we might expect that a letter string preview would produce inhibitory effects due to the dissimilarity of preview and target, but to a smaller extent than X-string previews, due to the fact that the letter string previews were constructed using visually similar letters to the target, and therefore, they share common features.One of the aims of our study was to see whether we could replicate in natural reading the findings for neural correlates that have been observed in RSVP-with-flanker-word presentation and saccadic word list reading experiments. We were particularly keen to explore for the first time whether differences in neural correlates could be observed at a target word that was preceded by a preview of a string of random letters compared with a string of Xs. Recall that in previous studies , an earlFirst, note that analyses of the eye movement data replicated the well-established preview effect. For all our eye movement measures see , fixatioNext, let us consider the FRP grand averages shown in Let us now turn to the permutation analyses see . As per preview positivity effect . These differences sustained through the 120-ms to 300-ms time window and into the 300-ms to 500-ms window also. In line with previous studies , we obtaIn sum, our FRP data suggest that differences between the identity condition and the X-string preview condition are more sustained than counterpart differences between the identity and letter string preview conditions. The reason for this is likely that the X-string previews prevent readers from engaging in effective parafoveal processing. Because strings comprised of Xs are perceived as visually unusual, further processing of the type that would occur naturally for a parafoveal word is inhibited, and full identification of the target word can only be initiated subsequently when the target becomes available in the fovea. In this way, the X-string preview results in a delay to processing in the temporal windows we considered in relation to target word foveation. In contrast, letter string previews are processed, at least at an orthographic level, in the parafovea. Thus, when the target is fixated, preprocessing of the preview in relation to the target, leads to inhibition, due to conflicting information between parafoveal and foveal vision, but to a lesser extent, due to preactivation of visual features of the target.One of the benchmark findings in experimental psycholinguistics is that the frequency of occurrence of a word determines the time for its recognition , 2009. AOur eye movement results showed a very clear word frequency effect. Low-frequency words led to significantly longer reading times than high-frequency words, and this effect increased from first fixation to gaze duration  or only occasionally examined  during natural reading of sentences.Our results replicated the preview effects reported both in the eye movement see . PreviewIn addition, we consider that the raster diagrams see  presenteFinally, the current study highlights the necessity to explore the timing of the word frequency effect under ecologically valid conditions. Neither our eye movement or FRP results provided evidence for extraction of word frequency from the parafovea—hence, providing no evidence to support parallel lexical processing. At the fovea, we replicated the well-established frequency effects in the eye movement data, but not in the FRPs. The reason why frequency effects did not appear in the FRP data for the present study is not clear, though again, we note that these effects also did not appear in the study reported by