Recently, short-term traffic prediction under conditions with corrupted or missing data has become a popular topic. Since a road section has predictive power regarding the adjacent roads at a specific location, this paper proposes a novel hybrid convolutional long short-term memory neural network model based on critical road sections (CRS-ConvLSTM NN) to predict the traffic evolution of global networks. The critical road sections that have the most powerful impact on the subnetwork are identified by a spatiotemporal correlation algorithm. Subsequently, the traffic speed of the critical road sections is used as the input to the ConvLSTM to predict the future traffic states of the entire network. The experimental results from a Beijing traffic network indicate that the CRS-ConvLSTM outperforms prevailing deep learning (DL) approaches for cases that consider critical road sections and the results validate the capability and generalizability of the model when predicting with different numbers of critical road sections. Real-time traffic state prediction plays a vital role in traffic management and public service. The ability to timely, accurately and efficiently predict the evolution of traffic can assist travelers and government agencies in reacting to possible congestion ahead of time. In recent decades, reliance on efficient access to massive traffic data, such as loop detectors and floating cars carrying global positioning system (GPS) devices, has provided an impetus for a considerable amount of data-driven computational approaches to estimate and forecast future traffic states. Compared with data-driven methods, mathematical or statistical models that are derived from macroscopic and microscopic theories of traffic flow exhibit difficulties addressing unstable traffic conditions and complex road settings due to their strong hypotheses and assumptions .Data-driven methods, including the autoregressive integrated moving average model (ARIMA), support vector machine (SVM), Bayesian network, neural network (NN) and so forth, have achieved promising results due to their greater potential in processing complex nonlinear problems . Among aThe influence between downstream and upstream areas has been studied for a long period of time . RecentlIn this paper, we propose a novel hybrid convolutional long short-term memory neural network model based on critical road sections (CRS-ConvLSTM NN) to predict future traffic conditions in the overall network. With the assistance of a spatiotemporal correlation algorithm (STCA), the critical road sections that have a significant impact on the local traffic networks are identified and extracted from the network. First, the spatial relevance among road sections is evaluated by establishing a spatial adjacency matrix. The matrix is constructed according to the topological relationships of the urban network. Then, the strength of the influence between a given road section and its neighbors under conditions of a 2 min to 1 h time delay can be calculated based on the correlation theory. The degree of influence can be adopted as a criterion for evaluating the importance of the roads. Because the value of the spatiotemporal correlation is different for each delay of each road section, the technique for order preference by similarity to an ideal solution (TOPSIS) is applied to synthetically consider its impact on the subnetwork by ranking their orders. Then, a certain proportion of road sections is regarded as the critical sections according to their ranks.In the prediction section, we introduce the ConvLSTM NN and show that it outperforms other models, such as artificial neural networks (ANNs), convolutional neural networks (CNNs), long short-term memory neural networks (LSTMs) and the stacked autoencoder model (SAE). The number of critical road sections we extracted from all the road sections affects the forecast performance in terms of the loss function of the CRS-ConvLSTM model. To the best of our knowledge, this paper presents the first report of a method for predicting the urban traffic evolution on a network-wide scale with partial sections that characterize the spatiotemporal features of the integral network.The contribution of the CRS-ConvLSTM model lies in addressing structural data failure problems by exploiting the potential of the predictive power among sections. With the preprocessing strategies exerted on the input data based on the STCA, our model can clearly reflect the spatial and temporal characteristics of the road network. Practical experiments on real-life speed data in a large-scale network of Beijing City demonstrate that the proposed CRS-ConvLSTM model can precisely predict the future traffic network states. Additionally, the fluctuation of performance under different numbers of critical road sections and various situations  is also tested.The remainder of our paper is outlined as follows. In recent decades, a considerable number of short-term traffic forecast models have been developed to predict future traffic states ranging from a few seconds to a few hours ahead. The approaches can be generally categorized into parametric and nonparametric approaches ,9. The pVlahogianni et al. indicated that artificial intelligence (AI) has gradually become the dominant approach for transportation prediction because of its inherent propensity for hypothesis-free techniques that are capable of addressing multidimensional and nonlinear problems . With thRecently, deep learning approaches capable of processing nonlinear high-dimensional problems have been increasingly developed; these approaches are capable of addressing complex traffic flow characterized by both spatial and temporal features. Different from other traditional ANNs models, deep learning algorithms extract inherent features of the data using multiple layers or deep architectures and thus achieve better results . Under tFurthermore, numerous efforts have been devoted to emphasizing the temporal characteristics and spatial dependencies of predictions . Min et In summary, since a wide spectrum of short-term prediction models have been applied in traffic forecasting, selecting the appropriate modeling approach for a specific circumstance is crucial for generating accurate and reliable prediction results. Among all the parametric and nonparametric approaches proposed for short-term traffic forecasting, deep learning methods have emerged as the alternative with the most potential due to their remarkable abilities for addressing nonlinear and high-dimensional problems. However, few prediction methods can achieve satisfactory accuracy under the conditions of structural missing data. Conventional techniques that compensate for incomplete data rely on filling in missing values with reasonable replacements before feeding the data into the prediction model . Our metThe study focuses on utilizing data from critical road sections to predict future traffic conditions of an overall urban transportation network. Specifically, the critical road sections are identified by a spatiotemporal correlation algorithm that simultaneously presents the spatial and temporal features of the traffic network. In this paper, a ConvLSTM NN that combines a CNN and LSTM is used to predict the future short-term traffic speed of the network.k-order spatial weights matrix through the topological adjacency relationships between road sections. If edges i and j are connected directly or indirectly, we call these two edges the “kth order neighbors” and their adjacency relationship can be expressed as Equation (1):A spatial weights matrix represents the spatial dependency among road sections in a traffic network. According to graph theory , a traffi and j are “kth order neighbors” and compose the square N × N spatial weights matrix k-order spatial relationship. For example, k = 1 means two road sections are directly connected; k = 2 means there is one road section between these two road sections; k = 3 means there are two road sections between these two road sections and so on. Then, the spatial weights matrix kth order neighbors. The spatial weights matrix E can be defined as the sum of the kth spatial weights matrix K is the highest order of the spatial weights matrix.The weights X and Y; the correlation distance between these two series is given by Equation (3):X and Y. X and Y, respectively and X and Y, respectively.Since the current states of a road section have predictive power on the future conditions of adjacent roads, the first step is to measure the spatiotemporal correlations among road sections. Several approaches, including the Manhattan distance, the Euclidean distance and so forth, are suitable to calculating the correlation . Allowini at time t, where , which is calculated by Equation (5):iR] is the number of the road sections, tT] is the time interval and E.The formulas listed above represent the basic expression of the correlation distance. To comprehensively consider the spatial correlation of the road network with the temporal correlation, a spatial weights matrix is introduced as a spatial indicator to improve the initial correlation distance. Let i and its adjacent road sections can be calculated by Equation (6):s = , (s > 0) is the time lag between the speeds of road section i and its adjacent road sections, i during a time duration of T, Then, the correlation between the speeds of road section i exert on the future state of its adjacent road sections. A greater value of i and its adjacent road sections under a time delay of s.The correlation distance provides a criterion to measure the spatiotemporal similarity among road sections. In this paper, if the correlation distance of a road section is smaller than that of the other sections, we call it a critical road section, which means that it has a powerful impact on the future traffic conditions of its adjacent road sections. To extract critical road sections of the traffic network, we need to rank the forces of all road sections based on correlation distances under every different delay.In the area of multiattribute decision analysis (MADA), several methods are commonly used to comprehensively consider the trade-offs between multiple indicators, including the preference ranking organization method for enrichment evaluation (PROMETHEE), elimination and choice expressing reality (ELECTRE), analytic hierarchy process (AHP), order of preference by similarity to ideal solution (TOPSIS) and so forth. . Among aStep 1: Create an evaluation matrix.R alternatives and S attributes that represent R road sections and S time lags respectively; hence, As mentioned above, we shall regard the correlation distance matrix (CDM) with elements Step 2: Calculate the weights under every time lag.s are calculated using the Euclidean distance as shown in Equation (8):i at time t and the other variables are similar to that of Equation (6). A smaller value of min(Ed) and max(Ed) denote the minimum and maximum of the Euclidean distance set, respectively.Different weights are assigned to different time lags with the consideration that closer time intervals have more similar traffic states. Thus, the weights under time lag Step 3: Determine the ideal worst solution s and the other variables are similar to that of Equations (7)–(9).The correlation distance is varied with the time lag; therefore, Step 4: Calculate the distance between and the ideal worst and the best solution. The weights are involved to identify different degrees of impact under each time lag, which can be determined by Equations (8) and (9). The distance between road section i and the worst solution The weighted Euclidean distance is applied to calculate the distance between i and the worst solution i and the worst and best solution, respectively. Other variables are similarly defined as in Equations (7)–(11).The distance between the correlation distance of road section Step 5: Calculate the similarity to the worst condition.i and its adjacent road section, which can be calculated by Equation (14):i stands for the number of the road section, which is the integrated correlation distance. Obviously, The similarity to the worst condition measures the degree of impact under all delays between road section Step 6: Rank the impact and identify the critical road sections according to i and its adjacent road sections. A greater value of i has a greater impact on the future conditions of its adjacent road sections. Thus, the impact is ordered by the value of Step 7: Split the time period and integrate the dataset.Due to the unbalanced distribution of living areas and working areas, the traffic tide phenomenon (TTP) frequently occurs in large cities . To distIn this paper, the deep learning approach ConvLSTM NN is used as the prediction method. The ConvLSTM NN presents the advantages of the CNN and LSTM methods and can simultaneously extract spatial and temporal features.i at time t. Then, the spatiotemporal matrix is established with the elements of i at time t. In addition, the row and column of the matrix are spatially and temporally associated, respectively. As a result, the daily traffic states can be mathematically transformed into a spatiotemporal matrix with T rows and R columns, which can be denoted as Equation (15):T is the length of time intervals and R is the number of road sections.Traffic information generated by the floating car with a GPS device consists of both spatial and temporal features, which should work in conjunction to predict future traffic states of the network. Assume that Along the time dimension , the time range that usually extends from the beginning to the end of the day is divided into narrow intervals of 2–15 min. Additionally, the speeds are also aggregated and averaged over 2–15 min in coordination with the time intervals. The length of the interval is determined primarily based on the sampling intervals and the data update frequency. Additionally, for real-time forecasting that focuses on the traffic evolution from 15 to 40 min, intervals that are too narrow or wide are inappropriate in practice.Along the spatial dimension , the speeds of each road section are arranged as a sequence representing the traffic state of the network during a certain time period. Notably, for our model, the spatiotemporal matrix of critical road sections and the overall network have the same temporal dimension but a different spatial dimension since the critical road sections are extracted from the latter in the light of the spatiotemporal features. Thus, to ensure the consistency of the two types of matrices in space, the sequence of the roads will not be disrupted. In simple terms, we draw the speeds of critical road sections from the original matrix and then merge these columns together into a matrix of critical road sections. The CNN has been successfully applied to traffic predictions because of its great potential to extract features using multiple layers. A typical CNN mainly consists of multiple convolution layers and pooling layers. The former contributes to mining spatial dependencies of road sections since every layer retrieves a distinct feature using different filters while the latter assists in reducing the number of parameters required for training the CNN under the premise of ensuring prediction accuracy. Because the input for a CNN can be intuitively regarded as an image in which each pixel value is associated with one type of traffic state during a certain time, a 2D CNN is naturally utilized to abstract spatial features between road sections. a is the prediction horizon and R is the number of road sections. The input of the CNN is the historical traffic speed of critical road sections {t−n), n is the look back step and lth convolutional and pooling layers with r filters and the weights and bias of lth layers be :To valid the efficiency of our methodology, data collected by a taxi equipped with GPS devices from 1 June 2015 to 13 October 2015 (92 working days) in Beijing, the capital of China, are utilized to train the ConvLSTM NN model and predict the future traffic speed of network. The updating frequency of the data is 2 min and the time period from 6:00:00 to 23:00:00 is considered for the high travel demand period, which has been repeatedly observed. Accounting for the condition that the traffic state varies with every time interval, we can observe 511 traffic states per day.2 area. Two thirds of the data (61 days) are regarded as training data, while the remainder (31 days) compose the testing data. Additionally, the structural missing data marked by pentagram can be obviously observed.In our paper, a sub-transportation network near West Second Ring Road is selected as the research objective, as shown in k-order spatial weights matrix is constructed to describe the topology of the traffic network. The spatial influence is measured by calculating the Euclidean distance between different road sections as shown in Equation (25):k denotes k-order, T is the total time period and i at time t.The spatial similarity of traffic flow can be easily obtained according to the topology of the network, which will be viewed as the static properties of the traffic flow. The k value when k is less than 7. It is obvious that if k is bigger than 7, distant roads have weak impact on the given road. And from To capture the stochastic and time-varying dynamic characteristics, the spatiotemporal features of the traffic flow should be mined by the spatial and temporal correlations. Considering the traffic flow is commonly repeated every working day , we averAnother important factor that should be taken into account for such predictions is the influence of TTP. The temporal features of traffic flow vary dynamically over time, which leads to the transition of the critical road sections. For example, one road section may play a relatively considerable role in the morning peak time rather than the evening peak time. By analyzing the evolution of traffic flow and using previous empirical knowledge , one daySubsequently, TOPSIS is employed to measure the degree of predictive power among road sections during each time period and then extract critical road sections according to the integrated correlation distances. First, the evaluation matrix of each period is constructed. Since the length of each period is different, the size of the CDM varies based on the context of each period as shown in Equation (26):The ideal worst and best solutions are the worst and the best values of the correlation distance under each time lag, respectively. In addition, the integrated correlation distance Eventually, the impact of roads is ranked according to the integrated correlation distance. Notably, to measure the performance under different road extracting rates In a traffic network, nearby road sections are more relative that distant ones. In order to retain the spatial correlation between adjacent roads, it is necessary to arrange adjacent roads together. An effective method is to straighten consecutive roads into a line and then construct a spatiotemporal state matrix. As stated in Ma et al. (2017) , a simplWithout a general regulation for constructing the structure of DL NNs, the selection of hyperparameters mainly depends on expert experience. In addition, compared with the prevalent CNN or LSTM NN, the spatial dimensions of the inputs and outputs in our CRS-ConvLSTM are unequal. In this section, we take 195 critical road sections , we have qualitatively described the influence between the road and its k-order road sections. And we can conclude that it is better to set k less than 7. Here, we adopted root mean square error (RMSE) and root mean squared error proportional (RMSEP) to evaluate the performance of the model in the case of different spatial weight matrices; the relevant formulas are show in Equations (27) and (28):k is 5. Consequently, we set k to 5 which means we regard at most 5-order neighbors as adjacent road sections.What to be noted is that we extract 195 critical road sections (i.e., extracting rate In order to test the performance of our CRS-ConvLSTM model, four popular deep learning-based algorithms are selected for comparison: ANN, CNN, LSTM and SAE. The ANN adopts a plain and shallow structure  to proceTo explore the effect of the extracting rate For drop rate in The decline of the accuracy compared with the overall prediction model is ranging from 1.638% to 10.180% for the RMSE and 1.841% to 14.537% for the RMSEP. In general, the decrease in accuracy is reasonable and within the acceptable limits, which demonstrates the validity and generalizability of the approach and indicates that certain road sections account for a lower contribution toward the prediction. Among all the extracting rates at less than 1.0, the value of 0.95 achieves the optimal result, which is primarily because the most minor road sections are removed. Additionally, when the extracting rate reaches a value of 0.65, the predictive performance gradually tends to become unstable, which is likely because too many roads are omitted. In the practical situation, corrupted or missing data generally occur because of monitoring equipment failure, extreme weather, data transmission errors and so forth and such data weaken the effectiveness of the prediction model or even disable the model . As statFor the purpose of testing the performance of our model under random structural missing data, we stochastically extract a part of the road sections at the rate of An extreme case is also involved so that the most critical road sections with greater values of However, although the proposed approach seems to be ineffective under stochastic and extreme cases, the problems caused by structural missing data are indeed alleviated, especially in the case of critical road sections, thus validating the effectiveness of our CRS-ConvLSTM. Moreover, the results present a meaningful and worthy instruction for traffic management that more monitoring devices should be allocated along critical road sections to improve the level of prediction.Structural missing data usually have a massive negative effect on short-term traffic state predictions. In this paper, we propose a ConvLSTM traffic speed prediction method to alleviate the problem caused by structural missing data. First, the spatiotemporal features are extracted to measure the impact of road sections on adjacent roads. Spatial weights matrices are constructed to represent the topology of the network and the correlation distance is adopted to calculate the degree of impact between a road section and its local network. Then, correlation distances are integrated by TOPSIS to rank the critical degree of all roads. Thus, critical road sections can be distinguished according to their order and subsequently extracted as the input of the prediction model at different extracting rates. In the prediction section, the proposed ConvLSTM NN is employed to forecast future traffic states of the overall network based on the speeds of critical road sections.To demonstrate the validation of the proposed model, a group of empirical experiments are performed using the traffic speed collected at 2 min intervals from a Beijing transportation network with 278 links. A sensitivity analysis about the different spatial weight matrices is conducted to prove that the availability of 5-order spatial weight matrix. Compared with the ANN, CNN, LSTM and SAE, our CRS-ConvLSTM achieves an acceptable accuracy with RMSE is of 6.955 at the extracting rate of 0.7. Furthermore, the prediction accuracy under different extracting rates is tested. The results show that in general, the RMSE increases as the extracting rate decreases and when the top 95% of road sections are used as input, we can acquire the optimal result. A bump up around 0.85 is probably because of the difficulty of recovering structural missing data and the distribution of the critical road sections. However, when the Subsequently, a situation of stochastic and extreme structural missing data in a real scenario is simulated by extracting certain roads at rates ranging from 0.60 to 0.95. All the results in the context of random and extreme cases are inferior to those of critical road sections. Although obvious regularities are not observed for the RMSEs of the model under different extracting rates, the critical road sections undoubtedly have more prominent predictive power. Moreover, the validity of the spatiotemporal correlation algorithm has also been confirmed.In summary, we tested the performance of the proposed in a large-scale network, which proved the availability of the proposed method. In practical implementation, considering that we adopted deep learning approaches, at least 3 months historical traffic data is necessary to train and test the prediction model. Before prediction, the spatiotemporal features should be extracted to draw the critical road sections out of the overall network. Once the structure of the model is determined and the parameters is trained by historical data, short-term prediction can be conducted based on the current states of the critical road sections. Thus, online prediction of the overall traffic network can be performed by states of critical road sections and the trained model.For future studies and applications, the quality of data should be addressed since random missing values affect the prediction accuracy. The imputation method such tensor decomposition is emerging as a potential approach to remedy missing traffic data. Also, the hierarchy, length and other inherent attributes of the road should be taken into account when extracting spatial correlations between roads. Additionally, since weather, temperature, air quality and other external factors affect travel demand, these factors can be comprehensively considered to improve prediction accuracy. Another strategy for improving prediction accuracy is the inclusion of more detection devices at critical road sections to achieve a more reliable prediction.


This paper investigates the secrecy performance of a cognitive millimeter wave (mmWave) wiretap sensor network, where the secondary transmitter (SU-Tx) intends to communicate with a secondary sensor node under the interference temperature constraint of the primary sensor node. We consider that the random-location eavesdroppers may reside in the signal beam of the secondary network, so that confidential information can still be intercepted. Also, the interference to the primary network is one of the critical issues when the signal beam of the secondary network is aligned with the primary sensor node. Key features of mmWave networks, such as large number of antennas, variable propagation law and sensitivity to blockages, are taken into consideration. Moreover, an eavesdropper-exclusion sector guard zone around SU-Tx is introduced to improve the secrecy performance of the secondary network. By using stochastic geometry, closed-form expression for secrecy throughput (ST) achieved by the secondary sensor node is obtained to investigate secrecy performance. We also carry out the asymptotic analysis to facilitate the performance evaluation in the high transmit power region. Numerical results demonstrate that the interference temperature constraint of the primary sensor node enables us to balance secrecy performance of the secondary network, and provides interesting insights into how the system performance of the secondary network that is influenced by various system parameters: eavesdropper density, antenna gain and sector guard zone radius. Furthermore, blockages are beneficial to improve ST of the secondary sensor node under certain conditions. Future communication will probably realize man–machine–object cooperative communication and ultra-densely connections to achieve full coverage, and higher spectral efficiency will become a key issue. Owing to the spectrum scarcity problem and the increasing data rate demand with different quality of service (QoS), new spectrum bands are needed to meet explosive traffic requirements. In order to improve spectrum utilization efficiency and avoid interference with other operational networks, it is necessary to enable cognitive radio characteristics in densely deployed future networks. On the other hand, as a promising candidate for future mobile networks, millimeter-wave (mmWave) communication  has attrMore recently, researchers have conducted some studies on cognitive mmWave networks, most of which focus on various techniques to dynamically avoid interference and lessen mutual interference. In order to minimize the mutual interference between cognitive users and other operational networks in fifth-generation 5G) networks, wavelet packet transform was applied to sparse code multiple access systems for spectrum sensing and multi-user access modulation in . Using aG networkPhysical layer security (PLS) attempts to protect wireless networks from wiretapping by utilizing randomness of wireless medium at the physical layer ,8. ThereOn the other hand, PLS in mmWave systems has aroused interest with enthusiasm ,21,22,23All the above works were either focused on the PLS of traditional microwave cognitive wireless networks or on the analysis of the secrecy performance of mmWave networks, but were rarely investigated on the PLS for cognitive mmWave wireless networks. In addition, as mentioned earlier, future communications will probably be ultra-densely heterogeneous networks, where communication networks can be built with a combination of macro cells and small cells. The existing macro cells guarantee coverage and mobility in the ultra high frequency band, while small cells are needed in order to achieve gigabit rates over the mmWave range of 30–300 GHz. Effective utilization of small cell spectrum bands is one of the ways to improve the obtainable capacity and data rate of heterogeneous networks. It is a challenging task that could be accomplished through the spectrum sensing capability of cognitive radio . TherefoBy using stochastic geometry approaches, we model the cognitive mmWave wiretap sensor networks to characterize the random spatial locations of primary and secondary nodes, as well as the eavesdroppers. Taking into account the effect of blockage, the links are either LOS or NLOS. A sector secrecy guard zone, synonymously referred to as the eavesdropper-exclusion area, is invoked around the secondary transmitter (SU-Tx) for improving the PLS.Considering that the random-location of both eavesdroppers and the primary sensor node may reside in the signal beam of the secondary network, we derive analytical expressions of secrecy throughput (ST) for the secondary sensor node to measure secrecy performance of the system with arbitrary system parameters. Subsequently, an asymptotic expression is proposed to gain additional insight into the performance evaluation and design for the proposed system.The results show that the interference temperature constraint of the primary sensor node can be used to balance secrecy performance of the secondary network, which indicates losing the interference temperature constraint of the primary node would worsen secrecy performance. Increasing the radius of the sector guard zone can improve ST. Furthermore, it is also shown that increasing the transmit power or antenna gain of the secondary network does not necessarily lead to the improvement of ST, and sometimes deteriorate ST. Besides, blockages can also be used to improve ST of SU-Rx under certain conditions.This paper studies the PLS in cognitive mmWave wiretap sensor networks. Our analysis considers the key characteristics of the mmWave channel and blockage density and the effects of different antenna arrays gains. In particular, the main contributions of this paper in specific terms areThe rest of the paper is organized as follows. The system model and mmWave channel characteristics are provided in As shown in r around the SU-Tx is introduced, where eavesdroppers are not allowed to roam. Similar security mechanisms have been used in the literature [For mathematical tractability, the antenna pattern is assumed to be a sectorial model ,26,30. Cterature ,31,32.In outdoor mmWave scenarios, a channel between SU-Tx and receiver may be LOS or NLOS link in the presence of blockages . PLrd reach link , and theTo guarantee the interference temperature constraint of the primary network, the instantaneous interference power at the primary sensor node from the SU-Tx should lower than a given threshold In this model, the eavesdroppers attempt to intercept the confidential information of the system, assuming that the SU-Tx do not know the instantaneous CSI of eavesdroppers. Non-colluding eavesdroppers are taken into account where eavesdroppers decode the information independently. For such case, the eavesdropper closest to SU-Tx is not necessarily the most detrimental, but the eavesdropper with the best channel to SU-Tx. In addition, we consider the worst-case scenario of cognitive mmWave wiretap sensor networks, in which the eavesdroppers are assumed to have powerful detection capabilities. In fact, this assumption overestimates the anti-jamming ability of the eavesdroppers. Then, the instantaneous SINR of detecting the information of the secondary sensor node at the most detrimental eavesdropper can be expressed as follows:Of particular note, we consider that the primary and secondary sensor nodes and eavesdroppers are located in the main lobe. This is because the main lobe has the greatest interference to the primary sensor node, which may cause the primary network performance to be deteriorated seriously. At the same time, the situation where the eavesdropper is located in the main lobe poses a serious threat to the security of the secondary sensor network. Thus, the considered scenario is the worst-case which have been widely adopted in mmWave systems with secrecy considerations; e.g., ,34.In this section, we elaborate to evaluate the ST of secondary sensor node for the proposed cognitive mmWave wiretap sensor networks.The ST of the secondary sensor node can be expressed asηU=RmsPrγTherefore, further derivation of Equation  can be eThen, we solve the first derivative of CDF and obtain the PDF of Now, we focus our attention on deriving the integrals Equation . Q1 can Whereas, we can get Equation , which cFrom the derivation of Let us now turn our attention to solving the integral Equation . Then Q2Consequently, we have From Equation , we can Resultantly, the calculation for the ST of the secondary sensor node is complete. Then, Equation  and Q2 fEquation .Remark 1.According to Equation , ηU is a.To extract additional insights about cognitive mmWave sensor wiretap networks, we derive the asymptotic expression of the ST with Equation , the asyEquation , and theHere, numerical results of the proposed cognitive mmWave wiretap networks are presented to evaluate its performance. The thermal noise power is assumed to be −90 dBm, and considering an mmWave system with carrier frequency at 28 GHz. As pointed out by , the parIn this paper, we have studied the secrecy performance of the secondary sensor network under the interference temperature constraint of a primary sensor node. Specifically, stochastic geometry-based techniques were adopted for modeling both the locations of sensor nodes and of the eavesdroppers in the networks considered. We considered a special case, where the eavesdroppers and the primary sensor node may be in the secondary network signal beam. Combining the mmWave characteristics, the exact analysis on ST of the secondary sensor node was conducted, and an asymptotic result of ST was further provided. Our results showed that the interference temperature constraint of the primary network can be used to balance the secrecy performance of the secondary network. Moreover, blockages were beneficial to improve the ST of the secondary sensor node under certain conditions. In future works, complex scenarios such as imperfect CSI, base-station (BS) cooperation and nonorthogonal multiple access (NOMA) will be considered. In addition, combining the results presented in this paper with UAV, the secrecy transmission capability may be analyzed.


Object tracking in challenging videos is a hot topic in machine vision. Recently, novel training-based detectors, especially using the powerful deep learning schemes, have been proposed to detect objects in still images. However, there is still a semantic gap between the object detectors and higher level applications like object tracking in videos. This paper presents a comparative study of outstanding learning-based object detectors such as ACF, Region-Based Convolutional Neural Network (RCNN), FastRCNN, FasterRCNN and You Only Look Once (YOLO) for object tracking. We use an online and offline training method for tracking. The online tracker trains the detectors with a generated synthetic set of images from the object of interest in the first frame. Then, the detectors detect the objects of interest in the next frames. The detector is updated online by using the detected objects from the last frames of the video. The offline tracker uses the detector for object detection in still images and then a tracker based on Kalman filter associates the objects among video frames. Our research is performed on a TLD dataset which contains challenging situations for tracking. Source codes and implementation details for the trackers are published to make both the reproduction of the results reported in this paper and the re-use and further development of the trackers for other researchers. The results demonstrate that ACF and YOLO trackers show more stability than the other trackers. With knowing the location of a desired object in the first frame, the task of tracking of this object becomes a fascinating topic for video processing from both scientific and industrial viewpoints. This task becomes scientifically more interesting when there is complexity involved in video sequences. This complexity can include a moving camera, uncertainty of the object, background clutter, small size and low resolution of the object, size variation, appearance changes, occlusion, articular objects, illumination change and out-of-plane rotation. Some of the challenges are shown in Recently many methods have been developed to overcome the challenging object tracking problem in videos. Some trackers focus on human tracking with various techniques such as common energy minimizing ,2 and daEach tracker has its own strength and weakness. For instance, the tracking problem can be formulated as a correlation between successive frames ,12,13,14Object tracking can be formulated in terms of object detection. For example, SURF features , which wChen et al.  introducDing et al.  modeled Yun et al.  use an oIn a method presented in , authorsRecently, there is a drastic progress in the area of object detection by using learning-based techniques like deep learning. However, this progress was not extended to trackers. In this paper, five famous training-based object detectors i.e., ACF , RCNN 4, FastRCNThe object detection term refers to find an object in an image and object tracking means to follow an object of interest within a group of frames in a video. In this paper, the trackers follow the object of interest using a tracking-by-detection method. The tracking-by-detection means the same object is detected in successive frames of the video.The object detector performs the first part and the second part is a time series analyzer for tracking. The online tracker trains a detector with the positive and negative data generated from the first frame and then the detector is applied to the next frames of a certain part of the video, the detector is re-trained with the recently detected objects within the video part.To the best knowledge of the authors, the five mentioned detectors have not previously been compared in online and offline trackers. The online tracker is far different from two-step trackers ,30,32,33The rest of the paper is organized as follows: This section explains the new object detectors i.e., ACF , RCNN 4, FastRCNFor an ACF (Aggregate Channel Features) detector, a channel refers to a certain component that defines pixel values in image . ACF theRegion-Based Convolutional Neural Network (RCNN) is an object detector based on Convolutional Neural Network (CNN). CNN performs convolution products on small patches of the input map of the layer. Thus, extracting features are carrying information about local patterns . A typicThe region proposals with greater than 0.5 IoU (Intersection over Union) overlap with a ground truth (defined by a user) are classified as positive and the rest of the proposals are classified as negative. The RCNN has 3 convolutional layers and 2 fully connected layers. RCNN is slow because it performs a CNN for each proposal, without computation sharing.FastRCNN made improvements to RCNN to increase the process speed. It shares computation of the convolution layers among different proposals. Since the convolutional layer does not change the spatial relationship between the adjacent pixels, it projects coordinates in the raw image to corresponding neurons in the convolutional layer. Therefore, the whole image can be computed through the convolutional layer once and the processing time is saved . FastRCNFasterRCNN  brings ox and y coordinates directly. YOLO is faster and more precise than FasterRCNN.YOLO is state-of-the-art in object detection. It uses deep networks and a hierarchical structure for object labeling and classification real-time . YOLO stThe steps of the online object tracking method are shown in r Tv see . The trat, then the following equation expresses the online tracker output for this frame  to investigate its effect on tracking accuracy. The number and types of the synthetic data are given in The above parameters are selected in an optimized way to preserve both tracker accuracy and speed. Initially, a small training set was chosen and poor results were obtained. Then, the above parameters were gradually optimized to maximize the tracking accuracy and speed. Therefore, they are independent on type of objects and videos. The detailed information regarding the parameter optimization is given in The detected objects within frames of the first group k is the frame index, and The length of each group shown in  and callThe offline tracker steps are shown in Unlike the online tracker , the weiFor more detailed information about the Kalman filter we refer to ,51. We uIf YOLO does not detect any object in a frame or if the Euclidean distance between the Kalman filter response and the nearest YOLO response is more than a predefined threshold (100 pixels), then the object is considered to be occluded with other objects or left the scene. YOLO threshold value is set to 0.15 to minimize the probable object missing. Lower and higher values lead to high false alarm rate and object missing respectively. Since YOLO has a strong pre-trained model for object detection (available in ) and dueIntermediate results of the offline tracker are shown in To validate the trackers, one set of 10 videos from TLD  and VOT The parameters concerning the online trackers are set according to These parameters were selected using initial experiments to preserve both accuracy and speed of the trackers. Since the datasets contain various type of objects with different sizes and shapes, in different scenes, backgrounds, illumination conditions and different degrees of occlusion, the generality of the selected parameters is guaranteed.For the synthetic data as mentioned in R, the precisionP and the F-measureF. The F-measure is calculated using The experiments in this section use the following evaluation procedure. The trackers are initialized in the first frame of a video sequence and track the object of interest shown in  up to th percent . The num percent .The detailed results of the trackers on the datasets in terms of Recall, Precision and F-measure are shown in and SET2  respectiIn another experiment, we changed the training vector length The trackers stability using average F-measure is shown in In an ablative study on the online ACF tracker, we removed the updating/training process. The results of the study are shown and compared in The visualization comparison of the trackers for selected frames of the 10 videos and the ground truth is shown in The trackers have been implemented on a hardware system with a specification as shown in For each tracker, the average running time of all 10 videos has been measured and shown in The ACF tracker has the best results among the online trackers from both accuracy and speed viewpoints. ACF has effective implementation, because it runs on the CPU machine instead of GPU for the other online trackers.Among the RCNN-based trackers , RCNN has the best tracking accuracy. Though, FastRCNN and FasterRCNN are very fast in test phase, their tracking process is slow because they are very slow in training phase.Since the YOLO tracker was implemented offline, it is the fastest tracker. YOLO is not qualified for online tracking, because it is very slow in training phase.For human tracking from the front and side views, the combination of YOLO and Kalman filter shows the best results.We recommend to use the ACF tracker in unknown objects tracking because YOLO does not detect them whereas the ACF tracker does.Compared to YOLO and ACF, the RCNN-based trackers show less accuracy because they don’t have a very deep structure . By using a deeper convolutional neural network like YOLO the accuracy is dramatically increased.The comparative study among the trackers is concluded as follows:The training vector length YOLO detector detects cars from the top view, but the object classification precision is low.For human detection, since YOLO was biased to the data from the front view, although the YOLO detection results from the view is very good the classification results is disappointing.Though, the object detection based on deep learning has recently improved, further improvement is still necessary. YOLO detection should be trained using a big dataset including more various views of the objects.Some results of our research are not limited to tracking. They are stated as follows:In this paper, we did a comprehensive comparative study in the context of object tracking using five famous recently proposed detectors. Two trackers based on online and offline tracking were used. The online tracker first generates positive and negative samples from the first frame and then trains detectors. The detector detects online the objects of interest in next frames and put them in a training vector. The detector is updated in certain intervals using the training vector. The detector detects the objects of interest in the next frames until the last frame. The ACF tracker showed the best results among the examined methods for the online tracking from both speed and accuracy perspectives. In the offline scenario, YOLO detector generates some candidates, then the tracker follows the object of interest using Kalman filter. Extensive experiments showed thatthe YOLO tracker outperforms the rest of the trackers. For the future work, YOLO detector will be trained using an updated dataset to improve the detection results from the top view. The experiments will be extended to other videos in VOT benchmark. We aim to extend our methods for multiple object tracking because all the detectors i.e., ACF, RCNN, FastRCNN, FasterRCNN and YOLO have the capability of multiple object detection. For the online trackers in each iteration of the training phase, instead of a single object, we will define multiple objects and the detectors will output different labels for different objects. In the case of offline tracking, YOLO is able to detect multiple objects as shown in


A predictive guidance obstacle avoidance algorithm (PGOA) in unknown environments is proposed for autonomous underwater vehicle (AUV) that must adapt to multiple complex obstacle environments. Using the environmental information collected by the Forward-looking Sonar (FLS), the obstacle boundary is simplified by the convex algorithm and Bessel interpolation. Combining the predictive control secondary optimization function and the obstacle avoidance weight function, the predicting obstacle avoidance trajectory parameters are obtained. According to different types of obstacle environments, the corresponding obstacle avoidance rules are formulated. Lastly, combining with the obstacle avoidance parameters and rules, the AUV’s predicting obstacle avoidance trajectory point is obtained. Then AUV can successfully achieve obstacle avoidance using the guidance algorithm. The simulation results show that the PGOA algorithm can better predict the trajectory point of the obstacle avoidance path of AUV, and the secondary optimization function can successfully achieve collision avoidance for different complex obstacle environments. Lastly, comparing the execution efficiency and cost of different algorithms, which deal with various complex obstacle environments, simulation experiment results indicate the high efficiency and great adaptability of the proposed algorithm. Autonomous underwater vehicle (AUV)  is an imIn recent years, significant contributions have been made by many researchers in developing obstacle avoidance methods and applying them to various obstacle avoidance environments for AUV. These problems can be classified into the global obstacle avoidance path planning problems and local obstacle avoidance methods. When the global environmental information including various obstacles are known, the global obstacle avoidance path planning problem becomes a nonlinear optimal programming problem to find global optimal solutions on the pre-requisite that global variables are known. For example, two missile guidance algorithms are proposed for the intercept and for the rendezvous of a maneuvering target while avoiding a static obstacle by a specified avoidance distance . HoweverSome significant achievements have been obtained in the obstacle avoidance problem. Masoud, Dadgar, et al. proposed an A-RPSO (Adaptive Robot PSO) algorithm , which cBased on the previously mentioned obstacle avoidance problems, and combining target search and tracking in unknown underwater environments with complex irregular obstacles, an obstacle avoidance method for AUV based on PGOA is proposed. The main idea of this algorithm is: The FLS (Forward-looking sonar) is equipped in the AUV front to obtain the obstacle information. Obstacles are classified according to the detected character information. Then, the contour convex algorithm and Bezier interpolation are used to change the irregular contour into a convex polygon, which simplifies the boundaries of the irregular complex obstacle, so that the AUV can respond quickly to various obstacle environments. In the second stage, by using the maximum turning radius calculation method based on the obstacle type and boundary data, different turning radii are obtained. Then, by using obstacle avoidance parameters and obstacle avoidance rules, the obstacle avoidance weight function is established. Lastly, the prediction parameters are obtained based on the predictive control second optimization function. In the final stage, the corresponding predictive guidance track points are calculated for the parameters, which are obtained by the obstacle avoidance rules and weight functions of different obstacle environment types. Combining with the guidance ,34 methoThe rest of this paper is organized as follows. In unknown underwater environments, AUV may encounter complex terrain such as reefs, islands, trenches, and valleys, and the obstacles they meet are different in shape, complexity, and quantity. While ensuring the completion of underwater missions such as target search and intelligence reconnaissance, guaranteeing its own safety, it is necessary that AUV has the ability to avoid these complex obstacles in each situation. Therefore, it is important to analyze the factors that may affect cooperative searching for multi-AUVs.1. Visual noise and threshold: Through the AUV-configured forward-looking sonar, the observation data is affected by Gaussian noise during the measurement of the observation target. In addition, AUV observation of the unknown environment through the forward-looking sonar is also limited by the sensor’s detection distance. As such, the sonar cannot observe and extract environmental features that are outside the view.2. Movement limitation: The AUV’s own motion state will be affected by equipment such as thrusters and rudders. In an unknown underwater environment, the AUV will also be affected by unknown factors such as ocean currents and submarine topography. Therefore, the influence of the AUV motion restriction features on the track planning during the AUV navigation process need to be considered.3. Obstacle types: When AUV performs tasks in an unknown underwater environment, it encounters a variety of obstacle types, such as simple convex obstacles, complex convex obstacles, and complex vortex obstacles. Therefore, a single obstacle avoidance algorithm cannot solve sudden problems. Therefore, different obstacle avoidance strategies and algorithms are developed in this paper for different obstacle types appearing in the AUV field of view, so as to achieve the optimal obstacle avoidance effect.4. Obstacle avoidance: AUVs may operate in an unknown underwater environment, and it is inevitable for AUVs to encounter obstacles, which may threaten the AUV’s normal trajectory in the process of performing target searching tasks. Therefore, the AUV is expected to have the ability to avoid obstacles in a timely manner, to ensure that AUV can travel safely and reliably. Ultimately, the economic cost caused by equipment damage can be avoided and the searching task can be reliably completed.The AUV four-degrees-of-freedom constant-speed motion model In this paper, according to the standard AUV equation of motion , the updr is a heading velocity variable in this case. AUV is affected by its own equipment, and its speed Because the roll movement is uncontrollable for AUV and the simulation environment of the obstacle avoidance algorithm is built on the 2-D environment, defining ified as :(3){u=vcR of 150 m, a horizontal opening angle Based on the unknown underwater environment, the real multi-beam active forward-looking sonar data is simulated through a mathematical model of forward-looking sonar  in this The mathematical model for FLS is established, which can describe the constraint relationship between the target and the FLS. Then the data information of the detected object is obtained. The model can be given by the equation below.Because the forward-looking sonar equipped on the AUV can be easily affected by the medium of water or other external factors during the data collection process, such as data interference, the measurement of the environmental characteristics is likely to be affected. Therefore, the description of the sonar is given in Equation (6) .(6)yx−qThe above description indicates that, when the relative position of the FLS and the detected object is outside the sight range, or there is an obstacle between the sonar and the detected object, obtaining feedback of characteristic information is unavailable. If there is characteristic information in the sight range, the observed data indicates the distance increases, and the measured disturbance increases, accordingly.For a wide expanse of unknown waters, when AUVs are dispatched to perform underwater operations, they often encounter unpredictable harsh environments with obstacles everywhere, which affects the normal movement of AUVs. For example, (1) Convex ObstaclesIn this paper, the sporadic distribution of a small number of convex obstacles is defined as a simple convex obstacle environment type in an unknown environment in (2) No-Convex ObstaclesIn some cases, AUV can encounter some non-convex obstacles, such as concave structures or helical shape. In such a case, a simple obstacle avoidance algorithm is not enough for AUVs escaping from obstacles in (3) Dense Convex Obstacles Part 2.1(3) describes the types of obstacle structures. In reality, there is also an environment of densely distributed convex obstacles in AUV uses FLS to realize the collision avoidance function. A real-time obstacle avoidance strategy is proposed in this paper. All obstacles in the environment are unknown, and their shapes and positions are randomly generated. The obstacle boundary is generated based on the detected information of the FLS. Considering that the AUV pitch angle rarely changes, the multi-beams sector on the horizon in the body coordinates is used. The purple lines are the sonar beams on the horizontal plane in the AUV body coordinates. The gray part is the obstacle, and the blue boundary is the obstacle contour curve .i, j are the serial numbers of beams, δ, ζ is the arbitrary nature number [To improve the effectiveness and safety of the AUV in obstacle avoidance when performing underwater tasks, the obstacles are divided into four conditions according to the positions of obstacles relative to AUV: bounded obstacles, left bounded obstacles, right bounded obstacles, and left and right edge unbounded obstacles. When the obstacle enters the FLS detection range, the sonar will classify the obstacle based on detected data by emitting 80 beams. The detection zone is a fan-shaped range of 80 m, as shown in e number .(1) If the boundary of the obstacle is in the beam range of the FLS, the current obstacle is considered as the bounded obstacle (BO).(2) If the right boundary of the obstacle is outside the sonar beam range, and the left boundary is in the range, the detected obstacle is called the left bounded obstacle (LB).(3) If the left boundary of the obstacle is outside the detection range of the sonar, and the right boundary is in the range, the obstacle is defined as the right bounded obstacle (RBO).(4) If both sides of the obstacle are outside the detection range of the sonar, the obstacle is defined as an unbounded obstacle (UBO).First, the obstacle information in the sight range is obtained using FLS of AUV. The detected data is stored in the matrix Data in FLS can output a data structure for the detected obstacles, where the obstacle bounder data are stored in the same array. In reality, the obstacle shapes are always irregular, so the output data from the sonar cannot be directly used to avoid obstacles. In this paper, the convex hull algorithm combined with the simplified Bezier interpolation algorithm is used to transform the obstacle outline into a regular shape, and the data is smoothed. It not only improves the accuracy of obstacle avoidance, but also avoids the adverse effects of complex obstacles. In Step 1: The data output from the sonar beam line is solved as an obstacle boundary point.Step 2: For the set X of the obstacle boundary points calculated by the solution, a convex hull set, which includes all the points in the X set, needs to be found to replace the X set. Then we select the leftmost and lowest point in the set X as the origin of the polar coordinates. Then we sequence all the points in the set, based on the principle that their distance from the polar origin is short to long and the polar angle ranges from small to large. Subsequently, the ray is selected, which is generated by the second point after sorting and the polar coordinates. Then the vector angle from small to large is found by using the vector cross method. The convex hull is found, according to the distance from small to large, if the angles are the same.Step 3: The convex hull set, calculated in step 2, represents a simplified set of contour points of the obstacle that are used as boundary points of the Bessel interpolation to generate the smooth obstacle boundary points. The specific calculation is shown below.Step 4: The last processed boundary points are sorted from left to right in the order of the sonar beam. The final result is shown in the following figure. The red line is the contour line processed by the convex algorithm, and the blue line is the final processing result.When AUV navigates at 2 m/s in an underwater environment without ocean currents, and the rudder angle is set at a maximum steering angle of 35°, the minimum turning radius is about five times the length of the AUV. It takes approximately 3.5 seconds for the rudder angle of the AUV to vary from 0° to 35°. If the time delay of the steering angle transition is considered, the trajectory deviation distance is 1–1.5 m, which is small compared to the turning radius. To simplify the problem that the deviation distance is neglected. In other words, the trajectory deviation rotation is replaced by an arc with a certain radius.o is the center of the circle where the maximum conversion radius is located. In The maximum turning radius is expressed as follows .(13)ADiBased on Equations (16) and (17):The maximum turning radius obtained by Equations (18) and (19) is as follows:If AUV avoids obstacles by rounding the right edge of the obstacle, then Otherwise, when the AUV avoids the obstacle around the left edge of the obstacle, the expected maximum turning radius is expressed as follows.Safe obstacle avoidance distance and the emergency obstacle avoidance distance is designed in the AUV obstacle avoidance process. Therefore, the following rules are formulated to deal with the two existing situations.Rule 1: When there is a safe obstacle avoidance distancea. If it is a left bounded obstacle that meets the conditions for safe obstacle avoidance, the AUV turns to the left side of the obstacle to avoid it.b. If it is a right bounded obstacle that meets the conditions for safe obstacle avoidance, the AUV turns to the right side of the obstacle to avoid it.c. If it is a bounded obstacle or an unbounded obstacle that meets the safe obstacle avoidance distance, the AUV turns in the direction close to the virtual target to avoid the obstacle.Rule 2: When an emergency obstacle avoidance situation occurs, a corresponding rule needs to be established to ensure the safety of the AUV. To ensure absolute safety, in practice, it is necessary to immediately turn off the propeller and initiate reverse propulsion in order to offset the forward speed caused by inertia. However, this extreme situation is not considered in this article.a. If there is an emergency obstacle avoidance setting where a bounded obstacle or an unbounded obstacle exists, the current AUV heading is taken as the dividing line to estimate the boundary point of the obstacle that is closer to the virtual target. Then based on the boundary point data generated, we use the method mentioned above to calculate the maximum turning radius. Lastly, the minimum turning radius that meets the safe obstacle avoidance distance is selected.b. If there is an emergency obstacle avoidance and the bounded obstacle is in the field of view of the sonar, in order to perform a safe and most energy-efficient obstacle avoidance, the AUV sails toward the bounded side while calculating the turning radius.The obstacle avoidance rule flow chart is shown in According to the principle of the predictive control model , the pro• The Weight Function of Safe DistanceTo improve the safety of AUV obstacle avoidance, the relationship between the reference obstacle point of the AUV selection and the current AUV distance as well as the safety distance must be considered. The relationships are shown below.Clearly, the weight function can reduce the impact of this problem on the obstacle avoidance algorithm. The risk of obstacle avoidance failure due to a single collision avoidance condition can be avoided.• Weight Function for the Rate of Change of the AUV Yaw AngleTo prevent the shaking movement of the yaw angle during the control process, the influence of the rate of change of the AUV yaw angle is introduced to achieve the smooth control of the AUV turning heading. The weight function is defined below.• Weight Function of the Turning RadiusThe predictive track points are used for guidance control to achieve local obstacle avoidance. According to the method of calculating the turning radius proposed above, the maximum turning radius cannot be directly used as the guidance parameter, because the turning radius is only one factor that affects the obstacle avoidance performance. The specific weight function is shown below.• Weight Function of the Obstacle Avoidance PathTo ensure that the AUV is able to reach the desired virtual target point by the shortest travel path within the predictable range, the weight function of the obstacle avoidance path can be defined by the equation below.• Weight Function of the Obstacle Avoidance AreaTo improve the reliability of obstacle avoidance, the fan-shaped field of view detected by FLS is generally divided into three parts: non-avoidance areas, safe obstacle avoidance areas, and emergency obstacle avoidance areas. Therefore, it is necessary to judge which area the obstacle detected belongs to, and adopt different obstacle avoidance strategies. Therefore, the weight function of the obstacle avoidance area is defined below.Due to the impact of physical structure characteristics of the AUV itself, and the obstacle avoidance environment, several constraints must be met to achieve the purpose of collision avoidance successfully.a. According to the structural characteristics of the physical design of the AUV, Maximum turning angular velocity meets: b. To ensure the safety of the AUV, the safety distance is set to meet the conditions: c. The maximum turning radius meets the conditions: Combining weight functions, model parameters, and the obstacle avoidance principle, the secondary optimization function based on predictive control is obtained below.Since Equation (27) introduces non-contiguous Boolean variables as the weight function, the objective function (28) is made non-convex. The corresponding problem becomes the mixed integer nonlinear programming (MINLP) problem, but, so far, this type of problem has not found a unanimous and mature solution . HoweverThe weight function before and after relaxation is shown in The predictive guidance control obstacle avoidance algorithm is a real-time collision avoidance algorithm that ensures the safety of the AUV when the AUV performs tasks. The details are as follows:(1) Always moving toward the virtual target point, which is the direction of the AUV’s minimum cost. When no obstacles or many obstacles detected by the FLS configured on the AUV are outside the obstacle avoidance distance, the AUV needs to move in the direction of the target point where the greatest profit is obtained.(2) Maintaining a safe distance: When the obstacle detected by the FLS meets the obstacle avoidance distance, the safety distance must be considered in the AUV collision avoidance. The length of the AUV is 5 m, and the safety distance is generally four times longer than the AUV’s length. When selecting the obstacle avoidance guidance point, it is necessary to judge whether the safety distance meets the safe distance of the obstacle avoidance.(3) Simple convex obstacles: When the AUV keeps the current heading in a certain area, and a single convex obstacle is detected by the FLS, the appropriate track points are chosen, according to the obstacle avoidance rule. This uses the dynamic guidance algorithm to adjust the heading in real time, and, at the same time, control the appropriate speed to achieve safe obstacle avoidance.(4) Vortex obstacles: In the current heading path of the AUV, there may be a complex vortex obstacle. Since the obstacle appears only partly in the detectable area of the sonar and it is too hard to identify, the AUV may enter inside the concave obstacle. Therefore, it is necessary to establish obstacle avoidance rules for concave obstacles, and, based on the obstacle avoidance rules mentioned above, the effective real-time algorithm can be achieved between the two obstacle avoidance guidance algorithms along the vortex obstacle wall for the AUV to drive safely and, lastly, escape the vortex obstacles.(5) Dense convex obstacles: When more than two obstacles were detected by the forward-looking sonar of the AUV, the above two methods of obstacle avoidance are not fully competent. For the AUV to sail safely in the multiple convex obstacle environments, we combed the predictive guidance obstacle avoidance algorithm and obstacle avoidance rules based on different classifications of multiple obstacle environments. The specific algorithm for such a setting is described in To ensure the AUV complete its task safely, it is necessary to design reasonable obstacle avoidance algorithms for different types of obstacle environments so that the AUV can adapt to the harsh obstacle environments when sailing in an unknown underwater environment with obstacles. Therefore, by using the predictive guidance control obstacle avoidance algorithm, we can obtain the shortest and smoothest obstacle avoidance path, so that the AUV can successfully achieve collision avoidance with less rotation and rudder angle correction. Obstacle avoidance track points are predicted based on the obstacles detected by sonar. Then, the AUV follows the obstacle avoidance track points by linear guidance and arc guidance (the radius is the turning radius) to achieve a safe obstacle avoidance path.The AUV uses the obstacle avoidance rule mentioned above, and the obstacle avoidance weight function is used to calculate the appropriate trajectory point to get the best track point and bypass the obstacle. Taking the right bounded obstacle in 1. By processing the boundary data of obstacles that has been detected by FLS, the number of obstacles and their relative positions to the AUV can be obtained.2. The relation between the obstacle points detected by sonar beam and the current AUV is calculated by using the optimal parameters obtained by the weight function and the coordinate information of all points are calculated by using the method mentioned above.3. Judging whether the current point is on the left or right side of the sonar center line is demarcated by the AUV heading. If it is on the left side and the result of the classification using step 1 is applied, for example, 4. If If 5. If If If If 6. Based on the obstacle avoidance track point obtained above, we can perform the following obstacle avoidance guidance algorithm to correct the position and heading of the AUV to sail a safe obstacle avoidance path.7. Based on the obstacle avoidance track points obtained above, we perform the following obstacle avoidance guidance algorithm to correct the position and heading of the AUV to form a safe obstacle avoidance path.8. End.To solve the trap problem caused by the vortex obstacles for the AUV in the actual underwater environment and improve the collision avoidance efficiency of the complex vortex obstacles, the line-of-sight guidance mechanism for predicting the update continuously of the trajectory segment is used, so that the trap problem caused by the obstacle can be overcome. In The predictive guidance trajectory segment calculation is performed with the partial vortex obstacle information detected by the sonar and the obstacle avoidance guidance trajectory segment obtained in the sonar’s field of view in The orange guidance trajectory segment in Regarding the termination condition of the vortex obstacle collision avoidance, a virtual target point is designed in this case, and the AUV moves toward the direction combing target’s direction , and the heading angle is set to 0° with the two-dimensional An obstacle avoidance algorithm is set for the vortex obstacle. This design requires AUV to have a dense convex obstacle environment with an obstacle avoidance capability to arrive at virtual target points and to verify the effectiveness and efficiency of the obstacle avoidance algorithm in dealing with a dense convex obstacle environment. As shown in The PGOA proposed in this paper deals with various complex obstacle environments, which includes complex convex polygon obstacles and complex concave obstacles. This algorithm can handle simple convex obstacles, dense convex obstacles, and vortex obstacles. The AUV’s obstacle avoidance trajectory is also close to smooth. The AUV through various obstacle environments easily reaches the predetermined virtual target point during the whole obstacle avoidance process. Furthermore, the AUV’s obstacle avoidance process has always met the requirements of safe obstacle avoidance distance, and the cost of the entire obstacle avoidance process is less than other traditional algorithms. During the process of facing the vortex obstacles, the AUV can escape the vortex obstacle environment once inside the vortex obstacles by remembering the direction of the path that it has traveled before, which avoids falling into the obstacles and causing an obstacle avoidance process failure. The next step in this paper is to optimize the proposed algorithm to adapt to the AUV pool experiment and extend the algorithm to the complex marine environment where ocean currents and dynamic obstacles exist. At the same time in the actual underwater environment, most obstacles have different shapes at different heights. The obstacle avoidance problem is much more complicated in the three-dimensional environment. Therefore, in the later research, the two-dimensional obstacle avoidance algorithm is used to penetrate into the three-dimensional environment.


Due to payload restrictions for micro aerial vehicles (MAVs), vision-based approaches have been widely studied with their light weight characteristics and cost effectiveness. In particular, optical flow-based obstacle avoidance has proven to be one of the most efficient methods in terms of obstacle avoidance capabilities and computational load; however, existing approaches do not consider 3-D complex environments. In addition, most approaches are unable to deal with situations where there are wall-like frontal obstacles. Although some algorithms consider wall-like frontal obstacles, they cause a jitter or unnecessary motion. To address these limitations, this paper proposes a vision-based obstacle avoidance algorithm for MAVs using the optical flow in 3-D textured environments. The image obtained from a monocular camera is first split into two horizontal and vertical half planes. The desired heading direction and climb rate are then determined by comparing the sum of optical flows between half planes horizontally and vertically, respectively, for obstacle avoidance in 3-D environments. Besides, the proposed approach is capable of avoiding wall-like frontal obstacles by considering the divergence of the optical flow at the focus of expansion and navigating to the goal position using a sigmoid weighting function. The performance of the proposed algorithm was validated through numerical simulations and indoor flight experiments in various situations. To navigate through unknown environments, micro aerial vehicles (MAVs) need to detect and avoid obstacles. There are many types of sensors to detect obstacles such as RADAR (RAdio Detection And Ranging), LiDAR (LIght Detection And Ranging), and ultrasonic sensor. RADAR and LiDAR have a good performance in terms of the operation range and accuracy to detect obstacles around the MAV. However, they have disadvantages in terms of operating time due to heavy weight and energy consumption. Although lightweight and low power consumption products have been developed recently , they arThere are various obstacle avoidance methods using the vision system. Tomoyuki et al.  and AbduIn this paper, we propose a vision-based obstacle avoidance algorithm based on the horizontal balance strategy using the optical flow. Along with horizontal avoidance, it is designed to change the height to avoid the obstacle vertically by balancing the optical flow generated in the upper and lower half plane of the image. Besides, if a wall-like obstacle is in front of the MAV directly, the MAV turns the heading angle according to the expansion of the optical flow. Previously, the MAV was made to turn inversely proportional to the optical flow difference  or take The contribution of this paper is threefold: (i) the MAV avoids the obstacle by changing the height as well as heading angle in 3-D environments; (ii) the MAV can avoid the frontal obstacle which generates a similar optical flow in left and right half planes such as a wall by using the expansion of the optical flow; and (iii) the MAV combines the different guidance strategies with adequate weights considering the environment around the MAV.This rest of this paper is organized as follows. In The optical flow refers to the movement of each pixel expressed in the image plane when an object moving in the three-dimensional space is projected onto the two-dimensional image plane. There are various methods to estimate the optical flow such as ,16,17. RIn the Horn–Schunck method, two assumptions are used to estimate the optical flow. First, as in I is the intensity of the pixel at the point t, respectively. By defining the optical flow vector (velocity vector of the pixels) as u and v are dI  5) by(6From Equations  and (8),This section introduces the obstacle avoidance strategies using the optical flow estimated from a forward looking monocular camera. The horizontal balance strategy vertical balance strategy to avoid the obstacle vertically, and the frontal obstacle avoidance strategy which exploits the expansion of the optical flow are introduced. In addition to obstacle avoidance strategies, the waypoint guidance strategy is described to reach to the goal point for the MAV. Lastly, the above strategies are integrated with the weighted sum of guidance commands from different strategies according to obstacle environment conditions.This subsection deals with the horizontal balance strategy. We modified conventional balance strategies ,20,21 usP on the obstacle of interest corresponding to the point H on the principal axis of the camera to the point P. The vector O. Note that the focal length f and As shown in x are functions of time t and f and It is worth noting that Equation  with resBy rewriting Equation , the optained as:Mr=fRVxx2Let the image be equivalently split into two horizontal (left and right) and vertical (upper and lower) half planes. Considering the fact that the larger the magnitude of the optical flow is, the closer the obstacle is, the desired heading direction of the MAV can be determined by finding the half plane which has the smaller sum of optical flows. More specifically, the optical flow is calculated within the horizontal calculation window shown as the red area in To avoid the obstacle in the three-dimensional environment, the MAV has to change the altitude as well as heading angle. In this subsection, the obstacle avoidance strategy by changing the altitude of the MAV is proposed. The optical flow for the vertical balance strategy is calculated within the vertical calculation window shown as the red area in The desired climb rate command  is introduced in . When thx. When x is small, the time-to-contact becomes small. Using this relation, the time-to-contact Using the geometry given in N is the number of pixels in the horizontal optical flow calculation window. Note that the high EOF implies that there is a high risk of the collision with the frontal obstacle. The magnitude of the desired heading rate to avoid the frontal obstacle is determined by the PD controller using the EOF as:In addition to the obstacle avoidance, the MAV may need to reach the waypoint to accomplish the given mission. The heading rate and climb rate for waypoint guidance can be determined as:The error for the climb rate is the difference between the height of the waypoint and the MAV:The hybrid obstacle avoidance strategy is designed to avoid various obstacles in 3-D environment while moving towards the goal according to existence of obstacles determined by the EOF The hybrid obstacle avoidance algorithm including waypoint guidance can then be designed as: Note that, unlike desired heading rate, the desired climb rate can be designed by just adding the vertical balance strategy and waypoint guidance without sigmoid weights. The sigmoid weights should not be multiplied to the vertical balance strategy considering the fact that the sigmoid weights are to change the heading angle of the MAV if there is a wall-like obstacle in front of the MAV.The RotorS simulator  was explT, the roll angle command p, velocity v, orientation q in the quaternion form, and angular velocity w. Using the obstacle avoidance algorithm, the desired position x, y, and z direction, respectively. In the case of the L-shape and T-shape maps, the collision of the MAV was determined if an obstacle was within a certain distance from the center of the MAV; the collision boundary was set to 1.5 times of the half of the y direction size of the MAV. In the case of the ramp-shape map, the MAV was considered to have a collision if the minimum distance was shorter than 1.5 times of the half of z direction size of the MAV.Numerical simulations were carried out to verify the performance of the proposed obstacle avoidance strategy. https://drive.google.com/file/d/1i9Cx2NoRTqqSM99g8u1YIcXqaS8sRJ8k/view.The hybrid obstacle avoidance strategy was also verified in the complex environment surrounded by walls and containing horizontal columns, as described in The obstacle avoidance strategy used in this research assumes that the MAV moves forward without the significant yaw motion. However, if the MAV meets an obstacle, it will turn to avoid the obstacle. If the yaw rate from turning is too large, then the assumption is no longer valid. In other words, a large yaw rate could generate the unintended optical flow and EOF. Especially, when the MAV encounters a frontal obstacle, if it rotates at a large yaw rate, it would make the EOF larger than expected. To verify the effect of yaw rates on the EOF, an illustrative simulation was conducted. To compensate the effect of the yaw rate on the EOF, the yaw rate compensator was added to the frontal obstacle avoidance strategy (Equation ) as:(32)The hardware for experiments was configured with the customized MAV, as shown in To verify obstacle avoidance strategies in a real world, indoor flight experiments were conducted. The waypoint guidance strategy was applied for maintaining the reference height when there was no vertical obstacle around. This paper proposes the vision-based obstacle avoidance strategy using the optical flow which can be used in various 3-D textured environments. In particular, it exploits the EOF to cope with the front obstacle efficiently along with avoiding obstacles horizontally and vertically. To verify the performance of the proposed approach, numerical simulations and indoor flight experiments were carried out. The proposed obstacle avoidance strategy using optical flows requires a light computational power; hence, it could be readily applicable to miniaturized MAVs requiring the lightweight CPU such as insect-inspired flapping aerial vehicles. It is worth noting that the use of obstacle avoidance based on optical flows is limited to only relatively well-textured environments since optical flows are rarely generated on textureless objects such as white walls, wires and poles.As the forward speed of the MAV is assumed to be kept relatively slow in our approach, the effect of pitch control on the optical flow computation was negligible. However, for high speed maneuvers, the effect of pitch control on the optical flow would be significant; this will be dealt with in the future work. Besides, although this study proposed the yaw rate effect compensator in the situation where the MAV encounters the frontal obstacle, more rigorous analysis and experiments need to be performed. For general environments, other vision-based algorithms using feature or color detection could be combined with the proposed optical flow-based approach, which remains as future work.


Localized Surface Plasmon Resonance (LSPR) sensors have potential applications in essential and important areas such as bio-sensor technology, especially in medical applications and gas sensors in environmental monitoring applications. Figure of Merit (FOM) and Sensitivity (S) measurements are two ways to assess the performance of an LSPR sensor. However, LSPR sensors suffer low FOM compared to the conventional Surface Plasmon Resonance (SPR) sensor due to high losses resulting from radiative damping of LSPs waves. Different methodologies have been utilized to enhance the performance of LSPR sensors, including various geometrical and material parameters, plasmonic wave coupling from different structures, and integration of noble metals with graphene, which is the focus of this report. Recent studies of metal-graphene hybrid plasmonic systems have shown its capability of promoting the performance of the LSPR sensor to a level that enhances its chance for commercialization. In this review, fundamental physics, the operation principle, and performance assessment of the LSPR sensor are presented followed by a discussion of plasmonic materials and a summary of methods used to optimize the sensor’s performance. A focused review on metal-graphene hybrid nanostructure and a discussion of its role in promoting the performance of the LSPR sensor follow. The Surface Plasmonic Resonance (SPR) sensor is a common and commercialized sensor used in different areas such healthcare and gas sensing ,2. When In addition to the low cost of fabrication, the LSPR sensor offers a simple sensor setup and detection of local change in the refractive index, which is not the case with SPR sensors ,7. HowevN, λ, The Mie scattering theory, with the help of Maxwell’s equations, is a very useful way to explain how metal NPs interact with incident light. Equation (1) shows the derived optical extinction for metallic NP with a radius smaller than the wavelength of the incident wave (2r << λ) :(1)σextωLSP=Mie resonance is a good example of the nonlinear optical effect where the optical response of the medium is nonlinearly depending on the electric field applied . In addiNanostructure supported LSPR extinction can be used in refractometric sensing, like a conventional SPR sensor. The detection principle in LSPR sensors is based on an intensity or wavelength shift upon variation in the permittivity of a sensing region (proximity region of the NP). In the transmission or reflection spectrum, the resonance peak position varies based on the refractive index of the sensing area, as is confirmed by Equation (3). Any change of permittivity in the proximity of the NP will shift the resonance LSP peak, which can be employed to detect molecular interaction between the NP and adsorbed material. The amount of adsorbed material can be found through monitoring the plasmonic resonance shift. The performance of the sensor can be assessed through Sensitivity (S) and Figure of Merit (FOM) measurements. Sensitivity (S) is defined as the extent of peak shift in position or intensity to the change of the refractive index of a dielectric medium. Therefore, a larger resonance shift at a small refractive index change results in increased bulk sensitivity . It is kMetal is a candidate material for a plasmonic device. It has an enormous number of free electrons, and upon excitation with electromagnetic wave, those free electrons produce collective oscillations, which is the plasmon phenomena used in optical devices. Any plasmonic material should have the ability to provide a negative real permittivity to be used in a plasmonic device. Free electrons in metal provide this property, and metal therefore is mostly used as a plasmonic material . HoweverGraphene, a single layer of carbon, has been reported as a new emerging competitive plasmonic material to metal. In the NIR-IR region, graphene has low losses and high confinement of surface plasmons and, thus, has a good advantage over metals . FurtherThe performance of plasmonic sensors is directly related to the amount of energy loss in the metallic structure of a sensor. For example, using a metallic structure with a high loss results in broadening of the resonance plasmonic waveband, and hence, the FOM of a sensor is decreased. It is known that silver has a lower damping rate and thus has a lower absorption loss than gold and copper . Gold anThe shape of the NP is also used as a tunable parameter to improve LSPR sensor performance. Changing the shape of NP causes resonance peak shift . Tuning By using two or more NPs separated by a specific distance in one-, two-, or three-dimensions, the interaction between these incident waves and the excited surface plasmon wave from NPs resulted in an additional shift in the resonance wavelength to the red region as a result of the coupling interaction, which enhances sensor sensitivity . Erik etPlasmon mode hybridization is another promising method that has been applied to promote the performance of LSPR sensors. Nanorings and nanoshells are the best nanostructures that represents the plasmon hybridization effects . In gene2S forms on the Ag surface due to the reaction of carbonyl sulfide (OCS) and hydrogen sulfide (H2S) with the Ag surface; this affects the plasmonic properties of the Ag nanostructure and results in an increase in material loss [Longer wavelength resonance, high resonance intensity or amplitude, and sharper resonance band (min FWHM) are indications of enhanced sensitivity and FOM of the LSPR sensor. Therefore, engineering a plasmonic material by choosing a proper type, tuning the geometry of the NP, or hybridizing it with another material in a way that improves one of the mentioned three plasmonic properties could lead to promoting plasmonic sensor performance. Starting with the type of plasmonic material, silver is the best choice among other plasmonic materials in terms of sensor performance due to low optical loss compared to other metals in the visible range. This is due to the lack of the interband transition as in gold and copper . Howeverial loss  Furthermial loss . Gold foreported . Howeverreported ,60. TherDecorating graphene film with metal NPs is another approach used to tune the plasmonic properties of an LSPR device. Xu et al. fabricated Ag NPs on top of a CVD graphene film using a thermally-assisted self-assembly method . They obAnother metal-graphene hybrid system that studied the effect of the hybrid nanostructure on the plasmonic resonance of the nanostructure is that reported by Nan et al. . They stA plasmonic material that has high absorption in the visible region, such as metal, is preferred in plasmonic applications such as an LSPR sensor. Graphene has very low absorption in the visible spectrum, and thus, enhancing its absorption will increase its chance to be used as a plasmonic material for an LSPR sensor in the visible region . The hyb3+ precursor and graphene [3+ precursor, the surface energy of graphene on the substrate, the spin-coating speed, and the coating cycles. However, they found that Au NP size is strongly affected by the concentration of Au3+ precursor and the coating cycle’s number. Upon increasing the concentration of Au3+ and repeating the cycles of the reduction process, the plasmonic resonance of the Au-graphene hybrid nanostructure was redshifted, and broadening in the band FWHM was observed. After repeating the reduction process, the Au NP size was distributed between 20 and 120 nm, and the resonance redshifted from 560–620 nm. This redshift in resonance wavelength is a tuning parameter to enhance LSPR sensor performance; however, as mentioned in the previous section, the FWHM of the resonance band has to be minimized to obtain a superior FOM of the sensor.Increasing the metal NP size results in red shifting in plasmonic resonance; for this reason, having a fabrication method that can control the size of the metal NPs on top of graphene film is important. Lee et al. introduced a tunable size and LSP resonance of Au NPs on top of a mono-layer of graphene film based on the reduction potential between the Augraphene . They moRecently, some published works showed the direct effect of hybrid graphene with metal on LSPR sensor performance. Maurer et al. studied the effect of the graphene layer as a spacer between Au film and Au NPs. They deposited graphene film on top of a gold film and then decorated the layer with an Au nanodisc array . They coIn this short review, the fundamental physics of the LSPR sensor was summarized followed by a discussion of different methods used to enhance the performance of LSPR sensors. It was shown that the size and shape of the nanoparticle give a broad tunability in optimizing the performance (sensitivity and FOM) of the LSPR sensor. A smaller NP size (<100 nm) is preferred for better plasmonic properties, as reported in ; however