Recent decades have witnessed the growing importance of human motion detection systems based on artificial intelligence (AI). The growing interest in human motion detection systems is the advantages of automation in the monitoring of patients remotely and giving warnings to doctors promptly. Currently, wearable devices are frequently used for human motion detection systems. However, such devices have several limitations, such as the elderly not wearing devices due to lack of comfort or forgetfulness and/or battery limitations. To overcome the problems of wearable devices, we propose an AI-driven human motion detection system (deep learning-based system) using channel state information (CSI) extracted from Radio Frequency (RF) signals. The main contribution of this paper is to improve the performance of the deep learning models through techniques, including structure modification and dimension reduction of the original data. In this work, We firstly collected the CSI data with the center frequency 5.32 GHz and implemented the structure of the basic deep learning network in our previous work. After that, we changed the basic deep learning network by increasing the depth, increasing the width, adapting some advanced network structures, and reducing dimensions. After finishing those modifications, we observed the results and analyzed how to further improve the deep learning performance of this contactless AI-enabled human motion detection system. It can be found that reducing the dimension of the original data can work better than modifying the structure of the deep learning model. The World Health Organisation (WHO) has reported that 37 million injuries are caused by falling and elderly people are at high risk of falling2. Numerous tragedies could have been avoided by the caregiver providing timely assistance if healthcare monitoring systems knew the patient was in danger and gave warnings in time, allowing the elderly and vulnerable people to lead more independent lives while keeping them safe through monitoring.The development of the Internet of Things (IoT) has enabled the application of human motion detection in many scenarios, especially in healthcare3 for example. However, it doesn’t work if people forget to wear or charge their devices. Using Radio Frequency (RF) signals does not require users to wear or charge devices. From the work described in Ref.4, we know that Channel State Information (CSI) can be used to detect human movements. CSI describes how a wireless signal propagates between an RF source and an RF receiver, and when a person performs certain actions, it impacts the CSI in a way that unique pattern changes are produced against each action. These patterns are the foundation for usage in deep learning .Human motion detection by wearable devices with an accelerometer is commonly used to build human motion health systems. A smartwatch can detect the fall of a patient and relay the information to a caregiver for help5. Since deep learning model can learn representations of data with multiple levels of abstraction6. Increasing the depth can help the deep learning model learn more levels of data abstraction while increasing the width can extract more dimensions of data at each level of data abstraction. Convolutional Neural Network (CNN) has been proven to be easier to train and generalized better than the traditional fully connected layers8. Long Short-Term Memory Networks (LSTM) have proved to behave better than conventional Recurrent Neural Networks (RNNs) and LSTM can have great performance in learning time-series data. ResNet can transform the computation of gradients in backpropagation from multiplicative to additive, which can solve the problem of vanishing gradients. This feature can help the neural network grow much deeper. Principal Component Analysis (PCA) can effectively decrease the sparsity of the data, ensuring that the deep learning model can learn the most important features of data easier. The result of PCA modification can achieve an accuracy of 99%, higher than other works about CSI-based human recognition12 to the best of my knowledge. This result indicates the fact that preprocessing data might be more crucial than adjusting the structure of the deep learning model if we want to further improve the performance of the deep learning.We adapt CSI data into different deep neural networks to find useful techniques to improve the performances. Firstly, we reproduce the deep neural network in Ref.13, based on the CSI dataset, the authors built a deep learning model, CSI-Net, which can be trained to handle falling detection. The CSI-Net extracts feature of CSI data by the combination of a deconvolutional layer and a convolutional layer. Although the CSI-Net can extract local features, it cannot extract the principal features that can help the deep learning model detect falls better.Deep learning has dominated machine learning for several years and it can achieve wonderful performance in human motion detection systems. In Ref.14 designed a deep neural network named WiSPPN, using CSI data for human pose estimation. The WiSPPN works greatly in estimating the human poses through CSI data, but it lacks discussion about how the depth, width, and the structures can affect the performance of the deep learning model in estimating the human poses.The authors in Ref.16, utilized mmWave data to reconstruct human poses. They have achieved great success in reconstructing the human pose through pose estimation. However, the pose estimation from mmWave suffers from the sparse representation, inter-person occlusion17, and the decay of mmWave signals. Besides, compared with the human pose estimation by video frames19, mmWave human pose estimation has a lack of data. The work in Ref.20 shows the potential of combining mmWave, red, green and blue (RGB)-D, and Inertial Sensors together for human pose estimation or human activity recognition. However, the applicability of such a multimodal system to real-life scenarios still needs to be tested.Similarly, Refs.9, both the RNN-based deep neural network and the CNN-based deep neural network have achieved a great performance in the RGB-D  based Human Motion Recognition. But the author does not mention the influence of data processing and modification of the network structure.In Ref.10, the deep learning model is applied to help the robot to understand human motion. In this work, the authors adapted the structure from the AlexNet21 without exploring the effect of structure modifications and data preprocessing. In Ref.11, after training the CNN-based network with image-preprocessed radar data, the CNN-based network can outperform the SVM-based approach. Another successful application of deep learning-based human motion detection in the data collected by radar can be found in Ref.12, deep learning-based human motion detection can detect walking, falling, sitting, and bending with higher accuracy than the PCA-based scheme. In Ref.22, the deep learning model can be applied to high-resolution range information to classify 7 different motions. The deep learning method can gain the highest average classification accuracy in contrast with the PCA-based method and SVM-based method. References22 have proved that the deep learning method can outperform traditional machine learning methods in radar-based human recognition, but they did not experiment with how to further make a progress in the deep learning model.In Ref.This work focuses on discussing how the model modifications can impact the performance of the deep learning model and how these modifications can be utilized improve the performance in terms of classification accuracy.This work considers two aspects of model modification, one is the structure modification, and the other is reducing dimensions.After substantial amount of experiments, it has been found that applying LSTM structures and extracting Principle Component Analysis (PCA) features can boost the performance of CSI-based human recognition to a promising accuracy of 99.1%.Overall, it is clear that deep learning can have wonderful performance in many datasets that contain human motion patterns, like CSI data, RGB-D data, and radar data. Besides, deep learning can gain better performance than the previous method, consisting of feature extraction and the traditional machine learning model training. However, these works only implemented the deep learning framework without analysing the effect of deep learning model structure modifications like increasing the width, increasing the depth, and adapting advanced structures in accuracy. They also did not analyze the effect of reducing dimensions on the RF data can have on the performance of the deep learning model. The main contributions of this work are as follows.The experimental part composes of two parts. One part is data collection and signal processing and the other part is reconstructing the basic deep learning network.In this work, we collected samples from sitting and standing at the centre frequency 5.32 GHz. CSI describes how the wireless signals propagate between the RF transmitter and RF receiver and it will perform changes in certain patterns when people perform certain movements, implying the potential of human motion detection. We record the amplitude of CSI using universal software-defined radio peripheral (USRP) when volunteers performed certain human movements (sitting or standing) to construct a dataset of CSI of human activities. USRPs were used because they provide a simple framework that can enable us to collect data easily. Besides, USRPs can transfer and receive frequencies in different bands.The project uses two USRP devices. One USRP acts as the transmitter and the other acts as the receiver. The device is cabled to two computers and configured to send a signal from one antenna to the other for 10 s. In the experiments to collect CSI data, we kept two USRP devices at a distance of 4 m. Experiments were conducted in an office setting, in the presence of desks, chairs, paintings, etc., simulating real-world situations. Volunteers were asked to complete a sitting and standing action between the two USRPs. When a volunteer acts, the CSI pattern of this action is collected. Specific parameters of the hardware used in this experiment are listed in Table People do not generally perform sitting and standing activities identically as there will be some variations paired with interference from surrounding factors affecting the CSI. However, the resulting CSI patterns are somewhat similar and correlated to each other. Multiple patterns are recorded from the CSI, which are learned through the deep learning models. To collect data, we asked 10 volunteers to participate. All volunteers signed an ethical approval provided by the Institutional Review Board of the University of Glasgow before the experiment. This process facilitates the collection of CSI for various human movements and creates tags on the CSI samples during collection. After collecting enough sitting and standing samples, the CSI database was successfully constructed. Figure We initially worked on signal processing to explore the capabilities of CSI and the possibility of using filtering to limit the effects of noise. Specifically, we first analyze the CSI by comparing the sitting and standing CSI in the time and frequency domains using MATLAB programming. Since the acquired data is measured in the time domain, it is displayed directly in the MATLAB graph, while in the frequency domain, the frequency information is displayed using a discrete Fourier transform (DFT). The results of the comparison are shown below in Fig. It is clear from Fig. 5, the author constructed a 4-layer multilayer perceptron and each hidden layer has 20 nodes. The structure of this multilayer perceptron can be found in Supplementary Table In our previous work23 optimizer. The ReLU activation function is used for the hidden layers while the Sigmoid function is applied for the output layer because this task is a binary classifying test.We adopted PyTorch to build the basic neural network. We adopted the Adam24 with random seeds, and we take an average of 10 times training and testing in each deep learning model to reduce the bias of our results. The mean of 10 times training and testing are reported in Tables 700 data samples are included in the dataset, 350 data for sitting and 350 data for standing. The train-test validation setting of the dataset is done by randomly splitting 30% of the data as the testing set and 70% of the data as the training set. The train-test split is done by the python library sklearnIn an attempt to assess the performances of those machine learning algorithms, we calculated four classification values, including False Positive (FP), True Positive (TP), False Negative (FN), and True Negative (TN). Then we got the performance metrics based on the above four classification values. The details of calculating performance metrics can be found in equations.After training and testing 10 times, we get the trained model, which could gain an accuracy of 93.8%, a precision of 93.7%, recall of 95.7%, and an f1-score of 94.7% on average. Compared with the previous work, which got 94.1% in accuracy, 94.3% in precision, 94.1% in recall, and 94.2%, we find that the result of the training is close to the original work. This comparison proved that our reproduction was successful. The loss of accuracy of the basic network is presented in Fig. The testing accuracy oscillates between 87% and 97% , according to the accuracy curve The trained basic network is more accurate at identifying sitting behavior than standing, with a 97% correct prediction rate for standing and a 91% correct prediction rate for sitting, according to the normalized confusion matrix of the basic neural network shown in Supplementary Fig. The necessity of increasing the depth originates from the assertion that a neural network can mimic any function. Such assertion is mathematically proved to hold for any function. However, it does not guarantee a computation cost of such approximation. The number of neurons sometimes increases exponentially as the function frequency increases. Thus, the size (i.e. the number of hyper-parameters) explodes quickly. The number of parameters is an important metric in that a large number of parameters usually means a few things. First, it is a lot more costly to train such models computationally. Second, it is more likely to cause under-fitting problems.The introduction of multi-layered neural networks solves such issues. Properly built multi-layered neural networks should be able to reduce the space complexity of neural networks from exponential to linear. Hence, the problem of parameter exploding is then solved. However, increasing the depth of neural networks introduces other problems too. In the back-propagation phase of training. The gradient accumulates to usually a smaller and smaller number as it propagates backwards because a reasonable learning rate is usually too small for the model to be able to achieve a close approximation towards the global optimally. When approaching the first few layers, gradients tend to be too small to be handled by the floating-point representation. To make this problem even worse, most optimizers have strategies that reduce the magnitude of learning rates as the training progresses. This is because the closer the current state is to the optimal point, the smaller the step length is the better to approach the optimally.We experimented with changes in the depth of our neural networks and how these changes will influence the result of training. A deep neural network with 12 hidden layers is tested. To test the influence of the depth, all hidden layers share the same number of hidden nodes with the basic neural network, 20 nodes. More details of the deep neural network structure can be found in Supplementary Table An accuracy of 68.1%, a precision of 68.5%, a recall of 72.2%, and an f1-score of 70.3% are obtained by the neural network with a depth of 12, taking average after 10 times training and testing. In comparison to the basic network, there is a significant decrease in all four evaluating metrics.When we expand the neural network’s depth to 12, we can see from Fig. This phenomenon of fluctuation in the accuracy and loss curves is expected due to the oscillation nature of deep neural networks.It is obvious from the deep neural network’s normalized confusion matrix shown in Supplementary Fig. In neural networks, computation is frequently structured into layers of artificial neurons. The layer width refers to the number of neurons in one single layer of the network. There are several reasons why we need a wide neural network.25. If a network is too narrow, each layer can capture a limited dimension of features, there is no possibility that we can extract enough information to propagate down the network, no matter how deep it is. Another reason is the usage of GPU. The GPU is considerably more efficient in parallel calculations on huge tensors, it is more computationally economical26 to expand the layers rather than have hundreds of little kernels. Wide residual networks, for example, can compute multiple multiplications in parallel, but deeper residual networks require more sequential calculations (since the computation depends on the previous layer).Firstly, the wide neural network allows each layer to extract a collection of features with more dimensionsAs a consequence, we may start with the width to enhance network performance, then increase channel utilization rates at each layer, supplement the thin layer with information from other channels, identify the lowest limit of width, and achieve better performance with as little computing as feasible.To test the effect of width in the neural network, we keep the number of hidden layers as same as that of the basic neural network. At the same time, we increase the number of nodes in each hidden layer from 20 to 320. The parameters of the wide neural network are listed in Supplementary Table While expanding the number of hidden layer nodes to 320 with the same number of hidden layers as the basic network, the accuracy, precision, recall, and f1-score are 69.0%, 69.5%, 74.4%, and 71.8% on average after 10 times training and testing, respectively. Both the accuracy in the testing set and the loss curve in Fig. From the normalized confusion matrix of the wide neural network shown in Supplementary Fig. The reason for the bad performance of the wide neural network can be concluded as overfitting: the model attempts to learn too many details from the training data while simultaneously accounting for noise. Overfitting occurs when a model becomes very strong at categorizing or predicting data from the training set but not so good at categorizing data it wasn’t trained on. Adding extra nodes causes overfitting since the original network already fits well with the data.When we produce a model on training data and use it on the test dataset, the error reduces to a very tiny value, but the error generated from fresh data climbs to a big value, this is an indication of overfitting during training. Because they learn millions or billions of parameters while developing the model, deep neural networks are prone to overfitting. Because it has the potential to do so, a model with this many parameters can overfit the training data. The primary notion behind addressing the problem of overfitting is to reduce the model’s complexity. We may accomplish this by simply eliminating layers or lowering the number of neurons, for example.The following are some probable causes of overfitting in neural networks: Firstly, the training dataset is rather small. When a network tries to learn from a limited dataset, it tends to have more control over the dataset and ensures that all data points are satisfied exactly. As a result, the network is attempting to memorize every data point while failing to grasp the overall pattern from the training dataset. Secondly, on noisy data, the model tries to make predictions. Overfitting may also happen when a model tries to generate predictions on excessively noisy data, which is produced by an overly complicated model with too many parameters. As a result, the overfitted model is erroneous since the trend does not reflect the reality of the data.27. Recurrent Neural Networks often referred to as RNN, are known for their ability to handle time-series data28. The success of ResNet is a milestone of deep learning that makes the extremely deep neural network possible29. Those three new architectures are implemented while keeping the size the same as the neural networks that we experimented with before for a better comparison.Recent decades have witnessed great progress in deep learning, new structures of Neural Networks are developed and have been implemented in the field of deep learning with a great outcome. Convolutional Neural Networks, usually named CNN, down-sample inputs to extract features through convolvingCNN is usually used for image or video recognition where 2D convolutional networks or 3D convolutional networks are implemented. However, since our dataset is only 1D data, we cannot adapt 2D or 3D convolutional networks as people usually do. Instead, we implement a 1D convolutional network and reduce the number of features from 999 to 20  after the 1D convolutional network. The 1D convolutional network, or feature-extracting network, consists of 2 layers of convolutional networks. Each convolutional neural network in the feature-extracting network is followed by a ReLU function. The rest of the network is the classification network that has four linear hidden layers and one output linear layer with a Sigmoid activation function. Each linear layer has 20 hidden nodes to ensure that the size is as same as the basic neural network. The details of the designed 1D CNN can be found in Supplementary Table The results after training are shown in Supplementary Fig. After training, the accuracy of the CNN achieves 98.6% in the testing set on average. From the above two curves, we can observe that both accuracy and loss become stable at the end of the training, proving that the training of the CNN fits the data well. Compared with the basic network, CNN has a higher testing accuracy  and the training loss is more stable. The precision, recall, and f1-score of CNN are 98.6%, 98.6%, and 98.6%. These results indicate that after extracting features by CNN, the performance of the network can be improved amazingly.We can learn more about how effective CNN is at categorizing human motion from its normalized confusion matrix, Supplementary Fig. The mathematical principle behind this phenomenon is that the CNN can select the most contributing features through down-sampling while the basic neural network does not have such a strong ability in extracting important features. To be more specific, the CNN selects features by convolving the input data while the basic neural network selects part of inputs through a linear combination that checks all inputs without a focus on a certain part of the input. Some parts of input can be more crucial in determining patterns of input instead of all parts being equally important. So, it is not difficult to infer that CNN, paying attention to the part of inputs, can gain better performance than the basic neural network, treating the whole input as the same.30. Compared with the naïve RNN, LSTM contains three gates  that can decide which information to throw away, to store in the cell state, and to output.RNN is specially designed for time-series data as it introduces hidden variables that contain information from the past. To have a better performance in this work, we implement a popular RNN structure, LSTMThe group work of these three gates is the secret behind the wonderful performance of LSTM. In this project, we firstly dropped input size from 999 to 20 through a linear layer, then implemented 4 layers of LSTM, each with 20 hidden nodes, and 1 linear output layer with the Sigmoid activation. The aim is to ensure that the structure of the RNN is as close to the basic network as possible. More details of the RNN can be witnessed in Supplementary Table Supplementary Figure After training, we finally get an accuracy of 95.2%, a precision of 95.2%, a recall of 95.4%, and an f1-score of 95.3% in the testing set after 10 times of training and testing. In contrast with the results of the basic network, the RNN can get higher accuracy in the testing set while the training loss is close. Compared with CNN, the curve of loss are more unstable, and the testing set accuracy is a bit lower. This result proves that although RNN can gain a great result in this time-series data, the fact that RNN pays attention to all input instead of part of the input leads to the RNN behaving a little worse than the CNN.From the normalized confusion matrix of the RNN shown in Supplementary Fig. From the loss curve of the RNN, we can find it converges to 0.3143 at the final stage of training. Unlike the naïve RNN, the LSTM can have a better gradient behavior. The mathematical principles behind this phenomenon are 1. LSTM preserves the error that can be back-propagated through time and layers 2. LSTM contains self-loops where the gradients can flow for a long time.31 to more than 100. Nowadays, it is common to see a Neural Network with a depth of more than 1000. The problem with the training of the Deep Neural Network is the vanishing gradients. ResNet solves this issue in a clever way, which will be illustrated later. What we do is implement ResNet modification in the basic network, the network with increasing depth, and the network with increasing width, to say how much improvement that ResNet modification can bring to train the basic network, the network with increasing depth, and the network with increasing width respectively.The appearance of ResNet made the depth of the Neural Network grow from 19 in VGGSupplementary Figure We can learn more about how much ResNet can help the network at classifying human motion from its normalized confusion matrix shown in Supplementary Fig. Then, we conduct the experiments of ResNet modification in the network with increasing depth with 12 layers. Compared with the original increasing the depth network, an increase of 29.0% can be found in the testing set accuracy (from 68.1 to 97.1%) while the training loss curve is more stable. The amazing increase in testing set accuracy shows the magic that ResNet can bring to the training of the deep network. However, this magic seems to have a side effect when it comes to the network with a larger width (320 nodes in each hidden layer). The wide network with the ResNet modification only acquired an accuracy of 67.6%, a precision of 68.0%, a recall of 71.9%, and an f1-score of 69.9% after training and the training loss oscillates greatly. Compared these measurements with these of the original network, we can find that the wide network with ResNet even performed a little worse than the original wide network.Compared the normalized confusion matrices of the deep network with ResNet shown in Supplementary Fig. When the normalized confusion matrices of the deep network with ResNet shown in Supplementary Fig. The mathematical principle behind the ResNet is that it successfully transforms the gradients’ backpropagation from multiplicative to additive. This transformation is achieved by the skip connection of the ResNet. The details of the skip connection can be found in Fig. From its skip connection, the input and the output of the residual block are added, which helps the gradients become additive. The gradient of ResNet can be expressed as:From this equation, we can find: (1) Any Although this modification can solve the problem of vanishing gradient that exists in the deep neural network successfully, it cannot alleviate the issue of overfitting in the wide neural network. Therefore, the deepest network gained the best improvement while the widest network gained the worst.Setting the Basic Neural Network as the baseline, we modified its structure by increasing the depth, increasing the width, applying the CNN, applying the RNN, and applying the ResNet. Next, we trained those modified deep learning models and measured their accuracy, precision, recall, f1-score, and running time when doing the tests in the testing set. We take an average after 10 times training and testing. The results of these measurements are recorded in Table From the table, we find that CNN, RNN, Basic Neural Network with ResNet, and Deep Neural Network with ResNet can gain better performance than the baseline, while Deep Neural Network, Wide Neural Network, and Wide Neural Network with ResNet perform worse than the baseline if we compare their f1-score.Although the performances of the Deep Neural Network have already been successful after we implement some advanced structures, the original system still suffers from the curse of dimensionality. That means the dimension of wireless signals is too high to have a good performance. To be more specific, as the dimensionality of data rises, the volume of space expands so quickly that the accessible data becomes sparse. Due to this sparsity, the amount of data required to train a deep learning model sometimes climbs amazingly. However, it is difficult to collect more data due to the limitation of the hardware. Therefore, to have a better performance, it is crucial to reduce the dimensionality of the data.32 is a commonly used technique for reducing dimensions. PCA converts a set of observations of possibly linearly correlated variables into a set of values of linearly uncorrelated variables an orthogonal transformation. Those linearly uncorrelated variables are called principal components. Since the number of principal components should be smaller or equal to the number of original variables, it can be a great unsupervised learning model for reducing dimensions. PCA was firstly proposed by Pearson in 190133, which could only be adapted for non-random variables. Later in 1933, Hotelling34 developed PCA for random variables.Reducing dimensions can effectively solve the problems of the curse of dimensionality. PCAIn this work, we adapted PCA into our data and reduced the dimension of our data from 999 to 111. A wonderful improvement can be witnessed in the result, illustrated by the following part.Firstly, we implement the data preprocessed by PCA to the Basic Network. Keeping other conditions the same and doing 10 times training and testing, we can achieve an accuracy of 98.6% in the testing set The reason that low dimensional data has better performance is that we extract principle components of the raw data by PCA, the influences of noise and high dimensions are removed, and it becomes much easier for the deep learning model to learn how to classify Sitting and Standing. Although ResNet can solve the problem of vanishing gradients, it cannot perform well if the data is hard to classify due to the influence of noise and high dimensions. Therefore, reducing dimensions should be an important part of the training the deep learning model.In this work, we first collect a CSI dataset of sitting and standing activities and build a basic network to classify these activities using PyTorch.Thereafter, we increase the depth and the width of the neural network, adopt advanced neural network structures, and reduce dimensions by PCA to observe how the testing accuracy and loss curve will change. Through these changes, we thoroughly explore how to modify the deep learning in this wireless CSI dataset.Although increasing the depth can improve the expressiveness of the neural network and extract abstract features of the input, problems of vanishing gradients can become more serious as the depth increases. This problem leads to slow training and a bad result in both the testing accuracy and the loss. Increasing the width can increase the dimensions of the feature map in the hidden layers. However, too many parameters in the hidden layers can lead to the problem of overfitting.Transforming the structure of the basic network to CNN can have an amazing improvement in the result as CNN can sample the most important features to improve the training efficiency of the classifying network. RNN can store the past information in its hidden variables, and LSTM improves the naive RNN by introducing the input gate, the forget gate and the output gate. Although LSTM is specially designed for time-series data, it focuses on all input instead of the most important features like CNN. Therefore, it performs a little worse than CNN.ResNet is a milestone of deep learning that makes the extremely deep neural network possible. From our experiments, we can find that the ResNet can increase the testing accuracy of a 12-layer neural network with 20 nodes in each hidden layer by 29% while it cannot work well for a 4-layer neural network with 320 nodes in each hidden layer. Therefore, ResNet is effective for a deep neural network, but it cannot be so effective for a wide neural network. In comparison with ResNet, reducing dimensions by PCA can increase the testing accuracy of the Basic Network, the testing accuracy of the 12-layer neural network with 20 nodes in each hidden layer, and the testing accuracy of the 4-layer neural network with 320 nodes in each hidden layer all to around 98%. CNN with PCA has not much difference while RNN with PCA improves accuracy, precision, recall, and f1-score to around 99%.39, semi-supervised training was applied radar-based Human Activity Recognition with an accuracy of 87.6%. The article35 applied ResNet29 to recognize human activities with an overall accuracy of 96%. While in Ref.40, the team designed the HARnet for mmWave based real-time human activity recognition. HARnet can achieve an off-line accuracy of 93.25% and real-time accuracy of 91.52%. According to Ref.41, a hybird framework composed by 1-D Convolutional Neural Network, Recurrent Neural Network, and 2-D Convolutional Neural Network is implemented for radar-based human activity recognition with an accuracy around 95%. In Ref.42, five distributed pulsed Ultra-Wideband (UWB) radars in a coordinated network is applied to human activity recognition, bringing an average improvement of 17.5%. 1D-CNN applied in Ref.36 gained an accuracy 94.60% while37 applied a multi-view CNN-LSTM, resulting in an accuracy of 92.00%. Although these recent works have a great performance in radar-based human activity recognition, they still do not surpass the accuracy of our method through combining LSTM and PCA (99.1%).The highest accuracy is gained by the combination of RNN and PCA, with an accuracy of 99.1%. In Ref.The training processing of PCA preprocessed data is more stable compared with the ResNet. The feature map of the raw data and the feature map of the PCA preprocessed data shown in Fig. In conclusion, modifying the neural network Structure and reducing dimensions can be two perspectives in improving the deep learning model for the contactless AI-enabled human motion detection system. Reducing dimensions might play a more crucial role in the improvement.Supplementary Information.


With the continuous development of artificial intelligence, data-driven fault diagnosis methods are gradually attracting widespread attention. However, in practical industrial applications, noise in the working environment is inevitable. This leads to the fact that the performance of traditional intelligent diagnosis methods is hardly sufficient to satisfy the requirements. In this paper, a developed intelligent diagnosis framework is proposed to overcome this deficiency. The main contributions of this paper are as follows: Firstly, a fault diagnosis model is established using EfficientNet, which achieves optimal diagnosis performance with limited computing resources. Secondly, an attention mechanism is introduced into the basic model for accurately establishing the relationship between fault features and fault modes, while improving the diagnosis accuracy in complex noise environments. Finally, to explain the proposed method, the weights and features of the model are visualized, and further attempts are made to analyze the reasons for the high performance of the model. The comprehensive experiment results reveal the superiority of the proposed method in terms of accuracy and stability in comparison with other benchmark diagnosis approaches. The diagnostic accuracy under actual working conditions is 86.24%. Prognostics and health management (PHM) is essential for modern industry ,2. HowevAs one of the advanced algorithms, deep learning has played an important role in computer vision, natural language processing, speech recognition, and other fields . RecentlIn addition to the above models, convolutional neural network (CNN) has been extensively studied for fault diagnosis. This method utilizes operations such as local receptive fields and weight sharing to extract fault features more quickly and accurately. With the continuous development of CNN, some excellent architectures have emerged, such as VGG, ResNet, Inception, etc. These algorithms have broad applications in fault diagnosis. Dibaj  proposedHowever, it remains a most challenging issue to establish the excellent fault diagnosis model with limited circumstances . The traAs an efficient classification algorithm, EfficientNet provides an amazing way to scale neural network models by enhancing depth, width, and resolution ,28. It iFor this purpose, a fault diagnosis model based on EfficientNet is proposed in this paper. Continuous wavelet transforms (CWT)  is emploMeanwhile, it is worth noting that noise exists in the actual working environment of mechanical equipment. The collected signals during the acquisition process contain characteristic features of the fault, but this feature information might be submerged in much noise, leading the model to learn inaccurate features ,31. HencIn this paper, a fault diagnosis method is proposed by combining time–frequency analysis and a new architecture, EfficientNet. The proposed method in this paper improves the efficiency of fault diagnosis with limited resources and also ensures diagnosis accuracy. In addition, the influence of background noise on the model accuracy is significantly reduced. The main contributions of this paper are summarized as follows.(1) A fault diagnosis method for rolling bearings based on EfficientNet is proposed. Compared with other methods, this method can obtain the optimal diagnosis efficiency with limited resources.(2) To reduce the influence of noise on diagnostic accuracy, a dual attention mechanism is introduced to refine the local features and effectively capture global features. The role of the attention mechanism is presented through the visualization of the results.(3) The programmed algorithm is evaluated with noisy samples, and the diagnostic accuracy is 89.54%. In the application of actual working conditions, the model also is superior to other methods, and the diagnostic accuracy is 86.24%.The rest of the paper is organized as follows: To accurately extract representative fault features and build the optimal model with the limited computational resources, EfficientNet is selected in this paper as the baseline model to construct a rolling bearing fault diagnosis framework. The EfficientNet model is primarily constructed with MobileNet ,34,35. IDepthwise separable convolution (DSC) layers are used as the fundamental building blocks of MobileNet. Therefore, this structure is also known as the mobile convolutional (MB Conv) layer. This operation successfully reduces the computation latency and parameter size. The DSC is composed of two parts: depthwise convolution (DWC) and pointwise convolution (PWC).The DWC is constructed with the same number of filters as the input channels, so it configures the corresponding filter for each channel. This ensures that the output image after passing through the layer still retains its depth. Specifically, DWC utilizes a single 3 × 3 convolution kernel to slide layer by layer over the input feature map, thereby generating an output channel after each slide.The main function of PWC is to adjust the depth of the input feature map. The process is implemented by adjusting the thickness of the output with a 1 × 1 convolution kernel. The output with a single kernel is generated after the PWC operation.The convolution process with the combination of DWC and PWC is shown in It proves that residual blocks are helpful to build deeper networks with a strong performance in ResNet. For the same purpose, a similar block is introduced in MobileNet. Traditional residual blocks connect the convolutional layers using skip connection, where the start and end layers are wide and the middle layers are narrow. In other words, the features are first down-dimensioned, and then up-dimensioned after a convolution operation. However, inverted residuals are the opposite, where features are first up-dimensioned and then down-dimensioned after a convolution operation. This is also designed for DSC to proceed smoothly, ensuring that the feature extraction process can be carried out in a high-dimensional state.To capture rich features, there is a preference for increasing the width and depth of the model in traditional data-driven fault diagnosis investigations. Indeed, many theories and studies have demonstrated that larger models are easier to be trained and for capturing fine-grained features. However, as the network width continues to expand, its precision tends to saturate such that the model is no longer actively capturing advanced features. Similarly, increasing the depth of the model improves its ability to learn complex features, but it is prone to gradient disappearance and explosion. Furthermore, it is also difficult to balance the model accuracy and speed in terms of the selection of the model resolution. The EfficientNet uses MobileNet as a basic framework, which improves this problem. This approach searches for the optimal model with limited resources by scaling the depth, width, and resolution of the model. In contrast to the arbitrary design concepts of standard convolutional models, this method uses the composite coefficients ribed in .1)DeptDeptϕ to Step 1: Assuming that twice the available computational resources are currently available, the small grid search method is carried out under Step 2: According to Equation , EfficieGiven the limited computing resources, EfficientNet-B2 pre-trained by ImageNet is adopted as the baseline structure of the model. More details are given in Since bearings operate under a variety of conditions, fault features are easily buried by complex background noise and interference. Background noise causes less variation between different levels of fault features of the same type. The traditional intelligent diagnosis methods are easily misled by noise when extracting fault features. Therefore, these factors seriously affect the diagnosis accuracy of intelligent fault models. It remains the key issues to enable the model to extract effective fault features without noise interference.Currently, we have noted in existing research that the attention mechanism can increase the receptive field of the underlying features via the attention map. Hence, it can be implemented to emphasize representative features and suppress irrelevant information. This prompts the model to distinguish fault features from noise, thereby improving the diagnosis accuracy.In this paper, an attention mechanism is introduced to the EfficientNet, which can not only increase the precision of fault feature extraction under complex background noise, but also adaptively integrate the dependencies between local and global features of fault samples.F denotes the input features, Sigm indicates the sigmoid function, and The introduced location-wise soft attention includes channel and spatial attention mechanisms . The chaThe main role of spatial attention is complementary to channel attention. After feeding the feature map, the global features are initially extracted on the same channel of the feature map, which is achieved by performing maximum pooling and mean pooling operations, respectively. Afterward, these two features are concatenated, and a spatial feature map is generated using a convolutional layer. The above spatial attention operation implements the recalibration of features in spatial dimensions. It is calculated bysoftmax associated with the actual fault classes is added to the head of the subsequent network for classification. In this paper, the proposed attention EfficientNet is built using MobileNet as the basic module. The process decomposes the traditional convolution into two steps: depthwise convolution and pointwise convolution. This significantly reduces the number of weight parameters computed by the network and improves the computation speed. In addition, the conventional EfficientNet is modified to enhance the model ability to learn fault features. Location-wise soft attention is introduced to the standard model, which improves the sensitivity of the model to fault features under heavy background noise. A To improve the efficiency of the intelligent fault diagnosis model, as well as improving the diagnosis accuracy of the model when the samples have complex noise, an intelligent fault diagnosis method based on EfficientNet and the attention mechanism is proposed in this paper. Specifically, (1) The vibration signals of the rolling bearing are collected by the acquisition equipment.(2) The TFRs are obtained by transforming the original vibration signals via CWT, and then they are labeled as training samples.(3) The hyperparameters and structural parameters of the attention EfficientNet model are initialized.(4) The fully trained model is applied to identify the test samples.(5) The performance of the proposed model is finally evaluated by the diagnosis results.In practical engineering applications, the operating environment of the mechanical system is complicated, and noise interference is a primary driver to contaminate the vibration signals of rolling bearings. It is a challenging task to improve the model performance for fault diagnosis under strong noise. The proposed method in this paper is developed to address this situation.Therefore, the effectiveness of the proposed method is evaluated by setting up experiments containing different levels of background noise in this section. In addition, the effectiveness of the proposed method is verified using an operational printing equipment test bench, which is designed to evaluate the model efficiency under actual working conditions.The effectiveness of the proposed method under simulation noise is investigated in this section. This experiment uses data provided by Case Western Reserve University Lab (CWRU) , which aConsidering the highly variable noise, we fail to obtain enough labeled samples under different noise conditions. To simulate complex noise scenarios in real industrial production, a composite signal is formed by combining different signal-to-noise ratios (SNRs) with the original signal with additive white Gaussian noise. These composite signals are adopted to validate the diagnosis performance of the attention EfficientNet under different strong noise. The SNR is defined as below.Although the raw vibration data contain sufficient information about the bearing health condition, it is not clear the interpretability of directly applying this information to fault diagnosis. Considering the non-stationarity of the vibration signal under actual working conditions, the continuous wavelet transform is utilized to process the raw data in this paper.According to existing investigations, it can be found that Morlet wavelets have a low error rate for extracting periodic shock signals from bearing vibration signals. In addition, the time–domain waveform of the Morlet wavelet is similar to the impact features of the bearing fault signal, which allows a well-matched relationship between it and the fault signal. However, the resolution of TFR directly processed via CWT reaches 1167 × 875; this undoubtedly increases a huge amount of model structure parameters and computational consumption. For this purpose, a bicubic interpolation algorithm is adopted to resize the original image to 260 × 260 (common size). These TFRs can be used as input to the attention EfficientNet.To verify the validity of the proposed method, we use the standard EfficientNet as a comparison method. In addition, we compare the proposed method with ResNet . ResNet softmax is chosen as the classifier. Meanwhile, corresponding ablation experiments are added to each method for investigating the role of attentional mechanisms.The above comparison methods are designed with the same input dimensions as the present method, and The proposed method is written based on the deep learning framework Pytorch, and the experiments were carried out on a Core i7-9700, NVIDIA RTX 2070. The model hyperparameters in this study are all obtained from cross-validation experiments, with an initial learning rate of 0.01, a batch size of 4, and the number of iterations set to 30. To minimize cross-entropy losses, a stochastic gradient descent (SGD) optimizer with 0.9 parameter momentum is introduced. In addition, a dropout with a scale of 0.3 is introduced to moderate the overfitting phenomenon.The validity of the model is tested with composite signals SNR, ranging from −6 dB to 6 dB. In the process, we simultaneously demonstrate the necessity of the small-batch training sample option. The batch sizes are established to the integer power of 2, and this is to conform to the coding patterns of the computer and enhance operational efficiency. The parameters remain the same for each experiment, except for the SNR and batch size. The results are displayed in the It is evident from the table that the diagnosis accuracy declines as the SNR decreases regardless of batch size. When the SNR is −6 dB, the diagnosis accuracy of the model is lower than in the other cases. However, as the SNRs increase, diagnosis accuracies of the models also increase. For instance, when the SNR is larger than 4 dB, the diagnosis accuracy generally exceeds 95%. These situations are attributed to the fact that noise enhances the difficulty of the model in feature learning, thereby reducing the diagnosis accuracy. In contrast, small batch sizes of learning patterns outperform large batch sizes for the same SNRs. When the SNR is −4 dB, the diagnosis accuracy of a batch size of 4 is approximately 4.5% higher than the batch size of 128. It can be concluded that a small batch size learning pattern is helpful for the model to extract more accurate fault features.Based on the results in the previous section, the optimal hyperparameters are determined for the attention EfficientNet. In this section, the comparison experiments are performed on the composite signal dataset. It is worth mentioning that the SNR is −4 dB in this case. In addition, we found that the deep model achieved better results than the shallow model. For example, the average accuracy of ResNet50 is 87.87%, while the accuracy of ResNet18 is 87.06%. Additionally, the accuracy variance indicates that ResNet50 is more stable. Compared to other methods, the Inception architecture reduces the number of model parameters and improves the performance degradation problems caused by overfitting. Although the improvement in accuracy is limited, results reveal that this approach enhances the stability of the model.The diagnosis results of the attention EfficientNet with different pre-processing methods are shown in The EfficientNet fault diagnosis method proposed in this paper is based on the attention mechanism to extract more representative features from TFR. Hence, it is necessary to employ attention visualization to explore the role of this mechanism. The main purpose of this mechanism employed in the proposed method is to enable the model to increase the emphasis on representative features in the TFR. Specifically, the attention mechanism is adapted to adjust the distribution of features during feature extraction, which enables the model to focus on local features and reduces the effect of noise.In this context, we attempt to show the degree of attention for attention EfficientNet using the attention weights. As shown in It is clear from the To verify the effectiveness of the proposed method in real-world noise, this method is examined on typical rotating equipment (printing press) with actual conditions. The detailed information of datasets is shown in the In this section, the same comparison methods are selected as in However, the proposed method has a relatively satisfactory performance with an average diagnosis accuracy of 86.24%. Additionally, the accuracy variances reveal that the proposed method has the best stability in real-world noise environments compared to other methods.The confusion matrices for four methods in the third experiment are given in To visualize the effect of the feature extraction process for different models, the high-dimensional features are displayed by the t-distributed stochastic neighbor embedding (t-SNE). In detail, this algorithm converts the high-dimensional features of the last fully connected layer of the four comparison methods into a two-dimensional distribution for visualization. The visualization results are shown in In this paper, a method based on attention EfficientNet is proposed to achieve the fault diagnosis of rolling bearing. The bearing vibration signal is first processed by CWT, and the multi-channel TFR is obtained. Next, these TFRs are then input into the MobileNet-based EfficientNet to extract hidden features for fault identification. In this process, the attention mechanism is introduced to guide the feature extraction process of the model, which enables the model to focus on the representative fault features and improves the diagnostic accuracy under complicated noise.The effectiveness of the proposed method is verified by two experimental cases. Compared with other advanced methods, this diagnostic method has the following advantages: (1) This method can build an optimal model with limited resources, and the introduced attention mechanism improves the ability of the model to focus on fault features. (2) Compared with other methods, the proposed method has better diagnostic accuracy and stability. (3) This method has excellent performance under strong noise, and the average accuracy is 89.54% in the dataset containing simulated noise and 86.24% in the dataset containing real noise.There is still much room for improvement, and our future work will focus on the following aspects. (1) More efficient signal preprocessing techniques will be explored to further improve the diagnostic accuracy of the model. (2) Different attention mechanisms are introduced to improve the feature extraction capabilities of the model.


The intersection analyses and comparisons are performed based on estimated traffic measurements such as travel time, density, lane density, occupancy, and normalized waiting time. Since the critical edge of each intersection network is identified based on a higher normalized waiting time, new traffic signal designs are suggested based on the actual critical turning movements and improvements in vehicle travel time are achieved to better accommodate the actual traffic demand.Traffic simulation is widely used for modeling, planning, and analyzing different strategies for traffic control and road development in a cost-efficient manner. In order to perform an intersection simulation, random vehicle trip data are typically applied to an intersection network, making them unrealistic. In this paper, we address this issue by presenting two different methods of incorporating actual turning movement count (TMC) data and comparing their similarity for intersection simulation and analysis. The TMC of three intersections in Las Vegas are estimated separately for one hour using a developed vision-based tracking system and they are incorporated into Simulation of Urban MObility (SUMO) for estimating traffic measurements and traffic signal design.  Intersections are defined as a point where different road user  movements converge, and they have attracted a large amount of attention for network design, management, and planning. Due to the existence of variant road users and conflicting route paths, an appropriate intersection design enhances mobility and safety goals in the public realm. Moreover, traffic signal design is crucial, since it manages different movements, conflict ratios, and road users’ travel time . As a reFor the analysis and design of intersections and their constructive elements , it is critical to monitor traffic flow and extract important traffic data that reflect how well the corresponding design works. For instance, by analyzing the vehicles’ trajectory and amount of delay that vehicles face during their trip, the intersection flow and signal performance are evaluated, and recommendations are considered for optimizing the intersection’s design and signals . MoreoveA conventional way of turning movement count (TMC) estimation is manual observation and recording of traffic by a human. However, its accuracy is affected in congested scenarios, and it is costly due to its requirement of a human. As a result, some studies ,5 have aSupposing that the TMC data is provided by another source , how can this data be useful in providing more insight about the intersection?How we can evaluate road users’ statistics regarding the current intersection design, for example, travel time and waiting times of vehicles at the given intersection?How we can evaluate the current design of the traffic light, revise its timing based on the flow distribution, and measure its effectiveness?As an alternative approach, computer vision can assist with the automatic collection of TMC  as long This paper provides answers to all the aforementioned questions. The basic approach is to incorporate the benefits of the traffic simulators with the actual turning movement data. The actual TMC data will help with minimizing the unrealistic effects of the simulator in addition to having a cost-efficient framework. We propose two different methods of incorporating the TMC data generated from the vision-based tracking system for intersection analysis. We utilized Simulation of Urban MObility (SUMO) , which aThe remainder of this article is organized as follows: In the realm of intersection analysis using software simulation tools, analysis of the traffic and evaluation of intersection and traffic lights have been addressed by variety of research studies. For instance, Harahap et al.  used MATIn addition to MATLAB, VISSIM is another common software used for simulation of traffic in different studies . For insAlthough VISSIM is an extremely powerful traffic micro-simulation tool that can even be linked to MATLAB, it is not free source, causing limitations in its usage. Alternatively, SUMO has been used by different studies for intersection simulation and analysis ,16,17. Ft-tests. Due to frame-based resolution of the route creation method for incorporating TMC flows into SUMO, experimental data from this method are further analyzed to provide comprehensive analysis on the simulated intersections. More insights about different edges of the observed intersection are provided using various critical measurements such as travel time, normalized waiting time, lane density, occupancy, and queue length. Finally, the intersection edges with the highest queue length are identified, and a new design for traffic signals is suggested based on actual critical turning movements to reduce the average vehicles’ travel time. The remainder of the paper explains the integrated methods and the extraction of measurements in more detail.While there are few recent studies addressing the real traffic scenarios, there is no systematic approach showing how to import real turning movement count (TMC) traffic data for intersection analysis and provide edge-based measurements of intersection networks through estimating crucial traffic measurements such as travel time, lane density, waiting time, and queue length. In this paper, we utilize a vision-based tracking system to provide the 5-min TMC count intervals of three connected intersections in Las Vegas over one hour. The collected data are followed by two different methods of incorporating TMC data into SUMO and comparing the similarity of simulation measurements  using The simulation system framework, which is conducted until intersection analysis, is presented in The tracking system utilizes computer vision techniques to provide labels for individual road users using traffic cameras. The main reason for using computer vision techniques is to provide the count data for a long time period in a cost-efficient way. Since intersections host a mix of road users  with varied motion patterns, state-of-the-art techniques are required to provide detection of each road user over the sequence of frames. As an outcome, You Only Look Once (YOLO) architecture is a reasonable choice since it provides predictions in one pass of an image, resulting in faster detection in comparison with other convolutional neural network-based techniques such as faster RCNN . We utilThe detection system generates a detection list used to initialize the tracking list. During the tracking phase, detection and tracking lists are updated and the tracking list will maintain the trajectory of each individual road user during their tracking lifetime.update, add, and remove are sequentially run for every frame captured from the input stream. The update method performs discriminative correlation filter tracking  for eacupdate when it has a high overlap with an existing track. Then, the intersection over union  or not.Equation ) IoU>0.5add method. If an existing track does not find a detection for a sequence of three frames, it is marked for deletion through the remove method.When a detection does not match the existing tracks in the track list during the association process, a new track is created. This task is performed by leveraging the The turning movement and pedestrian crossing counts were conventionally acquired by technicians who manually recorded the count number of vehicles and pedestrians going each direction during daily peak hours. This becomes tedious and costly for turning movement counts that needs count records from different directions that occur concurrently. The proposed system automatically recognizes the vehicle and pedestrian movement patterns using the longest common sub-sequence (LCSS) method with the modeled typical paths of each intersection see . LCSS isAn essential step in determining the turning movement count is to provide some reference trajectories for comparison, called typical paths. The typical paths consist of ) points  is returThe main objective for studying intersections and their research directions is the identification of the data collection system requirements and the selection criteria for the target intersection. Since our objective is to obtain TMC data estimation using the developed vision-based tracking system, we need to identify the intersections that have accessible traffic cameras. Moreover, intersection layouts corresponding to monitored intersections are required to be designed for simulation purposes. In this work, the NETEDIT tool, a road network editor for road traffic simulation in SUMO, is utilized. This step reflects the determination of the actual number of lanes for each direction of the intersection that can be verified though scene investigation using traffic cameras as well as Google Maps.green), g (green), y (yellow), and r (red). While G shows the priority, g causes vehicles to stop if another vehicle uses a higher-prioritized flow stream. Vehicles must stop if they face r, and y causes them to decelerate if they are far away from the junction or otherwise pass through the junction. More details about the actual design of traffic signal lights are stated in After creating the nodes and edges which contain the actual number of lanes for each observed intersection, traffic lights need to be designed and applied at a junction. The traffic signal states are defined for each moving directions using characters such as G  are conventionally generated to provide new labels for the edges as information identifiers. This will be applied into the network file to demonstrate the route paths of the simulated vehicles.Two methods are presented in this paper to incorporate TMC data of the vision-based system for the simulation: O/D (output/destination) matrix route creation and edge-based route creation. More details for each method are as follows:In this method, TMC data are organized in the form of an output/destination (O/D) matrix where each row represents the outward and inward flow edges and the amount of corresponding count data. This matrix provides SUMO with vehicle trips from each flow direction on the intersection network edge. Due to the creation of the TAZ, the O/D matrix can be simply arranged based on the new labels referring to the edges and TMC data that have already been defined within the TAZ file. For example, a row of 1i, 3o, 15 implies that a total of 15 vehicles took the WBT route in their travel path during the simulation time.Due to existence of the trip demand model for every flow direction as well as the intersection network, the DUAROUTER  tool genEdge-based route creation refers to creating the route paths directly as an output of the vision-based tracking system. Since the tracking system can determine the turning movement count with prior knowledge of the departing frame number, the turning movement routes can be created according to departure time based on the TAZ. This method does not follow the shortest path algorithm and reflects actual traffic based on a higher time resolution, since each vehicle is labeled based on departed time frame for all movement directions.In order to set up the simulation, the generated network file, route file, and simulation time with a list of additional files to address traffic measurements as output are prepared and identified for the traffic simulator. The traffic simulator receives the configuration file and conducts the simulation process by injecting the vehicles according to the route file into the intersection network. The output information regarding the simulated vehicle statistics as well as the edge-based traffic measurements is generated after completing the simulation experiments.Travel time (TT): Time in seconds that vehicles require to pass the edge/lane, which is calculated based on the mean speed for the front of the vehicles.Density (DE): Number of vehicles per kilometer (#veh/km) for the edge of the intersection network.Lane density (LD): The density divided by the number of lanes, i.e., the vehicle density on the edge per lane (#veh/km/lane).Occupancy (OC): Ratio of vehicles to the distance, presented in percentage. A value of 100 indicates that vehicles are placed bumper to bumper on the whole edge with a minimum gap of zero.Normalized waiting time (NW): The accumulated number of seconds that vehicles were considered halting (speed < 0.1 m/s) over the total number of vehicles on the edge.Intersection analysis is a data analysis process that utilizes extracted measurements and their visualization to obtain more insight regarding road user movement patterns and traffic management facilities. For instance, the vehicle waiting time and speed are two crucial measurements for addressing the efficiency of the traffic signals and intersection capacity. Here are some important measurements that we have utilized in this work for intersection analysis:The evaluation results for the vision-based detection system show how well the existing YOLO architecture can deal with the detection of low-resolution and small road users. The precision and recall metrics for evaluation of the detection system are defined in Equation .(2)PrecTrue positive  as a division of the Regional Transportation Commission of Southern Nevada (RTC). The vision-based tracking system for remotely monitoring the cameras was implemented with Python 3.6 and OpenCV 4.5.2 and it was run on a dedicated high-end PC with a quad-core Intel i7-7700 3.6 GHz processor, 16 GB RAM, and an Nvidia Geforce GTX 1070 GPU with 8 GB of total RAM and 2048 CUDA cores. Additional Python scripts were developed for trajectory collection of road users and preliminary analysis such as estimates of turning movement counts. Eclipse SUMO 1.13.0 and its package tools  were used for intersection simulation and further analysis.In order to simulate the intersections with appropriate settings, the actual number of lanes need to be designed for each edge of the intersection network. After creating the intersection network through creating nodes, edges, and revising the number of lanes on each edge, a traffic light is placed at the junctions of the edges to manage different flow directions. The default signal design consists of four major phases, where two phases are branch through and branch right from the west/east and north/south zones and two are protected branch lefts. The basic traffic signal is designed to provide a fair amount of time dedicated to each flow direction. While the main flow stream direction  has a higher phase time , it overlaps with the flow of the opposite direction to maximize the occupancy since they do not have conflicts. While the vehicles with a flow in the left-branching direction are permissive and should yield to the other conflicting income flow directions, they have a protected phase of 6 s to resolve the starvation issue. Since each major signal phase is followed by a yellow traffic light for a duration of 3 s, eight total signal phases will create a traffic light cycle to manage the traffic of all turning flows at a junction. As a result, the basic signal will have a 90-s cycle time where the four major phases one, two, three, and four constitute 33.6%, 6.6%, 33.6%, and 6.6% of the duty cycle, respectively, and each yellow signal is 3.3% of the duty cycle.The three intersections (INT1–3) were monitored separately using the developed vision-based tracking system on July 18th of 2022 from 7:00–8:00 am and their turning movement counts were estimated with 5-min intervals using typical paths and the LCSS method for route path similarity comparison.The estimated turning movement counts are utilized for calibration of each intersection simulation using two proposed methods: O/D matrix-based route creation (M1) and edge-based route creation (M2). As we stated in Wait to insertion time (WIT): The mean time that all vehicles until now and within the reported time step had to wait before being inserted.Mean latest travel time (MLT): The mean travel time of all vehicles that have left the simulation within the previous and reported times.Mean speed (MSP): The mean speed over all vehicles in the network.Mean relative speed (MSR): The mean speed of all vehicles in the network relative to the speed limit.In order to gauge the similarity of the two methods, the simulation of each intersection for the one hour of monitoring time was performed and the simulation-related measurements were estimated. The simulation-wide measurements provide statistics on the time that vehicles needed to complete their routes and the number of vehicles’ states in the simulation, such as loaded, inserted, running, waiting to be inserted, and reached their destination. The simulation measurement results that we utilized for comparison of the two methods are as follows:t-tests were performed to find the true difference between the two groups of measurements using the simulation output for each method in 3600 time step observations. The two-tailed tests for unequal variance for each of the two groups of extracted simulation measurements are shown in Based on the comparison between the two proposed methods, there is a significant difference in the simulated measurements if the start time order of turning movements (in seconds) is not respected, even though the total count might be same for the entire hour. So, Method 2 is preferred since it reflects the correct timing labels for each travel path movement and also provides a lower mean travel time for each injected vehicle into the simulation environment. Consequently, Method 2 (M2) is deployed in the rest of this paper for intersection simulation and further analysis.As result of simulating three different intersections, traffic measurements are generated for each edge of the intersection. The highest density and occupancy belong to INT3 and INT2, respectively, which are correlated with the higher turning movement count data shown in The normalized waiting time was calculated for each edge of the three intersections, and edge 1i demonstrated the highest waiting time value. This is due to the high volume of vehicles moving from the west zone to other directions and left-turning vehicles that cause obstacles in front of the following vehicles. Motivated by this issue, we extracted the queue length on edge 1i to compare this important parameter’s difference between the intersections.Similarly, Based on observation from the queue length on edge 1i, we re-evaluated each intersection to improve the vehicles’ travel times and waiting times. Since the actual turning movement count was provided by the vision-based tracking system during the monitoring time, the traffic phases can be designed based on the critical turning movement which is based on the highest count of movement in the same phase. The results shows that INT3 and INT1 hosted the highest number of vehicles based on their collected turning movement count data. This is supported by the highest density value in t-tests on simulation variables of these two methods, edge-based route creation method is deployed further for intersection analysis and important traffic measurements such as travel time, density, lane density, occupancy, and normalized waiting times are estimated for each intersection. The experimental results manifest the highest waiting time on west bound edges on all three intersections due to high volume count of vehicles departing from this and high portion of them intend to turn left. Further analysis on queue length estimate of this edge shows a necessity of re-designing the traffic signal for INT1, and INT3 to better accommodate the high volume of vehicles waiting at specific time intervals. Subsequently, new design for traffic signals is suggested and evaluated based on critical turning movement data to minimize vehicles waiting time and their travel time. This work can be further used as direction for traffic engineers and researchers to calibrate their traffic simulators with actual turning movement count data to better analyze the requirements of their current intersections design and improve their traffic signal design in smart cities.In this paper, two different methods of simulating intersections and performing analyses with real turning movement data are presented. Three intersections at Las Vegas are studied due to accessibility of their traffic cameras and the estimated turning movement count of 1-h periods were imported into SUMO using two methods of O/D matrix-based and edge-based route creations. As a result of


The neighborhood area network of a smart grid usually has hierarchical wireless communication. Due to forwarding and processing more data, the upper-layer nodes are more likely to suffer congestion and energy exhaustion. This phenomenon leads to the failure of uploading data to the control center. To solve this problem, this paper proposes a scheme for constructing a multi-factor fault-tolerant hierarchical network. This scheme firstly defines a criterion for the generation of redundant links by multi-factor method in a hierarchical wireless network with the characteristics of the neighborhood area network. Then the redundant links are used to reconstruct the existing topology of the neighborhood area network for improving fault tolerance. Finally, a greedy routing algorithm is put forward to select a proper data transmission path by bypassing low energy nodes, further reducing the failure of uploading data to the control center. The simulation results show that the proposed scheme can effectively improve the fault tolerance of the network topology of the wireless neighborhood area network and balance the network energy consumption. Compared with the original scheme, the proposed scheme improves the fault tolerance by 35% and the relative transmission rate by 21%. The infrastructure of the Smart Grid Communication Network (SGCN) includes Wide Area Network (WAN), Neighborhood Area Network (NAN), and custom networks such as Home Area Network (HAN), Industrial Area Network (IAN), and Business Area Network (BAN), as In NAN, AMI has been rapidly developed. AMI has many functions; smart meter reading is an important traditional function for AMI. For smart meter reading, smart meters generally aggregate data and transmit a few times a day. With the development of the smart grid, the power system is changing to a power-electronics-enabled distribution system, especially with the increasing penetration of distributed energy resources (DERs). To monitor and manage the electronic devices (smart inverters) and DERs (solar photovoltaic) at the grid edge, AMI with two-way communications presents great potential ,11. ReceCommon methods for fault-tolerant topology control of wireless networks mainly include power control ,20, addiTo solve the above problems, this paper proposes a method to construct an improved hierarchical network with fault tolerance by the multi-factor method (IHN). The contributions of the paper include: (1) a criterion is defined to generate the redundant links by multi-factor method for a hierarchical wireless NAN; (2) a topology reconstruction method based on the redundant link criterion is proposed (this works as long as a network topology exists); and (3) a greedy routing algorithm is put forward to select a proper data transmission path on the reconstructed network topology to bypass low energy nodes, further reducing the failure of uploading data to the control center.The paper is organized as follows. Notations used in this paper are summarized in This section mainly reviews common methods that improve the fault tolerance and survivability of wireless networks. Then it identifies the technical gaps to motivate this research in this paper.Existing studies have shown that proper topology control can enhance the self-healing ability and survivability of NAN ,32. PopuPower control is mainly to control the communication range of a node by adjusting the transmitting power. If some nodes fail, the communication reliability can be ensured by adjusting the bigger transmission range. A fault-tolerant topology control algorithm based on multicast trees is proposed, it adjusts the transmission range of each node according to the multicast tree structure, and periodically uses topology reconstruction to maintain communication . A methoRedundancy design mainly ensures the reliability of communication by adding redundant nodes or redundant links. It can generate redundant transmission paths as alternate routes in case of node failures. In , the resTopology evolving is widely used in the research of fault-tolerant topology control . In the Although previous research works were devoted to solving the problems that may occur in wireless networks such as balancing the network energy and tolerating node failures, those studies have obvious limitations when applied to wireless NAN. For example, wireless transmitters with adjustable power usually require complex control strategies, which usually means longer processing time and higher economic cost; they are not suitable for real-time two-way communication in smart grid NANs ,42,43; tTo solve the above problems, this paper mainly starts from the perspective of introducing redundant links instead of redundant nodes to improve the hierarchical wireless NAN so as not to introduce excessive network costs. The redundant link is expected within the existing communication range of the smart device so that there is no need to install a long-distance connection transmitter on the smart device in NAN.(1)Establish redundant links between the nodes at the bottom layer. Since the data are aggregated to the upper layer, the links between the nodes at the bottom layer help to share the data traffic of the nodes at the upper layer.(2)Establish redundant links between nodes at the same layer. Nodes at the same layer are usually within their existing communication ranges, and there is no need to adjust the transmit power of wireless signals when establishing redundant links. In addition, when nodes at the upper layer fail due to energy exhaustion, the links between nodes at the same lower layer can ensure that the underlying nodes can communicate with the control center through other branches. Of course, the transmission link is only logical and its characteristics are affected by many physical factors. In this paper, we assume that the physical layer constructions are the same.Generally, WMNs (wireless mesh networks) such as IEEE 802.15.4 g  are commMost of the nodes in wireless NAN are smart meters and other smart devices that have been running for many years. The locations of those devices are relatively fixed. Therefore, NAN is usually a relatively static wireless network. According to the characteristics of the NANs, the left part of (1)Each node in a NAN network can receive and forward information; that is, each node in the (2)r in NAN is isotropic; that is, the communication radius of the wireless nodes in Wireless signal transmission for each node with a transmission radius of (3)Each node in a (4)If there is a direct link between any two nodes in a NAN network, the link will not be established repeatedly; that is, if there is already an edge between the two nodes in the (5)Each node in NAN has a limited link with other nodes, and the maximum number of links is Based on the basic criteria for adding redundant links mentioned above, this section firstly defines a criterion for the generation of redundant links by a multi-factor method in a hierarchical wireless NAN. Then, the redundant links are used to reconstruct the existing topology of the wireless NAN through IHN. Finally, a greedy routing algorithm is put forward to select a proper data transmission path from the redundant links. Appropriately adding a small number of redundant links can greatly enhance the fault tolerance of a wireless NAN. Adding links to a node means more data processing for the node, which will cause the energy of the node to be depleted faster. To improve fault tolerance and survivability of NAN as well as balance the energy consumption, the generation probability The smaller value of st value . Di+Dj−2(1)(2)(3)(4)In an actual application, this method first performs a depth-first traversal of the entire network to obtain the characteristic attributes of each node in the network , and calculates the organizational distance of each two points, and then according to the Algorithm 1, we calculate the connection probability, normalize it, and establish connections in turn according to the value of the connection probability.Topology reconstruction in IHN mainly involves three stages:(1)Initialization of wireless NAN. Coefficients (2)Computation of the generation probability for the redundant link between nodes. According to Formula (1), the generation probability for the redundant link between any node (3)Creation of the redundant links. The maximum number of redundant links in a wireless NAN is Algorithm1: Topology reconstructionInput: the initial topology andOutput: the improved topology/* for each node pairfor for according to (1), calculate redundant link probability and store it in endendAchieve the index of each node pair withk = 1;while  if Build links between   endendThe algorithm of topology reconstruction for a wireless NAN (Algorithm 1) is described as follows:Algorithm 2: Routing algorithmInput: the received packet  data to a local aggregator (LA). LAs also consist of similar Zigbee transmission devices and can work in router mode. Each LA will aggregate data from all WTSNs with a scheduled time frame and following specific data structure. Then LAs send collected data to a zonal head aggregator (ZHA). A Zigbee coordinator receiver mode is set in ZHA. The transmission range of each node is 1 km. Here, star topology is followed again to aggregate all WTSN cluster data before sending it to the central server. Finally, the ZHA sends data to the central server from star topology to tree topology. We generate the initial network topology in a range of 5 km × 5 km, and the transmission range of each node is 1 km. The simulation is divided into two parts. (1) The fault-tolerance of IHN is analyzed by comparing the improved topology generated by IHN and the Original Topology (OT). (2) The survivability of IHN is analyzed by comparing the improved topology generated by IHN with the network topology generated by a long-range link (LRL). Adding a long-range link is a common method for topology improvement, usually establishing the long-range links between the control center and the node with the largest betweenness centrality, which can effectively improve the fault tolerance and survivability of the network. We first conduct simulation in a small-scale network, and then conduct simulation in a large-scale network.According to the goals of the simulation, we design two simulation scenarios.Scenario 1. Verifying the fault tolerance of IHN in small-scale networks.To better describe the fault tolerance of a wireless NAN, we define two performance measurements: fault-tolerance C and network efficiency E. With the comparison of IHN and OT on performance measurement C and E, we analyze the advantages of fault tolerance for IHN.In a wireless NAN network, reliable communication means that all nodes can communicate directly or indirectly with the control center. To test the fault tolerance of the network, some nodes closer to the control center  is shown by a horizontal dotted line. It can be seen that the fault tolerance of IHN (IHN-C) increases significantly with the increase in m. Compared with OT, IHN has higher fault tolerance due to the redundant links between the underlying nodes. As the number of redundant links increases, the growth trend of C slows down. This is because when the links between the underlying nodes gradually reach full connections, C will no longer be improved.The trend of the curve for network efficiency E of IHN (IHN-E) is shown by the circle-marked line in Simulation results of scenario 2.Case 1. The nodes at the bottom layer continuously transmit data packets until the first failure node appears in the network.The relative transmission rate of IHN is higher than that of LRL. Because the distribution of residual energy in the IHN network is more balanced than OT and LRL, the lifetime of IHN is longer than that of OT and LRL. This means IHN can transmit more data packets than OT and LRL.Case 2. The distribution of the residual energy when transmitting the same number of data packets in LRL, IHN, and OT.Simulation results of scenario3.To ensure the reliability of the communication in the hierarchical wireless NAN of a smart grid, this paper focuses on the construction of a hierarchical network with fault tolerance by the multi-factor method for wireless NANs in a smart grid. The proposed scheme generates redundant links for wireless NANs based on the existing hierarchical network topology without adding redundant nodes and installing power-control transmitters. Simulation results show that the proposed scheme has better fault tolerance and survivability than the original topology and the topology with long-range links, when using the same greedy routing algorithm.Our future work will aim to improve and extend IHN. There are many directions for future research. First, we will further study the optimal value for the number of redundant links and the weight adjustment of the four factors that affect the establishment of redundant links. Second, we will implement the testing on the actual platform of wireless NANs in a smart grid. Third, we will consider the impact of physical layer factors on a smart grid NAN communication network.