The growing risk of new variants of the influenza A virus is the most significant to public health. The risk imposed from new variants may have been lethal, as witnessed in the year 2009. Even though the improvement in predicting antigenicity of influenza viruses has rapidly progressed, few studies employed deep learning methodologies. The most recent literature mostly relied on classification techniques, while a model that generates the HA protein of the antigenic variant is not developed. However, the antigenic pair of influenza virus A can be determined in a laboratory setup, the process needs a tremendous amount of time and labor. Antigenic shift and drift which are caused by changes in surface protein favored the influenza A virus in evading immunity. The high frequency of the minor changes in the surface protein poses a challenge to identifying the antigenic variant of an emerging virus. These changes slow down vaccine selection and the manufacturing process. In this vein, the proposed model could help save the time and efforts exerted to identify the antigenic pair of the influenza virus. The proposed model utilized an end-to-end learning methodology relying on deep sequence-to-sequence architecture to generate the antigenic variant of a given influenza A virus using surface protein. Employing the BLEU score to evaluate the generated HA protein of the antigenic variant of influenza virus A against the actual variant, the proposed model achieved a mean accuracy of 97.57%. The fast-paced determination of circulating influenza strains is crucial for vaccine manufacturing. The virus lipid envelope consists of glycoproteins namely Hemagglutinin (HA) and Neuraminidase (NA). Given the changing nature of the influenza virus, it undergoes two types of changes identified as antigenic shift and antigenic drift. The antigenic shift in the influenza virus is a significant change in virus surface protein that leads to a new HA, NA, or both. This profound change happens when the virus, which affects a certain population, gains the ability to infect another population and this type of change is less frequent and unpredictable . The antThe changes occurring in the influenza virus are primarily caused by changes in the virus envelope. These changes in the outer surface of the influenza virus account for the virus evasion from antibodies created during prior infections . SeveralDramatic breakthroughs have been realized recently in deep learning. Applications such as natural language processing (NLP) , 9, imagMany studies were conducted to devise computational methods for specifying a new influenza virus antigenicity. Such studies employed a variety of machine learning models in an attempt to identify the antigenic pair of an influenza virus. However, the studies mentioned in upcoming sections, in general, formulated the problem such that the input is the sequence of an existing virus and the sequence of newly discovered one, and the output defines the antigenic distance between both sequences as a classification or regression problem. Additionally, these studies are mentioned even though the approach is to generate the antigenic pair based on a given antigenic distance. These machine learning methods are mainly categorized into four categories; classical, tree-based, stacking and ensemble, and deep learning methods. The following subsections elaborate more on the work done, under each of the mentioned categories.Liao et al.  comparedNeher et al.  conducteYin et al.  made an The authors of  used an Dij follows definition by Archetti-Horsfall distance [Dij is the antigenic distance, Hij is the hemagglutination inhibition titer for a given strain i in relation to the antisera caused by another strain j. If The value of the antigenic distance is greater than or equal to 4, the two viruses are said to be antigenic distinct. Otherwise, they are considered similar [We incorporated two data sets in this research; antigenic data, and sequence data of influenza A subtype H1N1. Antigenic data was based on hemagglutination inhibition (HI) assay and was collected from reports of international organizations and published papers including World Health Organization (WHO), European Centre for Disease Prevention and Control (ECDC), the Francis Crick Institute (FCI), and Food and Drug Administration (FDA). Additionally, the HA protein sequences were taken from Influenza Virus Resource (IVR)  and Globdistance  given by similar . MoreoveSequence to sequence model is typically an encoder-decoder architecture [itecture . The basitecture  or GRU [itecture  layers, In this work, various architectures of sequence to sequence models were utilized. relying on different types of recurrent neural networks and a number of consecutive layers of the same recurrent neural network to search for the best scoring architecture. Attention mechanism combats losing context in long sequences by accurately encoding the parts of the input sequence that surround a particular word, or in this case amino acid. In this research, the attention mechanism used in [The nature of sequence to sequence models depends on the final hidden state of the encoder; and in the case of lengthy sequences, there is a significant probability that the initial context has been lost by the end of the sequence.  used in  was utilembedding methodology [learning rate decay is not only recommended as depicted in [αn is the learning rate at epoch, α0 is the initial learning rate, d is the decay parameter, and r is the drop rate.The acquired sequences data represent the HA protein of an influenza subtype H1N1. Each sequence consists of 327 letters representing different amino acids. In resemblance to an NLP problem, considering each letter as a word and the whole sequence as a sentence. Therefore, word hodology  is emploicted in , it is aicted in . Thus, tTwo different architectures were used as shown in Biobython for reading sequence data and Keras were used for developing the proposed deep learning architecture. Full implementation and data are made available on GitHub.Architectures in pn, calculated for each n-gram length by summing over the match for a generated sequence S against the original sequence O from the test set as in As a final step, the generated sequences acquired during the test phase are evaluated against the original sequences. Since the approach taken in this paper is similar to approaches used in NLP, it was convenient to use accuracy metrics used in NLP as well. Hence, the utilization of the BLEU score  which isL is the output sequence length and r is reference sequence length.wn represents the weights and typically equals 1/N and N represents the number of grams. According to the definition given by Since the BLEU score is precision-based, and recall term will be inconvenient to formulate over multiple references, thus, the brevity penalty calculated in During the training stage, architectures utilizing bi-directional recurrent layers—both LSTM and GRU—performed better than other architectures in terms of both training and validation accuracy, as shown in Additionally, the bi-directional GRU architecture achieved a higher mean accuracy. However, the gap between this architecture and attention-based architectures is fairly high, which indicates that attention might not be helpful in this case. The fast-paced development of sequencing methods has made influenza protein sequencing a standard segment of the influenza research process and vaccine determination. The recently discovered influenza protein sequences are saved in influenza databases like the NCBI influenza virus resource (IVR) database. If effective sequence-based antigenic prediction tools can be created, much of the burden of influenza monitoring may be alleviated. This work highlighted the process of generating an antigenic variant for influenza H1N1 viruses with little antigenic data. Moreover, a model based on additional antigenic data would better reflect the overall structure behind antigenic variation. To the best of authors knowledge, and at the time of writing this paper, there is no study in the literature that uses a generative modeling technique to address the problem of the influenza virus antigenicity.In this work, a single-step methodology was adopted that does not require feature engineering, and it is not dependent on virus discovery chronological order in training or prediction. Hence, this methodology requires less computational effort. Furthermore, integrating this methodology into existing influenza virus databases will alleviate a colossal part of antigenic pair identification and vaccine design.Finally, the metric utilized in this work namely BLEU is generally accepted in NLP problems. Since there is no conventional computational method for measuring the quality of generated sequences, the BLEU score is seen as an effective metric in measuring the quality of generated sequences.Throughout this work, antigenicity data was used along with influenza H1N1 HA sequences to generate the antigenic pair of influenza viruses. Exploiting the advancement in deep learning to evaluate different architectures to find the best suitable architecture for the task at hand. The main gain in this research is obtaining an end-to-end model avoiding feature engineering, utilizing a single-step prediction, and evading chronological order training. This work opens a new door towards generating the antigenic pair of an influenza virus depending on the antigenic distance in a novel way. This work can help to reduce the time consumed to define the antigenicity of a newly discovered H1N1 influenza virus relying on the virus HA sequence and antigenic distance. Furthermore, this work can accelerate vaccine selection and manufacturing. As demonstrated in this search generative modeling techniques are viable for addressing the influenza subtype H1N1 antigenicity problem. For future work, the authors are planning to experiment with different generative methodologies to generate the antigenic variant for influenza type A viruses and attempt to devise a general method for all different subtypes.


Although commercial motion-capture systems have been widely used in various applications, the complex setup limits their application scenarios for ordinary consumers. To overcome the drawbacks of wearability, human posture reconstruction based on a few wearable sensors have been actively studied in recent years. In this paper, we propose a deep-learning-based sparse inertial sensor human posture reconstruction method. This method uses bidirectional recurrent neural network (Bi-RNN) to build an a priori model from a large motion dataset to build human motion, thereby the low-dimensional motion measurements are mapped to whole-body posture. To improve the motion reconstruction performance for specific application scenarios, two fundamental problems in the model construction are investigated: training data selection and sparse sensor placement. The problem of deep-learning training data selection is to select independent and identically distributed (IID) data for a certain scenario from the accumulated imbalanced motion dataset with sufficient information. We formulate the data selection into an optimization problem to obtain continuous and IID data segments, which comply with a small reference dataset collected from the target scenario. A two-step heuristic algorithm is proposed to solve the data selection problem. On the other hand, the optimal sensor placement problem is studied to exploit most information from partial observation of human movement. A method for evaluating the motion information amount of any group of wearable inertial sensors based on mutual information is proposed, and a greedy searching method is adopted to obtain the approximate optimal sensor placement of a given sensor number, so that the maximum motion information and minimum redundancy is achieved. Finally, the human posture reconstruction performance is evaluated with different training data and sensor placement selection methods, and experimental results show that the proposed method takes advantages in both posture reconstruction accuracy and model training time. In the 6 sensors configuration, the posture reconstruction errors of our model for walking, running, and playing basketball are 7.25°, 8.84°, and 14.13°, respectively. Motion capture has a wide range of applications in medical rehabilitation, virtual reality, sports training and other fields ,2,3. HowAt present, the research of motion-capture technology based on sparse inertial measurement is in the preliminary stage. Several motion prior models have been proposed for general motion capture, but the posture reconstruction accuracy is still limited. In addition, the existing research has not studied the influence of the position of the sparse inertial sensor on the reconstruction accuracy. The motivation of the work is to improve the accuracy of the sparse inertial sensor pose reconstruction, especially for certain target application scenarios. We believe that selecting appropriate training data and the optimal sensor placement on the human body are promising means to improve the accuracy of pose reconstruction under specific application scenarios.Current methods mainly work in general purpose motion capture, where a large human motion dataset with various movements is used for model training. The basic assumption of training data for machine learning models is that the collected data should be independent and identically distributed (IID) in the target application scenario . The modIn addition to selecting appropriate training data, the placement of the sparse inertial sensors also plays an important role in the posture reconstruction. It is also worth studied for the optimal performance in different applications. For example, in normal gait, the thigh and shank sensor measurements have strong relevance, and it is possible to infer thigh posture according to the shank posture. In this case, these two sensors have large information redundancy. However, when playing football, the leg movement is highly random in a long period of time, where the independence between the thigh and shank measurements is larger. Generally, the relevance and independence among groups of sensors on the human body vary under different scenarios. The choice of sparse sensor placement will have an obvious impact on the body posture reconstruction accuracy. It is necessary to establish a model of the information volume related to the sensor configuration. With this model, we can determine the optimum number and placement positions of the sensors, and in this way, we can improve the accuracy of motion capture while ensuring the ease of use of the system.A novel optimization-based efficient data selection scheme is proposed to select continuous and IID data with small redundancy from a large dataset for a specific application scenario.The inertial sensor placement problem in sparse inertial sensor posture reconstruction is first formulated into an optimization problem, and a heuristic optimization method is proposed based on a metrics of the amount of information and the redundancy of a group of sensors.In this paper, we first introduce a method to choose appropriate training data from the accumulated dataset, so that the selected data have similar distribution to data collected from the specific application scenario and has small redundancy. Then, the optimum number and placement position of the sparse inertial sensors are determined through information volume evaluation. Finally, the selected training data are used to train a deep-learning neural network, where the inputs are the motion measurements of the selected sensors and the output is the whole-body posture. The body posture reconstruction performance under different training data and sensor placements are evaluated through experiments. In summary, the novelty and contributions of this paper include:The rest of this paper is organized as follows. The inertial motion-capture systems include multiple inertial measurement units (IMUs) installed on various segments of the subject’s body, which can obtain motion measurements including acceleration, angular velocity and posture of body segments. Inertial motion-capture technology is relatively mature, and the device is portable, not affected by motion occlusion, and the data frame rate is high. Roetenberg et al. has proposed an inertial capture system based on Kalman filtering that can integrate 17 IMUs . In ordeStudies have shown that human motion includes a lot of redundant information and can be approximately presented by dimensions lower than the original degree of freedom of human motion ,17,18, wExisting research on human body posture reconstruction based on sparse inertial sensors mainly relies on motion prior data and models, including multilayer neural networks , local pTraining data selection is an active research topic in machine learning. Through effective selection of training data, more informative samples are extracted, redundant samples and noise data are eliminated, to achieve better learning performance . The curThe sampling method is the widely adopted basic strategy to reduce the training set, which is simple, easy and fast. Some methods randomly selected training data according to the spatial nature of the sample ,25,26. ARandom selection of training samples cannot guarantee that the model works reliably in actual tasks. Therefore, scholars proposed to effectively select the training set by considering the distribution characteristics of the data, such as clustering methods. Zhai et al. presented an instance selection method based on supervised clustering, and the main idea is to select instances belonging to inner boundary and outer boundary of clusters . Some reThere are also methods using information theory to evaluate and select data samples. Zheng et al. applied a partial mutual information (PMI) technique to find the optimal dataset . Liu et In existing research of sparse inertial sensors human motion reconstruction, sensor placement was usually manually decided, such as on the pelvis, head, forearms, and shanks . In the Instead of directly comparing the prediction accuracy of the sensor configurations, information-based techniques have been studied. These methods evaluated the value of sensor configurations without obtaining the testing performance in the final task. In this way, the searching efficiency can be improved. Kunze et al. determined the optimal sensor configuration based on mutual information between sensor data and the recognized action type , but thiThe current inertial motion-capture devices require the subject to wear about 17 sensors , and ourW. To express the bone kinematics in W as ..wRootR dy from the measurements of selected sensors x. Huang et al. tried to fit the model to predict the human body Skinned Multi-Person Linear Model (SMPL). SMPL is a parametrized model of 3D human body and pose that takes 72 pose parameters and 10 shape parameters, and returns a mesh with 6890 vertices. Unlike Huang et al. that train the model to predict both pose and shape parameters, we directly predict the body posture. Numbers in brackets in Human body motion pattern has temporal characteristics, and the pose of the current frame is often correlated with history and future motion data. Bidirectional Recurrent Neural Network (Bi-RNN) is suitable to learn the temporal relationship between history and future pose data and the current frame pose , so Bi-RThe inputs of the Bi-RNN are the acceleration vector and orientation in rotation matrix format of selected bones, and the outputs are the acceleration norm and orientation in axis angle format of all other unselected bones. Through comparison of the reconstruction performance using different orientation formats in the Bi-RNN, the combination of rotation matrix input and axis angle output achieves the highest accuracy in the experiments.Conventionally, the entire available dataset is used for deep neural network training. However, there may be redundant data or irrelevant data, which may affect the training performance of the model for specific application scenarios. For example, when learning motion model for walk reconstruction, the motion-capture data of other types of activities, such as run, jump, and dance, are irrelevant and may not contribute to the reconstruction performance of walk. In addition, human walk is a repetitive movement consisting of consecutive similar gait cycles, and a long sequence of normal walk data with consistent pattern may have large redundancy, which will only increase the training time of the model. Therefore, selecting appropriate training data according to the target scenario is desirable to improve the accuracy of pose reconstruction and reduce model training time.D that has similar distribution to the reference data with small redundancy. Then, the selected training data and reference data are used for the Bi-RNN pose reconstruction model training. The training data selection problem can be formulated into an optimization problem as  divergence is a measure of the difference between one probability distribution and another. We use the KL divergence to evaluate the marginal distribution difference between each feature of tance as . Here, Ngiven as  and 8)..D and DRIn this paper, we use information entropy to evaluate the amount of information of the data as . In the given in , which nD is discontinuous and In summary, the cost function of the training data selection optimization problem can be rewritten as . Consideatisfies  will be tisfying . Equatiotisfying  is iteraThe purpose of the sensor selection is to find a sparse sensor configuration that uses fewer sensors and has high body posture reconstruction accuracy. Generally, when selecting sensors, only the amount of information that can be measured by different sensors is considered. However, our objective is to estimate the measurements of the unselected sensors from the measurements of the selected sensors, thus, we need to maximize the correlation between the measurements of the selected sensors and the unselected sensors. Due to correlation between the sensors, the selected sensors will also contain related measurements. Traditional information-based feature selection methods usually only consider the correlation between the selected sensors and the unselected sensors. In this way, the sensors with more information redundancy are usually selected at the same time, and the sensor with less correlation with the unselected sensors but no redundancy is usually ignored. This is actually not preferred.In order that the selected sensors can measure as much body motion information as possible, this paper draws on the idea of Max-Relevance and Min-Redundancy (mRMR) feature selection method in sensor selection . It not All available sensor placements in an inertial motion-capture system are shown in a, b are the number of grids divided in the x, y directions, and B is about 0.6 power of the amount of data.The maximum information coefficient (MIC) proposed by Reshef et al. can evaluate most of the relationships between two sets of data, including linear and nonlinear relationships . CompareSuppose there are sensor position X and sensor position Y, and the dimension of measurement feature of each sensor is s, then the correlation between the two sensor position measurements is defined as . We sum We use Max-Relevance and Min-Redundancy (mRMR) to evaluate the measurement information quality of a sensor group. Then, the purpose of optimal sensor selection is to find a sensor group that has greater correlation with the unselected sensors and less redundancy among the selected sensors. First, we find the first sensor with the greatest correlation with all unselected sensors based on the principle of maximum correlation. Second, we use the principle of maximum correlation and minimum redundancy to find the sensor with the largest correlation with the unselected sensor minus the redundancy with the selected sensor as the next sensor to be chosen. This iteration is repeated until the number of sensors and the accuracy of posture reconstruction meet the requirements.Specifically, the purpose of maximum correlation is to find a sensor group S with the highest correlation with the unselected sensor group C. We add up the correlations between all selected sensors and unselected sensors, and divide by the number of selected and unselected sensors for normalization. The correlation between selected sensors and unselected sensors is defined as . On the A greedy algorithm is adopted to find the optimal solution of function .(19)maxIn the experiments, we collected motion data in daily activities and sports, including walking, running and jumping, playing basketball, football, table tennis, and so on. The entire dataset contains about 800,000 frames of motion data of 4 people with a total length of 2.5 hA total of 20,000 frames of walking, running, and playing basketball are selected as test set, and the rest data are regarded as the training set. On average, the Bi-RNN model was trained on GPU RTX 2080 for about 6 h until convergence.To verify the performance of data selection algorithm, we train the model separately with the entire dataset, manually selected data, and the data selected by the proposed data selection algorithm, and then test in the corresponding motion of the test set. In this case, the sensor configuration is the default 6 sensor configuration including head, forearms, shanks and pelvis. The pose reconstruction error and the number of data frames used for model training are shown in Moreover, the amount of data selected by our data selection algorithm is less than all the data, but higher than the amount of data selected manually. This indicates that in addition to data files with corresponding labels, other data files of other motion types may also contain useful motion data that are similar to target motion type. Our data selection algorithm can extract these effective data segments from different test files across the entire dataset. The smaller amount of data selected by the proposed algorithm significantly reduces the training time of the model to less than 1/3, compared to the entire dataset.To validate the effectiveness of the proposed sensor selection algorithm, we test the performance of the algorithm against some other sensor selection methods using the entire dataset. We compare our method with the modified PCA  for featTo evaluate the performance of combinations of both training data selection and sensor placement selection, we further conducted comprehensively comparison study on 6 sensors configuration of different motion types. The experimental results are shown in The proposed method has a good posture reconstruction performance for regular exercise reconstructed frames, such as simple walking and running. In this work, we propose methods for training data selection and sensor position selection in sparse inertial sensor human posture reconstruction under specific application scenarios. The problem of deep-learning training data selection is formulated into an optimization problem. We present a two-step information-based heuristic data selection algorithm by selecting similar data and optimal data from the entire dataset with respect to the reference data collected from the target scenario. Compared with model trained on the entire dataset and manually selected data, the Bi-RNN model trained on the algorithm-selected data have the better performance and less training time.On the other hand, the sparse sensor position selection is also studied to exploit most information from partial observation of human movement. A greedy algorithm is proposed to search a relative optimum sensor group for maximum motion information and minimum redundancy. Experimental results show that the posture reconstruction performance of the sensor configuration selected by our sensor selection algorithm is better than both PCA and the manually picked default sensor configuration. As the number of sensors increases, the pose reconstruction error of our sensor selection algorithm gradually decreases, which proves that the proposed algorithm takes into account the advantages of sensor measurement information redundancy.Currently, we proposed heuristic algorithms with greedy strategy to obtain approximate solutions of the established optimization problem. In the future, we will further investigate optimization methods for both training data selection and sensor position selection from a theoretical point of view. Moreover, we will also conduct research on the enhancement of deep neural network structure and the integration of motion analysis for sparse inertial sensor posture construction tasks.


Detecting pedestrians in autonomous driving is a safety-critical task, and the decision to avoid a a person has to be made with minimal latency. Multispectral approaches that combine RGB and thermal images are researched extensively, as they make it possible to gain robustness under varying illumination and weather conditions. State-of-the-art solutions employing deep neural networks offer high accuracy of pedestrian detection. However, the literature is short of works that evaluate multispectral pedestrian detection with respect to its feasibility in obstacle avoidance scenarios, taking into account the motion of the vehicle. Therefore, we investigated the real-time neural network detector architecture You Only Look Once, the latest version (YOLOv4), and demonstrate that this detector can be adapted to multispectral pedestrian detection. It can achieve accuracy on par with the state-of-the-art while being highly computationally efficient, thereby supporting low-latency decision making. The results achieved on the KAIST dataset were evaluated from the perspective of automotive applications, where low latency and a low number of false negatives are critical parameters. The middle fusion approach to YOLOv4 in its Tiny variant achieved the best accuracy to computational efficiency trade-off among the evaluated architectures. Reliable real-time detection and recognition of other road users enable safe maneuver planning and execution in autonomous driving. Among those other traffic participants, pedestrians are considered the most vulnerable road users. According to the World Health Organisation (WHO) reports , about hCurrent technologies for considering pedestrians in the design of an autonomous vehicle were recently surveyed . While iWe investigated how the detection frame rate  influences the recall measure in a realistic scenario wherein the goal is to detect a person and brake before a collision occurs. This analysis allowed us to conclude that low latency during detection is a key factor in pedestrian detection. We ought to increase speed of detection algorithms so they can spot pedestrians and initiate safe breaking in time.In the context of a realistic scenario, we investigated five different fusion schemes for multispectral images that were inspired by the state-of-the-art, but are our original contributions to the YOLOv4 architecture. These fusion schemes range from very simple early fusion at the level of image data to elaborated middle and late fusion schemes.As a result of those investigations, we developed a new YOLOv4-based architecture that allows for middle fusion and scored the best on average in the experiments while processing the multispectral images at 35 fps.Being aware of the limited computing resources of autonomous cars, we prepared a lightweight model. This detector exceeds 400 fps on the desktop Nvidia RTX 3080 GPU, provides the lowest latency when detecting vulnerable road users from a moving vehicle, and can be deployed on edge computing devices.We re-visit the general fusion paradigms that were already investigated for pedestrian detection with the Faster R-CNN architecture : early fThe remainder of this article is organized as follows. Related works are reviewed in Traditional object detection techniques use hand-crafted features and their descriptors to determine the properties of objects in an image. Such techniques applied to pedestrian detection on visible light images  have incObject detection algorithms, which use CNNs, fall into two main types: two-stage and single-stage detectors. Typical two-stage detectors come from the region-based convolutional neural network (R-CNN) family. In the Faster R-CNN  version,The most widely used single-stage CNN architectures belong to the You Only Look Once (YOLO) family. The YOLO architecture  treats oThe methodology of multi-sensory fusion using deep learning methods in the context of automotive applications was presented in a recent survey . ImplemeThe first question concerns the sources of information that are most relevant to the task at hand. In the pedestrian detection task, combining regular RGB images and thermal images acquired in the infrared spectrum improves the reliability of detection. The RGB and thermal sensing modalities perform best under different scenarios, with the RGB images yielding precise cues related to visual details, if the lighting conditions permit, whereas thermal images recover cues related to pedestrians in deep shadows and night-time scenes ,31,32.The second question relates to the processing stage when the information is being fused into a coherent structure that makes it possible to decide if a pedestrian is detected or not. In general, the information about pedestrians conveyed in the visible and thermal modalities can be combined at the level of images, or the level of detection, in the form of locations of the bounding boxes produced from two distinct processing pipelines . This leThe last question concerns the ways of performing sensory fusions and is closely related to the transformation of input information inside the network. The standard feature fusion methods include the addition or concatenation of feature maps, but more complicated procedures are also applied. Such methods can exploit additional context information, e.g., produced by semantic segmentation , or weigRecently, multispectral fusion methods for pedestrian detection have been implemented using single-stage neural network detector architectures, such as the Single Shot Detector (SSD)  and the Pedestrian detection for intelligent vehicles requires low latency of information to make steering decisions on time while also performing inferences several times per second to observe the changing environment. Therefore, we selected YOLOv4 as a baseline, state-of-the-art network that meets these requirements. We start the introduction of our approaches with a classical RGB image-based YOLOv4 detector, as described in The main requirements for developing pedestrian detection systems for autonomous vehicles are high recognition accuracy and the ability to operate the system in real-time. Those requirements are fulfilled by the YOLOv4 detector, which belongs to the family of single-stage detectors. Among the key features that distinguish YOLOv4 from earlier versions is the detector architecture, consisting of a backbone, neck, and head as presented in The detector layers that are directly connected to the backbone are called the neck, and for YOLOv4, they take the form of PANet (Path Aggregation Network). Their task is to extract feature maps at various stages of backbone processing, allowing more efficient detection of objects of different sizes. What is more, SPP  is also often included as part of the neck, which extends the receptive field of the detector, increasing its final accuracy. This is possible thanks to performing max-pooling operations with different kernel values for given feature maps. The final components of the YOLOv-4 detector are its heads, which have not changed since the previous version. Their multiplicity allows them to be combined with layers of various feature extraction levels, which helps detect objects of different sizes. Important also is the collection of techniques called Bag of Specials—low computational cost modules for both the backbone and the detector of the YOLOv4 architecture, including the new activation function Mish.RGB cameras are commonly used in scenarios assuming observations in the visible light spectrum. However, these cameras perform poorly in poor lighting conditions, such as during adverse weather conditions or at night. Moreover, poor performance can also be observed during rapid changes in illumination or when a single source of light, e.g., the sun, blinds the camera.Capturing the light spectrum at different wavelengths provides different views of the scene. If we decide to capture information at wavelengths up to 14 μm, we get a thermal camera. Thermal cameras perform their measurements in the infrared spectrum, observing objects’ heat emissions. Thermal images are relatively independent of lighting conditions, making thermal cameras effective at detecting objects that are invisible to RGB cameras.In our experiments, we used the 1-channel, 8-bit thermal data encoding initially provided by the authors of the KAIST dataset, which we normalized with a histogram to use the whole range of 8-bit channel information. The 8-bit thermal data encoding instead of raw 14-bit representation is a known limitation of this dataset . As a rePerforming the fusion of different sensor modalities requires proper alignment of these measurements, understood as finding the correspondence between individual measurements from each sensor. We assumed that the RGB and thermal images are aligned, based either directly on the hardware used to capture the sensory data  or by adEarly fusion approaches focus on combining the RGB and thermal images on the raw data level prior to being processed by the neural network. In our case, it means that the RGB and thermal data were packed into a single, multi-channel image that was passed to the network for further processing. In these approaches, the latent features created in the backbone of YOLO contain information from both cameras from the first computational stage. The main idea is that the input streams are complementary at the low level of processing. The deep learning network can determine which input properties are essential for object detection.The original network architecture of YOLO processes the three-channel image as an input. Therefore, we divide our approaches into subgroups of methods that preprocess the initial data to fit these input types  and a method that modifies the network architecture to take a four-channel image as an input (YOLO4-RGB-T).Providing the RGB and thermal images as a three-channel input means some information must be compressed to fit the input requirements. In the two proposed methods, we convert the image in RGB color space into a different color representation and select two channels containing the essential information for pedestrian detection.In the YOLO4-HST approach, the RGB color space is converted into the HSV , also known as HSB . Then, the HST space is created using the hue, saturation, and thermal components. In this representation, the brightness component is omitted. The YOLO4-HST is trained and evaluated on the input data converted into HST space.In the YOLO4-GST approach, the GST representation consists of the G channel, which is the color image transformed to grayscale, containing the input from all RGB channels; the S channel, which is the saturation from the HSV representation of the color image; and the T channel which is the grayscale thermal image. Exemplary images converted into HST and GST representations are presented in YOLO4-RGB-T fusion is an early fusion approach that modifies the input of the deep learning network. In this approach, the 4-channel image is taken as an input and consists of a 3-channel RGB image and 1-channel thermal information, as presented in The modified architecture of YOLOv4 has an extra input dimension for the first convolutional layer, which raises the total number of parameters of the network when compared to YOLO4-RGB or YOLO4-T. Moreover, direct weight transfer from the original YOLO4-RGB is not feasible. Nevertheless, we decided to use pretrained weights from YOLO4-RGB for all common parts between architectures. To increase the robustness of the learning procedure, we first trained the weights of the modified input layer for the first two epochs with the weights of the remaining layers (frozen). The duration of two epochs was chosen experimentally based on the learning curve to provide a good bootstrap of the network’s weights while avoiding convergence to a local minimum. Once the first two epochs were completed, the weights in the whole network were trained for the remaining 23 epochs. In total, the training process lasted for 25 epochs.The previous section proposed a neural network that performs the fusion scheme at the data input level. An entirely different approach is to perform independent processing of the information coming from RGB and thermal cameras while fusing the processing results. In this fusion approach, called YOLO4-Late and presented in The detection results obtained from both networks are processed by a proposed, non-learnable component, which we call the prediction matching system (PMS). The responsibility of the PMS is to provide a single detection output based on the results obtained from both processing branches, thereby fusing the results of both networks. In YOLO4-Late, the PMS takes as an inputs the value of the intersection over union (IoU) parameter for bounding boxes for the outputs of both neural networks and the probability that a detected object belongs to a given class calculated by each network. In the proposed implementation, the PMS follows a preset rule that reports detection of an object if the IoU value for the predictions from both detectors is greater than 0.6 or if the probability of detection for either of the networks is greater than 0.45. The provided thresholds for the IoU value and probability detection were determined with a simple exhaustive search on the training data to provide the best mAP measure.The last proposed method, YOLO4-Middle, performs the fusion on the feature-level processing of the input. In this configuration, the input is a 6-channel tensor input, where three channels are filled with RGB image data, and three channels contain the thermal image. In the beginning, the processing is performed independently for RGB and thermal inputs using the CSPDarknet53 neural network—see the initial feature processing presented in The feature maps obtained from processing RGB and thermal inputs are concatenated and then processed using convolutional layers with a kernel size of 1 × 1. This operation reduces the depth of the feature maps, making it possible to keep the remaining processing layers of the YOLOv4 detector architecture intact compared to processing with a single sensory input.The proposed architecture of YOLO4-Middle was initialized with original pretrained weights for the YOLO4-RGB detector for all the processing blocks that are the same as those of the original pipeline. However, similarly to YOLO4-RGB-T, some components differ, and the weights had to be trained without initial guesses. Therefore, the neural network training was split into two stages—a warm-up and the main training. The warm-up lasted for two epochs and was used to determine the initial values of weights that could not be initialized with transfer learning, i.e., for new convolutional layers that combined the output feature maps from both sources. The transferred weights were frozen during the warm-up stage of training. The main training started from the third epoch when all the weights in the network were trained.An experimental evaluation of the proposed networks that fuse the RGB and thermal inputs in pedestrian detection requires a dataset that recorded from a moving car that contains aligned images from both sensor modalities. The KAIST dataset  meets thThe dataset consists of over 60,000 images recorded at different times of the day. The authors of the dataset divided these sequences into day and night parts. Each recording has object annotations defined by the authors of the dataset, which can be used during training and to evaluate the proposed system. The ground truth labels are available for three classes: a person, a group of people, and a cyclist. In our approach, we merged all of these classes into a single class “person”, as the dataset was not balanced, and each annotated class represents a person in the end.Each sequence in the KAIST dataset was used only during the training, validation, or testing phase, which avoided the overlap between data. This approach ensured that the detector would not know the test data, allowing for an unbiased prediction of its accuracy. The typical way of measuring the detection performance is based on the mean average precision (mAP). The mAP metric determines how many of the predictions turned out to be correct and what was the overlap between the detected and ground truth locations of the objects. The mAP is therefore useful for comparing the performances of different detection algorithms. Furthermore, the values of mAP are strongly connected with the preset threshold that determines the overlapping of the predicted bounding boxes with the ground truth ones. The overlap threshold is denoted by the Intersection over Union (IoU) value. The lower the IoU is, the lower overlap is needed to be considered correct, increasing the number of detections that are assumed to be accurate. In our case, we computed mAP values for IoU thresholds of 0.15 From this visual representation of the mAP values for all IoU thresholds, the highest metric scores were reached by the YOLO4-Middle detector, which combines extracted information from two sources at different processing levels. Slightly worse results were achieved by the YOLO4-Late detector, which combines the detection results from parallel processing pipelines for RGB and thermal images.Marginally worse results can be observed for the YOLO4-GST and YOLO4-HST fusions, which barely outperformed the detector processing only the thermal images (YOLO4-T). Despite using all available information, similar but slightly lower performance than for YOLO4-T can be observed for YOLO4-RGB-T. We believe it might have been caused by a partial inability to use weight transfer learning to boost the initial training of the network for YOLO4-RGB-T. The worst mAP metric values were obtained by the YOLO4-RGB detector, proving the importance of performing data fusion for data from RGB and thermal domains.The ranking of the considered approaches is almost the same regardless of the chosen IoU threshold. However, a single IoU threshold value has to be chosen during the operation. To select this value, we decided to analyze the plot of recall and false positives depending on the IoU threshold for YOLO4-Middle, as presented in As previously seen in The remaining fusion approaches had similar performances for day and night conditions, showing that fusion methods can perform well regardless of these conditions, even though YOLO4-RGB performs very poorly in night conditions.The results show that the YOLO4-Middle architecture was the best performing network among the verified solutions according to the mAP measure. A single mAP value is insufficient to determine the method’s viability to be deployed in real-world scenarios. Therefore, we decided to additionally analyze the performances of the presented approaches based on the ground truth pixel sizes of the objects. The ground truth pixel size of an object is defined as the sum of pixels that were used to mark the object in the dataset on the particular images. Then, we computed the mAP performance of the network for all images in the testing part of the dataset for a considered bounding box size. The In https://github.com/NVIDIA/TensorRT accessed on 20 November 2021).In automotive applications, we are interested in obtaining the best possible results considering real-world scenarios. While the vehicle is in motion, the system has a limited time budget to perform efficient detection and decide, i.e., to perform emergency braking. The obtained processing fps values for the analyzed fusion models are presented in After optimizing neural network models, the processing times for all analyzed methods dropped below 30 ms for a single image. The longest inference time was observed for YOLO4-Middle, which has additional connections between processing blocks for RGB and thermal pipelines, leading to a long processing time. As it turns out, the middle fusion architecture provides the best results concerning the accuracy of pedestrian detection in both day and night scenarios. Still, it is slower than the completely independent processing pipelines for both modalities for the YOLO4-Late approach. Although the achieved speed of 35 fps satisfies our assumptions of low latency, as we can process the incoming images essentially at the fps of a typical RGB camera, this was achieved using the total processing power of the Nvidia RTX 3080 GPU.Considering the speed versus accuracy trade-off, we decided to extend our investigations to the tiny-YOLO architecture, which is known for its superb computational efficiency. We considered our YOLO4-Middle fusion scheme as the baseline for further development, as it performed well for the day, night, and smaller objects. We created the YOLO4-Tiny-Middle version starting from this fusion scheme and considering the YOLO-Tiny backbone architecture . CompareThe average Although the main aim of our research was to investigate how the fps of a deep learning detector influences its ability to detect a vulnerable road user within a sufficient distance to allow for breaking. We also draw brief comparisons between the performance of our best model, YOLO4-Middle, and other YOLO-based detectors that were published recently. While most of the works concerning multispectral pedestrian detection adopt the log average miss rate over the range of  dataset  is used  dataset  and knowMultispectral Channel Feature Fusion from  scored wWe assumed that each detected person object represents only one person of consistent height and width;We assumed that each object detection was independent;We assumed that the detection rate was not limited by the camera’s frame rate;We focused on recall without the consideration for false positives.An analysis of the practical viability of the proposed pedestrian detection method has to capture the behavior of the system in typical driving scenarios. Measuring the performance of the pedestrian detector with mean average precision does not illustrate how well the system would perform once deployed in an actual autonomous car. Therefore, we decided to simulate a typical scenario when a person appears on the road, and the car has to either brake or has to perform an avoidance maneuver to avoid a collision. In this scenario, we can estimate the performance of the pedestrian detection system depending on the inference time needed to process a single image, the speed of the vehicle, and the distance to the detected object. Unfortunately, the KAIST dataset lacks the depth information to infer the performance based on the distance to the obstacle and vehicle speed for each image. In order to do our simulation, we had to make several assumptions:Therefore, the following analysis is only valid if these assumptions are held and do not cover the infinite number of cases that might occur while driving.The following processing step of our simulation involved estimating the relation between the size of the bounding box measured in pixels and a metric distance to the obstacle. In our simulation, we assumed the supposed worldwide average height of an adult, approximately 1.7 m, and the average width of 0.4 m . For simw is the width of the image in pixels. Our thermal and RGB cameras have isotropic lenses and can be modeled with a pinhole camera model to approximate the same focal length b is the size of the bounding box measured in pixels. For the KAIST dataset, we found out that the pedestrian detections occur for bounding boxes b ranging from 50 to 5000 pixels.With the knowledge that the vertical field of view of the camera was equal to b, we were able to roughly estimate the distance from the camera to the pedestrian. The relation between the bounding box size b and the distance to the pedestrian d is presented as a blue line in Based on the prior assumptions, for a selected bounding box size TP is the sum of all true positives defined as correct detection with an IoU measure above the defined threshold of 0.50, and FN is the sum of all false negatives defined as the sum of detections that should have been but were not made by the proposed algorithm.This section reports the performance of the method based on the distance. In our simulation, we assumed that the person was standing on the road, and the goal was to detect the person and brake before the collision occurred. Therefore, we were interested in ensuring that the person was consistently detected , ignoring the number of false positives at this time. The recall measure was computed as:We computed the recall measure for each analyzed fusion method based on ground truth and estimated bounding boxes. Like the previous computations, we did them for different sizes of the ground truth bounding boxes used to determine the recall depending on the distance to the detected objects. The results for day and night conditions are presented in None of the methods could enable detection at distances exceeding 28 m. For all methods, the average recall dropped almost linearly with the distance of the person from the camera. The best recalls were found for YOLO4-Middle for almost all considered distances for the day and the night. For daytime, the worst performance for more considerable distances was observed for YOLO4-Tiny-Middle, which struggle with recognition of small bounding boxes. The worst performance was the nighttime detection by YOLO4-RGB, which performed poorly for distances larger than approximately 10 m.i-th interval that ranges for metric distances v and the fps of the chosen detector f. Then, we were able to compute the probability of detecting a person in the selected interval of bounding box sizes (distances) with index i, using equation:Based on the computed recalls and chosen vehicle speed, we were interested in a simulation to determine the safe driving speed with which a pedestrian detection system could initiate the emergency braking procedure and avoid a collision. More precisely, for a selected vehicle speed, we were interested in finding the distance from the sensors where our system produces correct pedestrian detection with the required 0.99 probability. The further the detection distance, the better the method performs, as the system has more time to break or perform an avoidance maneuver. We assumed that each image processing was independent during our simulation, and started off by dividing the feasible detection ranges into intervals of 50 pixels, defined by bounding box sizes, with objects sizes of 0 px  observed for YOLO4-Tiny-Middle. During the day, YOLO4-Tiny-Middle can safely brake when driving at a speed of approximately 65 km/h, and the other solutions could work safely at about 60 km/h. The more significant differences between the methods can be observed for the nighttime attempts: YOLO4-Tiny-Middle performed best and YOLO4-RGB worst. The difference in speed for when we can avoid a collision using these two methods is approximately 11 km/h (62 vs. 51 km/h). Naturally, the person could be detected with the required probability from a greater distance (greater detection distance) with lower vehicle speeds by all considered methods.r is the recall for a single detection, and t milliseconds. This measure captures the recall of the method in the chosen time interval, assuming independence for all of the detections. For our further considerations, we chose The lack of clear visual distinction between presented methods suggests that despite the differences in recall performance, the amount of fps is the critical feature determining the performance of a solution. This proves that YOLO4-Tiny’s short inference time and high fps make it an excellent algorithm for pedestrian detection. Based on this observation, we also propose introducing a performance measure that combines the recall of a method with the feasible fps. We call this measure the accumulated recall over time:It can be seen that due to the high speed (above 35 fps) of all considered methods, their average recall performances are quite similar, with the exception of YOLO4-Tiny-Middle. Despite lower mean average precision, the high fps makes it the best method during the day. The best performance during the night was by YOLO4-Tiny-Middle; clearly worse performance was obtained for YOLO4-RGB  or both modalities , we determined that the fusion performed in the middle stage of the YOLO detector processing pipeline (YOLO4-Middle) provides the best performance during the day and at night, which corroborates earlier results from the literature . The YOLMeasuring the mAP and/or log average miss rate scores on a public dataset is usually the final step in other works concerning this problem. We went further, by proving that YOLO4-Middle can process over 35 images per second on the desktop RTX 3080 GPU. However, as automotive applications call for lower-end or embedded GPUs, we aimed to further improve the processing speed with YOLO4-Tiny-Middle, a lightweight implementation of our best YOLOv4-based fusion architecture. Although with this architecture we achieved slightly lower Measuring the mAP and fps is not enough to verify a solution for real autonomous cars. Therefore, we measured the performance in a simulated scenario in which a pedestrian emerged in front of the car, and the system had to brake to avoid a collision. From this analysis on the KAIST dataset, it can be seen that high-fps detectors outperformed their more classic counterparts, due to their ability to perform more frame processing passes within the same time window. We also proposed an “accumulated recall” measure that jointly captures recall and fps based on this information. Accumulated recall is suited to capturing the real-world feasibility of pedestrian detectors. Hence, we argue that this new measure can be used to assess the readiness of detectors when product deployment is considered.Our experiments proved that thermal and visible light images can be fused in the modern YOLOv4 architecture without additional overheads in processing, to achieve very high fps processing, as long as both images are synchronized and aligned. These assumptions are somewhat limiting, as no out-of-the-box co-aligned RGB and thermal cameras are available for automotive applications. Therefore, we see two exciting branches for further research. The first will focus on utilizing data streams from both modalities when the images are not co-aligned, making a practical solution when two separate cameras are combined in a single sensory setup. The other direction of future research will investigate the detection of small objects in images and the addition of preprocessed LiDAR data  for bett


Finite quantum many fermion systems are essential for our current understanding of Nature. They are at the core of molecular, atomic, and nuclear physics. In recent years, the application of information and complexity measures to the study of diverse types of many-fermion systems has opened a line of research that elucidates new aspects of the structure and behavior of this class of physical systems. In this work we explore the main features of information and information-based complexity indicators in exactly soluble many-fermion models of the Lipkin kind. Models of this kind have been extremely useful in shedding light on the intricacies of quantum many body physics. Models of the Lipkin kind play, for finite systems, a role similar to the one played by the celebrated Hubbard model of solid state physics. We consider two many fermion systems and show how their differences can be best appreciated by recourse to information theoretic tools. We appeal to information measures as tools to compare the structural details of different fermion systems. We will discover that few fermion systems are endowed by a much larger complexity-degree than many fermion ones. The same happens with the coupling-constants strengths. Complexity augments as they decrease, without reaching zero. Also, the behavior of the two lowest lying energy states are crucial in evaluating the system’s complexity. The study of finite many-fermion systems has been enriched in recent years with the incorporation of new mathematical tools inspired in information theory. In particular, information measures and information-based complexity measures have been successfully applied to elucidate various aspects of the physics of atoms, molecules, and atomic nuclei ,12,13,14Encouraged by the above results we revisit here a similar but not identical scenario to deal with two different exactly solvable (interacting) finite fermions-model of the Lipkin kind , with N in D for our two models  are the ones exhibiting the most varied features and rich chemical properties.A remarkable benefit of the information-theoretic approach to physics is that it suggests new connections between apparently unrelated problems. For instance, information-theoretic ideas based on Fisher information helped to establish interesting links between Schroedinger’s equation and Boltzmann equation . We hopeWe intend to deal with SU(N) symmetries in a future effort.


Motor bearings are one of the most critical components in rotating machinery. Envelope demodulation analysis has been widely used to demodulate bearing vibration signals to extract bearing defect frequency components but one of the main challenges is to accurately locate the major fault-induced frequency band with a high signal-to-noise ratio (SNR) for demodulation. Hence, an enhanced fault detection method combining the maximal overlap discrete wavelet packet transform (MODWPT) and the Teager energy adaptive spectral kurtosis (TEASK) denoising algorithms is proposed for identifying the weak periodic impulses. The Teager energy power spectrum (TEPS) defines the sparse representation of the filtered signals of the MODWPT in the frequency domain via the Teager energy operator (TEO); the TEASK helps determine the most informative frequency band for demodulation. The methodology is compared in terms of performance with the fast Kurtogram and the Autogram methods. The simulation and practical application examples have shown that the proposed MODWPT-TEASK method outperforms the above two methods in diagnosing defects of motor bearings. Rolling element bearings are one of the most critical supporting components in rotating electrical machinery due to the fact that approximately 41–42% of induction machine failures arise from faulty bearings [Vibration signals measured from bearings usually contain rich information on the machine condition and have been widely used for bearing fault diagnosis . When a When impacts induced by rolling bearings, they will excite the resonance of bearing elements and housing structure, which causes the modulation of vibration signals in the high frequency bands. Envelope demodulation has been widely used to demodulate vibration signals to extract bearing characteristic defect frequencies, which strongly require vibration data with high signal-to-noise ratio (SNR) ,4. To enTo address this issue, Duan et al.  proposedRecently, numerous methods have been proposed to filter the signal for noise reduction before envelope analysis to improve the SNR. One of the first systematic approaches based on kurtosis to find the optimal frequency band before envelope demodulation was undertaken by Antoni and Randall ,8. The KIn order to obtain a higher SNR and enhance fault features in the presence of strong background noise, a number of improved Kurtogram-based methods have been proposed by many researchers. For example, Barszcz and Jablonski  developeAlthough the Hilbert transform (HT) has been the most widely used for bearing fault demodulation in recent years, the HT of noise-corrupted signals usually produces spurious amplitudes at negative frequency . MoreoveIt has been apparent from the studies mentioned above that these SK-based techniques are capable of yielding valid results under certain conditions but some cases in which these methods are not always effective in identifying the bearing optimal frequency band due to the inclusion of other components, such as strong background noise, interference from other vibration components, and non-stationary slippage caused by rolling elements. For the further improvement in the accuracy and efficiency of the algorithm in determining optimal frequency bands and implementing demodulation, this study introduces a new indicator named the TEASK for bearings diagnostics. The TEASK includes the following processes: the MODWPT was first used to decompose the vibration signal for noise reduction, then each sub-band after decomposition was demodulated by the TEPS via TEO. Finally, the kurtosis values of the overlapped and segmented TEPS sequences were calculated to provide a quantized TEASK profile for each frequency band with a bandwidth and a center frequency. The methodologies were validated on simulated and experimental data. Furthermore, the performance of the methodology was compared with the FK and the Autogram methods for analysis of the same signals.(1)By integrating the MODWPT with TEPS, a new bearing detection tool, named the MODWPT-TEASK, for the analysis of vibration signals resulting from bearings with localized defects is presented, which can be used for the diagnosis of single and early weak faults.(2)The MODWPT-based sub-band analysis is used to analyze the bearing vibration signals and extracts the characteristic features of various bearing faults. The sparse representations of fault features of small incipient bearing defects are obtained by calculating the TEPS of each sub-band signal at different decomposition levels via TEO, thereby enhancing the accuracy of bearing fault detection.(3)The TEASK dynamically measures the protrusion of the sparse representation to highlight bearing fault characteristic frequency and its harmonics, thereby providing accurate detection results.(4)The proposed fault detection method based on the MODWPT-TEASK can effectively detect the very weak fault characteristics of bearings under very low SNR. The MODWPT-TEASK also detects bearing fault more accurately than two state-of-the-art methods (FK and Autogram).The novelty and main contributions of this paper can be summarized as follows:The rest of the paper is outlined as follows. In The MODWPT is an enhanced version of the discrete wavelet transform (DWT). Compared to the DWT, the MODWPT is undecimated and each decomposition level has the same number of wavelet coefficients, thus ensuring that it contains all the essential bearing vibration information and possessing the same circular shift properties to maintain a consistent cycle time between two consecutive fault-induced impulses at different nodes. Moreover, the MODWPT is an energy-conserving transformation, that is, the sum of the total energies of the MODWPT coefficients equals the energy of the signal. The MODWPT algorithm is briefly described below, while the detailed information can be found in ,27.M is the length of filter and Let us consider a discrete time sequence A scaling of the MODWPT filters is required to conserve energy and their filters can be given by j of the transform, At the first level, n Ψc[saBy combining Equations  and 9),,9), the It can be seen from Equation  that theTo take advantage of the capabilities provided by the TEO, this technique is used to demodulate each wavelet coefficient, accuracy ,31. ThenAccording to the abilities to extract fault signatures, the TES outperforms the envelope spectrum ; therefoIt is demonstrated that the envelope power spectrum of the signals can provide a good sparse representation of bearing fault signals and enhance the fault characteristic frequency effectively ,33. AccoQ is the shift samples at a time under the window Then a rectangular sliding window R is the number of segments and the largest integer satisfies Moreover, the process of merging the overlapped and shifted frequency segments can be expressed asThe spectral kurtosis (SK) is a spectral statistical indicator which can detect not only the transient impulses in the presence of strong background noise but also reveal the impulsive components in frequency domain ,8. MoreoIt is noted that the Equation , the SK The flowchart of the proposed method is shown in Preprocess the digitisation of the original vibration signal measured by an accelerometer to obtain the normalized signal J using Equation  used in ,38 were The above simulation example indicates that the proposed method achieves higher performance, allowing to effectively extract fault signatures with a low SNR, when compared to the FK and the Autogram methods.To verify the effectiveness of the proposed method in practical applications, we established a simplified bearing test rig, as shown in The type of test bearing is NTN 6206ZZ, and parameters of the bearings are listed in One of the main goals of this experiment was to detect the weak periodic impulses generated by small defects of the test bearings and then identify the bearing characteristic frequency components using the proposed method. Moreover, the bearing conditions under the motor operating at 100% full-load were presented because the majority of motors spend a higher percentage of time under high-load conditions in many real-life applications.When the motor was operated full-load at a speed of 1741 rpm (To verify the effectiveness of the proposed method in bearing fault detection, the FK and the Autogram methods were used to analyze the same experimental bearing vibration signal for comparisons. The plot of the FK for the faulty outer-race bearing signal is shown in The proposed method was next applied to the same faulty outer-race bearing signal. The TEASK plot is shown in For the faulty outer-race bearing signal under full-load, the proposed method is superior to the other two methods in interference suppression, the SNR enhancement and characteristic frequency components identification. So the proposed method performs better than the FK and the Autogram methods in bearing outer-race defect detection.For this test, the shaft rotational frequency The time-domain waveform of the acceleration vibration signal form a test bearing with an inner-race fault under full-load condition is shown in For comparison, both the FK and the Autogram methods were applied to the same bearing faulty inner-race signal. In The Autogram was then used to analyse the same inner-race fault signal. The paving of the Autogram is displayed in Finally, the proposed method was applied to the same inner-race fault signal. The plot of the TEASK is shown in p = q = A is the amplitude.It is well known that bearing signals are usually modulated by other sources such as shaft frequency. This phenomenon can be observed in rding to , the vibdulation . The ampThe occurrence of sidebands is evident for the strong non-linear relationship between the shaft rotational frequency and the inner-race fault characteristic frequency. This confirms that the proposed method can successfully detect a small detect in the bearing’s inner-race.Compared with the FK-based and Autogram-based methods to demodulate the filtered signals using Hilbert square envelope demodulation technique see b,d, the In contrast to the diagnosis of bearing inner-race fault, it can be clearly seen from From the comparison of experimental results, the proposed method has better demodulation performance and faults extraction ability than the FK-SES and the Autogram-SES in the presence of other vibration interferences.This paper proposes a new bearing diagnostic method to determine the fault sensitive frequency bands, where the weak periodic impulses can be accurately revealed and identified in the presence of strong noise. At first, the MODWPT was used as a filter to decompose a signal into different frequency bands and central frequencies. Second, the TEPS of the signals filtered by MODWPT at each node was calculated to enhance the SNR of the spectrum and sharpen the spectral peaks to extract the weak impulses generated by a defective bearing. Third, an adaptive spectral kurtosis was adopted and then computed by combining the overlapped and shifted TEPS segments at each node to construct a two-dimensional color map. From the paving of the TEASK, the sub-band signal with the highest kurtosis is selected as the optimal frequency band for demodulation. In analyzing the simulated signal corrupted by heavy noise, the proposed method has a better ability to detect fault frequencies even though the SNR is very low. Moreover, the proposed method is able to find out the optimal resonant frequency band to locate the center frequency and the bandwidth at different nodes; however, the FK and the Autogram methods used in this study fail to diagnose the same simulated signals under heavy noise. The proposed method has been further tested on real bearing fault vibration signals obtained in the laboratory and the bearing faults are outer-race and inner-race. In terms of demodulation performance, the proposed method has been compared with the FK-based and the Autogram-based demodulation methods. The experimental results demonstrate that the proposed method achieves higher performance and is effective to extract and separate weak fault-related signatures from signals with heavy interferences. The main limitations identified during the presented research are the selection of the mother wavelet function for identifying the given types of faults and the number segments for the computation of the TEASK to determine the optimal frequency band. Although the proposed MODWPT-TEASK approach has been applied in this paper to the diagnosis of the single fault in ball bearing operating in steady state, it can be extended to diagnose compound faults of bearing under non-stationary operating conditions. In addition, the applicability of the TEASK index will be considered for the multi-component amplitude and frequency demodulation.


In this work, we discuss the idea and practical implementation of an integrated photonic circuit-based interrogator of fiber Bragg grating (FBG) sensors dedicated to monitoring the condition of the patients exposed to Magnetic Resonance Imaging (MRI) diagnosis. The presented solution is based on an Arrayed Waveguide Grating (AWG) demultiplexer fabricated in generic indium phosphide technology. We demonstrate the consecutive steps of development of the device from design to demonstrator version of the system with confirmed functionality of monitoring the respiratory rate of the patient. The results, compared to those obtained using commercially available bulk interrogator, confirmed both the general concept and proper operation of the device. In contemporary medicine, there is an obvious, clearly visible trend in the continuous improvement of the quality of medical care by introducing various advanced technology-based diagnostic, monitoring, and life-support systems, which may improve the quality and efficiency of diagnosis, therapy, and patient care. To enumerate a few of these–a sophisticated and expensive medical equipment deployed in intensive care units for vital signs monitoring  should be mentioned, together with advanced (and even more costly) diagnostic tools like ultrasound devices, computer tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), etc. The omnipresence of advanced technologies is evident. However, in typical everyday patient care, these are not commonly deployed, due to the lack of comfort, high cost, dimensions, limited availability of appropriate devices or sometimes distrust of modern technologies. Typically, numerous nursing staff takes care of the patients, periodically monitoring their parameters like temperature, blood pressure, blood oxygen saturation level, heart rate (HR), respiratory rate (RR), etc. Such monitoring is necessary, in the case of in-patients where there is also a number of medical examinations during which continuous monitoring of patient’s vital signs is highly desired, and specifically when the general condition of the patient is unstable or unknown . An example might be an MRI scanning procedure, a non-invasive technology that enables obtaining three-dimensional anatomical images of exceptional resolution by the use of detecting the changes of protons energy in the presence of strong magnetic field and radiofrequency waves pulsed through the patient.During such an examination, patients may experience discomfort due to a claustrophobic environment of the MRI measuring chamber, which typically has a form of a relatively tight tube, with the bed for the patient in the center. The dimensions and shape of the chamber might result in anxiety, claustrophobia, and panic attacks, specifically when the scan requires several or a few tens of minutes to be taken properly. This can potentially lead even to deterioration of the patient’s health. It should also be noted that any patient’s movements  inside the capsule might worsen the image quality or result in the necessity of a repeated examination. Therefore, monitoring the patient’s condition is highly desirable–the symptoms of increasing stress can be detected based on increased heart rate and/or respiratory rate, thus giving feedback to the medical personnel, who can react accordingly. The monitoring is even more important in the case of unconscious patients, who are not able to provide the staff with information on their actual condition.Due to the presence of a strong electromagnetic field inside the MRI chamber, inherent to this imaging technique, it is impossible to use standard electronic sensors for monitoring the patient’s condition. Typically, MRI-certified electronic monitors are used, carefully shielded from the electromagnetic fields. An interesting alternative, however, may be offered by optical sensors, and specifically fiber-optic sensing elements like fiber Bragg gratings (FBGs) ,2. The iThe potential of the InP-based integration technology has been used to develop an integrated spectrometer device, comprising a wavelength demultiplexer, an array of photodiodes, and an optical amplifier, the combination of which enables effective interrogation of a fiber-optic sensor system based on Bragg gratings. In this work, the ASPIC design and the obtained characterization results are presented. System-level test results are presented and discussed with respect to the interrogation of a single FBG, as well as to detect the respiratory rate of a person lying on a bed with a mounted sensor plate.The monitoring system consists of two major parts–a network of FBG sensors integrated with MRI bed overlay and an interrogator located outside the measurement chamber connected to the sensors using a standard single-mode fiber. The system is depicted in The heart of the system is the interrogating device, which is responsible for the proper detection of signals reflected from FBG sensors. Considering the intended use of the device, it should be highly reliable, preferably compact and cost-optimized. All the above-defined properties are characteristic of photonic integrated circuits (PICs). Therefore the investigations described in the following part were focused on this specific approach.The commercially available devices usually exploit a read-out scheme basing on a broadband optical signal, e.g., from a superluminescent diode (SLED) source, launched to the sensors network, reflected, and analyzed with an optical spectrometer. In the alternative approach, a tunable monochromatic light is used, while the reflected signal is monitored with a single broadband photodetector. This enables the reconstruction of the FBG spectrum by analyzing the recorded signal in the time domain. In this work, the first approach was implemented. A general concept of the readout system for an FBG sensor network based on a broadband light source and an integrated wavelength filter–a demultiplexer with dedicated photodiodes is shown schematically in To miniaturize the system shown in Both interrogators were designed as 36-channel devices, differing only in channel spacing, which was 50 GHz (which corresponds to 0.4 nm) and 75 GHz (0.6 nm), respectively. The free spectral range (FSR) of AWG was chosen as the channel spacing multiplied by the number of the AWG channels. Therefore, a 50 GHz channel spaced AWG was designed with an FSR of 14.4 nm and a 75 GHz one with an FSR of 21.6 nm. The chosen channel spacings of AWG were a trade-off between the readout device resolution and reasonably small chip size. The layout of the chip and its photograph are shown in Both designed interrogators share the same general layout scheme. Each of them is equipped with two optical inputs, using angled waveguides. Out of two inputs the one is directly coupled to the AWG and the other one is connected via a build in semiconductor optical amplifier (SOA) to increase the power level of the input signal. The output waveguides of the AWG are connected to an array of PIN photodiodes. The layout is compatible with a CORDON generic packaging scheme, having the electrical interface at the bottom, the top and the right side of the chip, and optical inputs/outputs at the left side see .Fabricated circuits were carefully characterized and tested with respect to the design parameters. The experimental setup consisted of a tunable laser source covering the spectral range 1545–1570 nm, a 3-D translation stage enabling precise positioning of a tapered optical fiber against the input waveguide, a chip temperature control unit, and a source measure unit (SMU) for driving the SOA, biasing the photodiodes and measuring their photocurrents. The tunable laser and PIC were connected using polarization-maintaining (PM) optical fibers.The recorded transmission spectrum of 50 GHz AWG is presented on the left side of The measured values of FSR and channel spacing (average 14.4 nm and 0.4 nm respectively) are in good agreement with the design values. The device exhibits proper operation as a WDM demultiplexer having crosstalk better than 17 dB. For AWG’s central wavelength (λc = 1561.6 nm) the optical power should reach its maximal value. The side channels were expected to have lower peak values according to Gaussian distribution and this tendency is clearly seen in The use of a built-in SOA allows increasing the level of the photodiode current signal by 10 dB while maintaining other parameters of the AWG transmission spectrum unchanged. Due to the smaller channel spacing of the 50 GHz AWG interrogator, and hence the better resolution of the reconstruction of the FBG reflection spectrum, this particular AWG was selected for the subsequent work. As the signal level in the interrogator without the use of SOA was satisfactory for this particular application, the SOA has not been used in the following experiments.To evaluate the operation of the interrogator in conditions equivalent to operational, a measurement system shown in The SLED (Dense Light DL-BP1-CS5169A) equipped with a built-in optical circulator was used as the broadband light source. The single mode fiber (SMF)-based (type SMF-28) FBG single sensor/network of sensors was connected to one output of the light source, while the other port was connected via polarization control elements to the integrated interrogator (denoted as “device under test”-DUT) and to a reference device (REF)-a commercially available FBG interrogator (Ibsen I-MON 256 USB) or an optical spectrum analyzer . The photodiodes of the integrated interrogator were connected to a source measurement unit , which was used to provide reverse bias to the diodes and monitor photocurrent.In our previous work , we haveIn the first experiment, the possibility of properly reconstructing the shape of the FBG wavelength response was evaluated. An FBG with FWHM equal to 3.0 nm was used, which is significantly larger than the value of the channel spacing of the AWG (50 GHz/0.4 nm). The experiment was conducted by reading out the response at 14 consecutive AWG channels and comparing the result with the reflection spectrum collected with an optical spectrum analyzer. The obtained spectra are shown in The measurement results confirm the possibility of the proper reconstruction of the FBG reflection spectrum with an AWG-based interrogator. The shape of the FBG response collected with the OSA corresponds well to the received signal of consecutive PIC outputs, however, a small shift (of the order of 0.5 nm) towards longer wavelengths is clearly seen. Such a shift, resulting from the technological imperfections of AWG is not critical as in principle the interrogator does not monitor the exact value of the wavelength, but its relative change. It is also noticeable that the signal-to-noise ratio (SNR) collected with OSA is about 15 dB larger than SNR in the case of the signal collected with an AWG. The smaller SNR measured with AWG is caused by the inter-channel crosstalk of AWG. As the AWG channel spacing (0.4 nm) is significantly larger than the wavelength scan step of the OSA (0.01 nm), each measurement point collected with AWG can be considered as averaging of 40 neighboring points collected with OSA. Consequently, the FBG response measured with AWG is smoother than the one collected with OSA.The interaction of an AWG-based interrogator with a destined 0.7 nm FWHM FBG was evaluated in the experiment, where the 27 cm long section of SMF with an FBG inscribed was mounted in a translation stage allowing fiber tensioning and loosening with a precision better than 50 µm. At the beginning of the measurement cycle, the fiber was completely loose. It was subsequently tensioned step by step until the risk of fiber break occurred. The fiber was then relaxed until completely loosen. Then, the cycle was repeated. Finally, in order to determine the dark current of each photodiode, the optical signal was switched off completely, and the dark current was recorded.With the elongation of the FBG, its wavelength response shifts towards longer wavelengths. Consequently, the optical signal coupled to the AWG-based interrogator is redirected from one interrogator receiving diode to a neighboring one, accordingly with longer wavelengths reflected from FBG. This phenomenon is depicted in During the cycle, the results of which are shown in Having confirmed the general properties and proper response to mechanical stress applied to an FBG sensor, the more sophisticated version of the system has been developed, and first attempts to obtain a pattern of human breathing have been made. The developed system, designed and configured analogous to the setup shown in The dedicated electronic driver, shown in To detect respiratory rate a fiber Bragg grating was mounted on a polymer board, which was placed between the patient’s back and the mattress, similar to . This soPrior to starting the experiments with recording human breathing, the influence of the temperature of the human body has been analyzed. To distinguish the impact of the temperature on the detected signals an experiment with a heating pad placed on a sensing plate was performed, the results of which are depicted in At the beginning of the measurement, the FBG plate was conditioned for 5 min at room temperature. After that, a heating pad filled with water of a temperature of 35.7 °C was placed on the sensor plate, which resulted in heating the sensors, and in turn, in a continuous shift of the Bragg wavelength towards longer wavelengths. After around 20 min the temperature of the FBG sensors (which went up during this time) and the heating pad (which went down) equalizes. Then, as the heating pad temperature consecutively decreases, the FBG temperature begins to decrease, so the Bragg wavelength starts to shift towards shorter wavelengths. At the 57th min of the measurement, the heating pad was removed from the sensor plate, which then started to cool down more quickly. Such changes of the temperature can be clearly visible in the time domain characteristic, presented in The analysis of the response of the photodiodes is more complex. Signals from diodes F and I, corresponding to the shortest wavelengths decrease, due to the shifting of the FBG response towards longer wavelengths. With FBG plate warming, the adjacent photodiodes  signals are reaching their maxima successively. Diode Q, which is the next after P, detects some signal change, but due to its malfunction, is responding only partially. The next diodes  also receive minor voltage changes consistent with temperature increase, but the FBG response is too narrow to have a considerable impact on their signal. After the FBG reaches the maximum temperature (around 20th min) and starts to cool down, the spectrum of the reflected signal shifts back to shorter wavelengths, which is visible in a weaker response of diodes P-T. Optical power is transferred back to the diode O and then M. When the heating pad is removed the FBG starts to cool down even faster, thus the diode O signal decreases rapidly while diode M reaches a maximum for a second time. When the FBG reaches back the room temperature the signals from diode I and then F increase, similar to the initial part of the measurement. It can be observed that the voltage curves recorded at the beginning and at the end of experiment differ slightly. The possible explanation is that the temperature conditions varyied during the experiment . Nonetheless, it does not affect the ability to proper detect respiratory rate.Finally, the experiments on detecting the breathing activity of a real person lying on the bed equipped with the FBG sensor plate were performed, the results of which are presented in It can be noticed that stabilization time is considerably long (ca. 20 min). However, it should be noted, that the reliable RR signal can be recorded irrespective of the temperature-induced wavelength change .The last measurement was made with the person lying on the mattress and breathing at a different rate for 3 min. Initially, the FBG plate was at room temperature and conditioned on an empty mattress. The volunteer then laid down and was breathing at a normal rate (1 breath for 5–6 s) for 90 s, then held his breath for 15 s (apnea), then began to breathe rapidly (1 breath for 1–2 s) for the next 20 s. Then he calmed his breathing and returned for the next 60 s to the pace with which he had started, and then he left the bed. The time traces recorded during this measurement are shown in The corresponding patterns may be clearly distinguished with respect to the characteristics recorded using both devices. In addition to the breathing pattern, the graph shows the global effect of the FBG heating from the human body, manifested by an increase of the Bragg wavelength. This effect is noticeable even in a relatively short section of the apnea. The disturbance recorded by both devices in ~62 s of the measurement was the result of the patient’s movement. Due to a slight change in body position, the tension on the FBG changed, which is also visible in the graph. It should be noted that any physical movements do not influence the breathing pattern recorded. Moreover, since the MRI examination should be carried out with the patient body immobilized, the possibility of detecting the patient’s movements may be considered as an added value of the system, which could enable informing the technician to repeat the procedure.In this paper, we have presented the compact optoelectronic system for monitoring fundamental vital signs of the patient under MRI scan, based on FBG sensors and photonic integrated circuit as an interrogating device, with so-far demonstrated functionality of the proper detection of the respiratory rate, while the work on heart rate detection is still in progress. The heart of the system is 36 channel AWG-based spectrometer, designed and manufactured on the indium phosphide generic platform provided by SMART Photonics company and driven by a dedicated, in-house developed electronic circuitry. The experiments confirmed the applicability of the developed system to targeted application–the PIC-based interrogator reacts properly to the changes of pressure caused by the breathing person’s chest to the FBG sensors and the temperature changes resulting from the human body warmth. Signal interrogation is provided with time resolution sufficient to track the speed of breathing and precisely distinguish between individual breaths. The developed device offers all the advantages inherent to photonic integrated circuits-high reliability, due to the elimination of volumetric elements, small size, low power consumption, and low cost, assuming a sufficiently large scale of manufacturing.Although the system has been originally designed for monitoring the condition of the patients exposed to MRI diagnosis its features might be effectively used in other areas of monitoring the condition of the patients, for example in intensive care units, nursing homes, or private houses . Complementing the functionality with the heart rate (HR) monitoring will further increase the usefulness of the device and open the gate to market deployment. The implementation of this functionality would require the sampling frequency to be increased (the presently used electronic circuitry was too slow) and for the the detection algorithms to be upgraded. As it is a subject of current development, it will be reported in future papers.