Non-destructive testing of concrete for defects detection, using acoustic techniques, is currently performed mainly by human inspection of recorded images. The images consist of the inside of the examined elements obtained from testing devices such as the ultrasonic tomograph. However, such an automatic inspection is time-consuming, expensive, and prone to errors. To address some of these problems, this paper aims to evaluate a convolutional neural network (CNN) toward an automated detection of flaws in concrete elements using ultrasonic tomography. There are two main stages in the proposed methodology. In the first stage, an image of the inside of the examined structure is obtained and recorded by performing ultrasonic tomography-based testing. In the second stage, a convolutional neural network model is used for automatic detection of defects and flaws in the recorded image. In this work, a large and pre-trained CNN is used. It was fine-tuned on a small set of images collected during laboratory tests. Lastly, the prepared model was applied for detecting flaws. The obtained model has proven to be able to accurately detect defects in examined concrete elements. The presented approach for automatic detection of flaws is being developed with the potential to not only detect defects of one type but also to classify various types of defects in concrete elements. Concrete structural members need to be diagnosed at different times and for different reasons. For many years, attempts have been made to investigate geometrical and material imperfections in concrete members by exploiting the propagation of elastic waves. The current trend in the diagnosis of these elements is to apply a non-destructive testing equipment that allows one to obtain an image of the inside of the examined elements for early detection of flaws ,2. AcousCurrently, the recorded images of the inside of the examined concrete members, obtained from ultrasonic tomography, are inspected mainly manually, which can be expensive, time-consuming, and can be prone to errors. Moreover, there are no standards available, which could make it possible to objectively interpret the results obtained in this way. In that situation, a data-driven approach can be helpful because recent developments in computer hardware and software together with the corresponding advances in deep learning algorithms and availability of large datasets make fully automatic analysis of images possible ,23,24,25With the above in mind, this article contains a full description of the innovative methodology developed by the authors. The main objective of this work is, therefore, set to investigate the novel application of ultrasonic tomography and a convolutional neural network (CNN) for automating the assessment of defects and flaws in concrete elements. However, because it is the first attempt to tackle the flaws detection problem, this work involves few limitations. First, only one type of defect is considered. This assumes that the images with defects, obtained from ultrasonic tomography, belong to one category only. Second, in the experiments, only the images with easily visible flaws are considered. In future experiments, however, these two limitations will be considered to get a fully automatic system for detection and classification of defects in concrete elements.The remainder of the paper is organized as follows. A brief description of ultrasonic tomography is presented. This is followed by a short introduction to artificial neural networks and a brief overview of convolutional neural networks. These provide the theoretical basis of the work undertaken in this paper. Next, we briefly discuss the proposed methodology for flaws detection. This is followed by a description of specific samples. The dataset is prepared, the neural network model is applied, numerical experiments are carried out, and the results of flaw detection are obtained. Lastly, a short discussion of the results and final conclusions are given.Elastic wave stimulation in the tested element is called the ultrasonic tomography technique. Elastic wave stimulation in the tested element is called the ultrasonic tomography technique. The excitation source is a multi-head antenna. It contains several dozen integrated ultrasonic heads that generate 50 kHz ultrasonic pulses. This frequency is the most suitable for concrete and similar materials . It is aEach head is telescopically mounted separately in the antenna, and the head adapts to the testing surface. Thanks to dry contact, no coupling agents or special surface preparation for testing is required. The distance between the heads is 30 mm and 40 mm, respectively, both vertically and horizontally. The results of testing using the ultrasonic tomography are then collected in a matrix table. This matrix is three-dimensional and is subsequently processed by the special software. In that way, it is possible to receive three scans: C, D, B, and 3D view in the mutually perpendicular directions, as shown in The convolutional neural network (CNN) is a special type of a layered feed-forward artificial neural network (ANN), designed for processing signals in the form of multiple arrays like visual and audio signals . The staThe typical neural network like MLP or CNN has an input layer, at least one hidden layer with nonlinear units (neurons), and an output layer with linear units (for regression problems) or nonlinear units (for classification problems). A unit (neuron) computes a weighted sum of its inputs called activation of the unit and then the activation is sent to the activation function, which is, in general, an S-shaped function such as a sigmoid function or a rectified linear unit (ReLU) function.The number of inputs in the input layer is equal to the total number of features in the input dataset. For example, for RGB images (three channels) with 150 × 150 pixels for each channel, the number of inputs is 67,500. The number of outputs depends on the problem at hand. For example, for the binary classification problem, there is only one output. For multi-class classification, the number of outputs corresponds to the number of classes.A CNN has a slightly different architecture. It consists mainly of two types of hidden layers, which are a convolutional layer and a pooling layer. There is also a fully-connected (dense) layer that forms the network outputs. These layers are stacked to form a convolutional neural network structure. The structure of a typical CNN with 11 layers is presented in The aim of the convolutional layer is to detect the input image local features such as horizontal, vertical, or diagonal edges. It is done by using a convolution operation or by computing cross-correlation more strictly .The pooling layer is used for dimensionality reduction of the processed data by combining values of small clusters (matrices) of input data. The most common pooling operation is defined by taking the maximal value. Additionally, the flattened operation is needed to change the input matrix to the output vector, which was then processed by the fully-connected (dense) layer. The fully-connected layer processes the input data by sending data from every unit (neuron) in the previous layer to every unit in another layer.A layered neural network is qualified by using the backpropagation algorithm to efficiently compute the gradient of a loss function and the mini-batch stochastic gradient descent algorithm (SGD) for learning the weights of the neural network model. During training of convolutional neural networks with many parameters , especially on a small dataset, the main issue is overfitting the model to the training dataset. There are several methods and techniques to cope with overfitting .One of the most effective techniques is image data augmentation . This teAnother problem with learning the large convolutional neural network is the extra-long time needed for a successful training process. One of the possible solutions is to apply transfer learning. Transfer learning is a method used to adapt the pre-trained model to another dataset . It is oThere are many commonly used pre-trained CNN models , which can be used in other problems by applying transfer learning. The pre-trained network has parameters that were computed on a very large training dataset from a specific domain or task. For example, the benchmark dataset called ImageNet is used for testing novel CNN structures and training algorithms .The methodology for detecting flaws in concrete elements using ultrasonic tomography and convolutional neural networks is shown in the form of a flowchart in The first step in the testing methodology consists of marking a grid of measuring points i = 1…j evenly spaced at every 50 mm, with a minimum distance of 50 mm from the edge of the tested concrete member. The spacing can be increased to 100 mm if the surface of the investigated member is considerable. The tomograph is then calibrated by repeatedly measuring ultrasonic wave  velocity and computing its mean value.At subsequent steps, ultrasonic wave velocity is measured in each antenna position in each of the testing points. During measurements, a preliminary analysis of the ultrasonic signals is performed to find out if the thickness of the member can be identified or the defect can be detected on this basis. If this is not the case, the measured signals are transformed using the customized software. The transformation consists of compiling the registered data for a given measuring point.If the results are acceptable, they are recorded. Lastly, flat scans B, C, and D are obtained in the three mutually perpendicular directions, showing the inside of the investigated concrete member. By using the dedicated software, it is also possible to build a three-dimensional scan. The next step is analyzing scans B, C, and D, and forming a set of data used for building (training and testing) the convolutional neural network (CNN). Lastly, the trained CNN is presented with a new scan and the network analyzes the new scan and decides if the scan contains a flaw or not.The presented methodology of measuring and processing obtained results using ultrasonic tomography and convolutional neural networks can be useful when developing a prototype of a computer vision system to automatically detect flaws in concrete elements.3 concrete cubic specimens were prepared. The specimens were made of C25/30 concrete based on aggregates with 8-mm maximum grading. Next, the specimens were tested in the laboratory using the technique of ultrasonic tomography and the images containing B-scans were recorded. The images were cropped to prepare a dataset, which was used to train a convolutional neural network. A CNN-based detection model was built using a dataset containing only 246 B-scans of concrete elements with flaws (52%) and without flaws (48%). The dataset was divided into three subsets: for training (56%), for validation (22%), and for testing (22%). For the purpose of this work, ten 1000 × 1000 × 1000 mmIn Similarly, in Selected examples of B-scans without flaws from the validation set are shown in Selected examples of B-scans with flaws from the validation set are shown in Selected examples of B-scans without flaws from the testing set are shown in Lastly, selected examples of B-scans with flaws from the testing set are shown in In this work, to achieve better results in automatic flaw detection, we adopted transfer learning and used a pre-trained convolutional neural network called VGG-16. It was proposed in 2014 by K. Simonyan and A. Zisserman from the Visual Geometry Group at the University of Oxford . VGG-16 We use the convolutional base of the pretrained network trained with an image augmentation binary classifier set on the top of the convolutional base. We also apply fine-tuning, introduced in The input for the network was the normalized image of the CT scan. The input image was then processed by several convolutional and max pooling layers. The kernels in the convolutional layers have the same size of 3 × 3 and a different number of filters. All convolutional layers used the ReLU activation function. Max pooling layers used a stride of size 2. After all convolutional and max pooling layers, the flattening operation is applied and then the resulting vector of features is processed by a fully connected layer with a ReLU function. Lastly, the input is classified by the last layer, which is fully connected and has the sigmoid function. The convolutional neural network was trained by using the Adam optimizer to optimize the cross-entropy loss function.The training process includes only fine-tuning of the last block of convolutional layers during 100 epochs of regularized training with image augmentation and dropout (50%). The CNN model is trained, validated, and tested by applying the corresponding datasets presented in The numerical experiments were prepared in the Python ecosystem using Keras library . The comIn this section, the training process and the results of the experiments as well as the final performance of the convolutional neural network (CNN) for image-based detection of flaws in concrete elements are presented.In In After the training process, it is possible to visualize the filters of the convolutional and pooling layers. It can be useful to better understand how the CNN model represents the visual information from the training dataset. In In Lastly, the trained convolutional neural network was checked against generalization capacity by using the testing dataset. After analysing the new B-scans by the network after fine-tuning, the generalization accuracy for the final network was close to 99%.The presented work is concerned with the development of novel methodology for fully automated analysis of images of the inside of the examined concrete elements obtained from an ultrasonic tomograph. The proposed methodology is based on the combination of the ultrasound tomography technique and the convolutional neural network (CNN). The CNN model is applied at the final stage of the developed methodology for classifying images into two categories: “element without a flaw” and “element with a flaw.”Because the main challenge was to collect enough data, instead of building the CNN model from scratch, we applied a pre-trained convolutional neural network called VGG-16 trained on the ImageNet dataset. The parameters of the pre-trained model were fine-tuned by using the collected images and the transfer learning technique. Moreover, two regularization techniques were applied to avoid problems with overfitting and obtain a satisfactory generalization property of the final CNN model. The first technique was image data augmentation by using basic transformations of an image-like rotation, horizontal flipping, and shifts. The second technique was network weights pruning called dropout. In the process of building the neural model, only 246 RGB images were used. After 100 epochs of training, the network achieved 98% accuracy on the training set with a loss of 0.05. The validation accuracy was 97% with a loss of 0.09. Lastly, the generalization accuracy was close to 99%, which confirms that the CNN model was built properly.Analysis of the obtained results shows that it is very possible to detect defects in concrete samples with very high accuracy. In the provided example, the concrete cubic laboratory specimens were tested using ultrasonic tomography and the trained convolutional neural network. The experiments were carried out according to the proposed testing approach.The innovative methodology of non-destructive detection of defects in concrete elements using ultrasound tomography and a convolutional neural network has been presented in detail. There are two main stages for detecting defects in concrete elements, according to the presented methodology. The first of these is to carry out non-destructive testing using ultrasound tomography techniques. The second stage is automatic defect detection using a convolutional neural network. Interesting results on building practices are given in the provided case study.A practical example of the application of the proposed methodology was presented,By using two regularization techniques and transfer learning, we were able to significantly reduce the risk of overfitting the model during training,In the end, we obtained the CNN-based classifier, which is able to generalize well on unseen images,The accuracy of detecting flaws using convolutional neural networks has been confirmed,The usefulness of the presented methodology for non-destructive identification of defects in concrete elements was shown,The proposed solution can serve as a prototype in the context of future practical applications.The benefit of the proposed methodology is based on the fact that the combination of ultrasonic tomography and convolutional neural networks offer a method to build a computerized system for complete automatic detection of flaws in concrete elements when given a very small number of recorded image datasets. Based on this research experience, the authors have found that the developed methodology has practical applications in future automatic inspection systems.In the future, because this was the first attempt to tackle the problem of automatic flaws detection, the limitation and challenges that were faced in this work will be addressed. First, we considered only one type of defect simulated by a PVC pipe embedded in the concrete element. As a result, the images with such a defect belong to one category only. Second, only the images with easily visible flaws were taken into account. In future work, the presented limitations will be considered by testing concrete specimens with other defects to be able to get closer to an automatic system for detection and classification of flaws in concrete elements.


In order to obtain accurate dynamic parameters of materials, the influences of friction and inertia should be considered in the SHPB tests. In this study, the effects of the friction conditions, specimen shape, and specimen configuration on the SHPB results are numerically investigated for rate-independent material, rate-dependent elastic-plastic material, and rate-dependent visco-elastic material. High-strength steel DP500 and polymethylmethacrylate are the representative materials for the latter two materials. The rate-independent material used the same elastic modulus and hardening modulus as the rate-dependent visco-elastic material but without strain rate effects for comparison. The impact velocities were 3 and 10 m/s. The results show that friction and inertia can produce a significant increase in the flow stress, and their effects are affected by impact velocities. Rate-dependent visco-elasticity material specimen is the most sensitive material to friction and inertia effects among these three materials . A theoretical analysis based on the conservation of energy is conducted to quantitatively analyze the relationship between the stress measured in the specimen and friction as well as inertia effects. Furthermore, the methods to reduce the influence of friction and inertia effects on the experimental results are further analyzed.The dynamic properties of materials should be analyzed for the material selection and safety design of robots used in the army and other protective structural applications. Split Hopkinson pressure bars (SHPB) is a widely used system for measuring the dynamic behavior of materials between 10 Designers, engineers, and manufacturers are always in search of new and better materials for performance improvements and cost reduction in their robot products to remain competitive in the market. This requirement has resulted in the limited selection of thousands of material choices for designers. Metal, composite material, and polymer are still the main choices for the structural design of robots, especially for their bearing components. In engineering applications, quadruped bionic robots , exoskel2 and 104 s−1 [The split Hopkinson pressure bar (SHPB) is a widely used experimental system to investigate the dynamic compression properties of materials between strain rates of 10 104 s−1 . There a 104 s−1 ,7,8. Exp 104 s−1 ,10. A me 104 s−1 ,12. TherInterfacial friction and inertia effects in SHPB tests have been realized by analytical ,14,15,16A theoretical model was firstly established by Davies and Hunter . They fo−1. However, it is generally  accepted that inertial confinement plays an important role in the SHPB results when the strain rate is higher than 1000 s−1; According to experimental observation, Bertholf and Karnes [Various experimental analyses have been conducted by researchers. Jia  suggested Karnes  establisd Karnes  introducd Karnes  conducted Karnes  was not d Karnes  conducteIn the present study, the influence of friction effects and inertia on the dynamic mechanical properties is numerically analyzed, particularly for different materials and loading conditions. An analytical model is established to quantitatively analyze the interfacial friction and inertia effects in SHPB tests. Methods to minimize the influence of interfacial friction and inertia are proposed based on the simulation results and the analytical model.The structure of the paper is as follows. In A conventional SHPB experimental device consists of a striker, an incident bar, and a transmit bar, as shown in E is Young’s modulus of the bar, The elastic stress wave propagation speed Based on the assumptions of SHPB, the stress In a SHPB experiments, for the sufficiently lubricated contact condition the friction coefficient on the interface is zero. The specimen is uniformly shortening in the axial direction (Z direction) and freely expanding in the radial direction  constitute model whose parameters are obtained from experimental testing  and exprIn the SHPB experiments, when the incident stress pulse crosses a variable cross-sectional specimen, the confining effect due to friction will bring a non-uniform stress distribution in the specimen axial direction. The value of the transmitted stress pulse is influenced by the radius difference in the cross section area and the friction coefficient. As the assumption of uniform stress distribution is challenged, the results obtained by the traditional SHPB processing method become unreliable. The analysis model of the inertia effect of the specimen is shown in The deformation velocity Assuming that the strain rate in the cross-section area of the specimen is uniform, the velocity field of the specimen in the axial direction can be expressed as:Assuming that the volume of the specimen is constant during compression, the velocity field of the specimen in radial direction can be expressed as:Assuming that the interfacial friction has little influence on the velocity field, temporal change rate of the kinetic energy The temporal change rate of the deformation energy sE is the Young’s modulus of materials.In which The temporal rate of change of the external work can be expressed as:Assuming that the friction is uniformly distributed on the end surface, the temporal rate of change of the external work can be expressed as:The equation of motion in the axial direction is:Assuming that the deformation process of the specimen is adiabatic, the conservation of energy can be expressed as:Substituting Equations (9)–(15) into (16), it can be concluded in Equation (17) if the average stress .The dynamic response of specimens with different geometry shapes under SHPB compression for rate-independent material, rate-dependent elastic-plastic material, and rate-dependent viscoelasticity material are numerically analyzed by considering the cubic specimen and cylinder specimen (both specimens with 0.66 slenderness). Two initial impact velocities were considered, 3 and 10 m/s. The contact condition is set frictionless. The results of the stress pulses of the incident and transmitted bars are presented in For rate-independent materials, the duration of the waves is not affected significantly by the specimen geometry for both the 3 and 10 m/s impact velocities. When the impact velocity is 3 m/s, the transmit stress level of the cubic specimen is 30% higher than that of the cylinder specimen, and the value decreases to 16% when the impact velocity is 10 m/s. For rate-dependent elastic plastic materials, the duration of the waves is still not affected by the specimen shape for the 3 and 10 m/s impact velocities. When the impact velocity is 3 m/s, the transmit stress level of the cubic specimen is 22.5% higher than that of the cylinder specimen, and the value decreases to 11.7% when the impact velocity is 10 m/s. For rate-dependent viscoelasticity materials, similarly the duration of the waves is not affected by the specimen shape for both impact velocities. When the impact velocity is 3 m/s, the transmit stress level of the cubic specimen is 27% higher than that of the cylinder specimen, and the value decreases to 16% when the impact velocity is 10 m/s. Besides this, the transmit wave shape of the cylinder specimen is a triangle, but it transforms into square wave when the specimen shape is cubic.The specimen geometry has a significant effect on the reflected and transmitted wave stress levels, but not the stress wave duration. Cubic specimens present a higher stress than cylinder specimens regardless of the testing material properties. This is due to the fact that the stress waves would converge at the endpoints of the cuboid specimen and result in a mistaken higher flow stress for testing materials. Rate-dependent viscoelasticity material is the most sensitive to specimen shape among the three materials. However, the increase rate of the transmitted stress value decreases with the increase in the impact velocity. Rate-dependent elastic-plastic material is sensitive to the specimen shape, whose transmitted stress level increases significantly with the impact velocity increasing; rate-independent material is influenced by interfacial friction, and its effect decreases with the impact velocity.The dynamic response of the cylinder specimen with different slenderness under SHPB compression for rate-independent material, rate-dependent elastic-plastic material, and rate-dependent viscoelasticity material are numerically analyzed by considering 0.5, 1, 1.5. The contact condition is set frictionless. The results of the stress pulses of the incident and transmitted bars are presented in For rate-independent materials, the duration of the waves is not affected by the specimen slenderness regardless of the impact velocities. When the impact velocity is 3 m/s, the transmit stress level of the specimen first increases then decreases with the specimen slenderness increasing. The maximum stress value is 31% higher than the minimum stress value. When the impact velocity is 10 m/s, the transmit stress level shows a similar trend and the ratio of increment is 44%. For rate-dependent elastic plastic materials, the duration of the waves and the transmit stress level are not affected by the specimen slenderness for different impact velocities. For rate-dependent viscoelasticity materials, similar observations as for rate-independent materials are noticed. The duration of the waves is not affected by the specimen slenderness. The transmit stress level of the specimen first increases then decreases with the specimen slenderness increasing. The ratio of increment is 18% for the 3 m/s impact velocity and 17% for the 10 m/s impact velocity.The specimen geometry has a significant effect on the reflected and transmitted wave stress levels for rate-independent materials and rate-dependent viscoelasticity materials, but not for rate-dependent elastic plastic materials. The transmit stress value first increases then decreases with the increasing specimen slenderness.Thus, ways to minimize the effect of inertia effects and end friction can be considered as follows:In order to eliminate the effect of term, the coefficient should be approximately zero. Thus, the optimum specimen slenderness ratio should be 1/3 during experiments, which is also the criterion derived by Davies and Hunter in 1963. However, the specimen slenderness ratio increases with the specimen deformation increasing. The criterion derived by Davies and Hunter is only suitable at the experiment’s beginning when the deformation is small. Since it is impossible to ensure that the slenderness ratio stays constant during the experiments, a new criterion which minimizes the absolute sum of the coefficient can be applied to obtain the optimum specimen slenderness ratio. According to the constant strain rate hypothesis and the plastic incompressibility hypothesis, the specimen thickness and diameter can be expressed by its final strain as:Thus, the optimum specimen slenderness ratio to minimize the inertia effects can be expressed as:The optimum specimen slenderness ratio is related to its final strain, whose relationship can be fitted by Equation (25), and expressed in The effect of interfacial friction can be ignored if its influence is less than 5%. Thus, the friction coefficient can be calculated as:The above analysis is an efficient method to assess the influence of interfacial friction. A more general method to eliminate the interfacial friction effect is to correct the experimental results according to the theoretical analysis.Distinguished with the available literature, friction effects and inertia effects were analyzed theoretically and numerically for different materials in this study. Typical materials were represented by rate-independent material, rate-dependent elastic-plastic material, and rate-dependent visco-elastic material. Inertia effects were investigated by two factors: specimen shape and specimen slenderness. Two impact velocities of 3 and 10m/s were considered. An analytical model was established to quantitatively analyze the interfacial friction and inertia effects in the SHPB tests. Methods to minimize the influence of interfacial friction and inertia were proposed based on the simulation results and analytical model. This study provided material selection suggestions for army and other protective robot designers when they analyze the dynamic properties of materials for a safety design. The main conclusions are as follows:The interfacial friction brings extra stress for the SHPB specimen, whose values is related to the testing material properties and impact velocities. Rate-dependent viscoelasticity material is the most sensitive to friction effects among the three materials. However, the increase rate of the transmitted stress value is not influenced by the impact velocity; there is a cross in the viscosity deformation process, after which point the reflected stress level for the case with a smaller friction coefficient was found to be with a higher transmit stress level than the case with a larger friction coefficient. The rate-dependent elastic-plastic material is sensitive to the interfacial friction. The increase rate is significantly influenced by the impact velocity; rate-independent material is influenced by interfacial friction, and its effect is independent of the impact velocity.The specimen geometry has a significant effect on the reflected and transmitted wave stress levels, but not the stress wave duration. Cubic specimens present a higher stress than cylinder specimens regardless of the testing material properties. Rate-dependent viscoelasticity material is the most sensitive to specimen shape among the three materials. However, the increase rate of the transmitted stress value decreases with the impact velocity increasing; rate-dependent elastic-plastic material is sensitive to the specimen shape, whose transmitted stress level increases significantly with the impact velocity increasing. Rate-independent material is influenced by interfacial friction, and its effect decreases with the impact velocity.The result of impact response of different materials with different interfacial and geometry can be used in the structural design of these materials, especially under impact loads. Under the same loading condition, the difference in the mechanical response of different materials can be attributed to the difference in microstructure. This result can provide some theoretical basis for researchers to improve the material properties.The specimen geometry has a significant effect on the reflected and transmitted wave stress levels for rate-independent materials and rate-dependent viscoelasticity materials, but not for rate-dependent elastic plastic materials. The transmit stress value first increases then decreases with the increasing specimen slenderness.The influence of the friction and inertia effect on the SHPB test results is related to the specimen size, deformation strain, strain rate, density of the specimen, and friction coefficient of the interface. Methods to reduce the influence of the inertia effect on the experimental results can be achieved by reducing the influence of these factors, such as using constant strain rate loading experiment technology and a lubricated contact interface. The optimum specimen slenderness ratio is related to its final strain, whose relationship can be described by a linear function. The interface friction is a very complex dynamic process; when the object is deformed in the friction experiments, the process will be even more complex. However, this needs to be researched, since it always occurs in daily life.


As an essential mechanical device in many industrial applications, reciprocating compressors have a high demand for operating efficiency and availability. Because the temperature of each part of a reciprocating compressor depends considerably on operating conditions, faults in any parts will cause the variation of the temperature distribution, which provides the possibility to distinguish the fault type of reciprocating compressors by differentiating the distribution using infrared thermal imaging. In this paper, three types of common fault are laboratory experimented in an uncontrolled temperature environment. The temperature distribution signals of a reciprocating compressor are captured by a non-contact infrared camera remotely in the form of heat maps during the experimental process. Based on the temperature distribution under baseline condition, temperature fields of six main components were selected via Hue-Saturation-Value (HSV) image as diagnostic features. During the experiment, the average grayscale values of each component were calculated to form 6-dimension vectors to represent the variation of the temperature distribution. A computational efficient multiclass support vector machine (SVM) model is then used for classifying the differences of the distributions, and the classification results demonstrate that the average temperatures of six main components aided by SVM is a promising technique to diagnose the faults of reciprocating compressors under various operating conditions with a classification accuracy of more than 99%. As an essential mechanical device in many industrial applications, reciprocating compressors have the advantages of high compression efficiency, a simple device system, and mature technology. Therefore, the reciprocating compressors have a wide range of industrial applications. However, the shortcomings such as too many moving parts and complicated structures can lead to various common faults. In order to avoid causing significant economic losses and casualties, it is very important to diagnose the fault of reciprocating compressor in time.Among sensing technologies such as acoustic emissions, vibration, acoustics and pressure curve synthesis, vibration-based methods are mainstream and actively studied for different applications such as machine tools , productCompared with the model-based methods, big data and machine-learning technologies have unique advantages in establishing the relationship between parameters and fault types, and provide new technical ideas for diagnosing the failure of reciprocating compressors. Qi et al.  proposedReciprocating compressors produce a certain amount of heat during the process of compressing gas. Experiments have found that the temperature distribution changes during the early stage of failures in reciprocating engines , i.e., tInfrared thermal cameras are a non-contact temperature measuring device that can accurately obtain the surface temperature distribution information of the measured object in real time. According to this function, thermal imaging technology has been widely used in scientific research and industrial production ,27,28,29In order to verify the feasibility of using the infrared images to identify the faults of reciprocating compressors, experiments were conducted based on a two-stage V-shape reciprocating compressor which is driven by a three-phase, 2.5 kW induction motor through a transmission belt. The structure of the experimental test rig is shown in The experiments were conducted under different ambient temperature. Additionally, the camera was located at a constant distance to the compressor when it was running. However, the camera position changed slightly due to vibrations caused by the compressor and human disturbance during installation resulting in a slight image shift during the operation of compressors with different faults. The pressure in the tank rose from atmospheric pressure to about 0.8 MPa (or 120 psi) which is recommended by the compressor manufacturer during the operation. This process lasted for about 10 min with the thermal video recorded.Faults in different parts of reciprocating compressors often induce changes in the temperature field. For example, motor faults will cause changes in the motor temperature distribution. This feature has been widely used in motor fault diagnosis ,17, accoAccording to the above simple analysis, this study experimented in the laboratory with three common faults of reciprocating compressors: air filter blockage (AFB), asymmetrical stator winding (ASW), and discharge valve leakage (DVL). The first fault shown in The higher the object temperature is, the faster the particles move. Consequently, more energy radiates outward at higher temperature and vice versa. Based on this principle, infrared cameras convert the infrared radiation energy of objects into electrical signals, and display them in the form of images. Because color images can express a wider color range than grayscale images, and because the color range that a color image can express is wider than the color range that a grayscale image can express, an infrared image is expressed by converting electrical signals into a color image through a specific process such as rainbow encoding or hot-metal encoding. All of these encoding methods have a one-to-one correspondence between temperature and color, so the temperature of the object can be analyzed by analyzing the color of the image.In order to understand the law of temperature rise in different parts of the reciprocating compressor, six interest regions were selected from color image, which corresponds to the temperatures of the motor (Motor), first stage inlet (1st inlet), first stage outlet (1st outlet), cooling system inlet (Cooling inlet), cooling system outlet (Cooling outlet), second-stage outlet (2nd outlet) respectively, as shown in Due to the inconsistent surface structure of the selected parts, the rate of temperature rise at different points in the same part are also different. This study ignores the heating process at each point, and focuses on the variation trend of temperature compared with the final state of the temperature at each point. The last frames of the infrared videos obtained under different working conditions was converted to HSV color mode which is shown in Divide each region according to the threshold in Each frame in the acquired infrared video is converted into a grayscale image as show in Although there are clear differences between the grayscale values for the entire operating process, it is not easy to use a simple method to differentiate the four cases. This study chooses a popular SVM algorithm for classifying these complicated data patterns because of its better generalization and nonlinear classification abilities, showing outstanding performances in diagnosing different faults . In partUsing the trained classifier to classify the test data, the results show that the classification accuracy reaches 99.7%. The testing result is shown in In order to further confirm the robustness of the method proposed in this paper, 50% of the data are randomly selected as test data, and the data involved in training are continuously reduced from the remaining data. We use the test data to verify the trained classifiers, and the trend of the proportion of correct classification is shown in One of the important merits using the proposed method for the thermal based monitoring is that it needs much less computational effort, compared with the deep learning-based methods . MoreoveAccording to basic common sense, the temperature range  of the compressor during normal operation is usually distinctive and relatively stable. In this study, the SVM classifier was trained using the all dataset from the entire process, and it allows the faults to be detected from a cold start process. However, when the compressor has a warm-up cycle, the detection performance probably will low for a short period before the temperature becomes stabilized. Once the temperature becomes stable, the distribution of temperature will follow the profiles and thus the detection performance will be recovered.Of course, we also note that the ambient temperature will have a significant impact on our method. The actual working environment of the compressor has a larger temperature change than that under-laboratory conditions. To maintain the performance of fault detection, the SVM classier should be trained with such wide ambient temperatures. However, it needs a further study to evaluate the robustness to ambient temperature ranges. According to the current study in which the SVM can perform sufficiently well for the ambient temperature from 20.5–24.7 °C, it may need only a few ambient temperature points to have a robust SVM trained, which is not only feasible but also efficient as the object-based thermal image segmentation, proposed in this study needs little computation effort. This paper proposed a non-intrusive diagnosis method based on thermal images and SVM classifiers for the condition monitoring of reciprocating compressors. Six image regions were selected based on the key subsystems of the reciprocating compressor. The average grayscale values of each region in the grayscale infrared images was calculated to form 6-dimensional feature vectors, which highlights the differences of the temperature variation between subsystems while significantly reducing the data dimensions. An easy implementation classifier, SVM, was employed for fault type classification under different operating conditions. The statistical average accuracy of the classification can reach up to 99.7%, which verifies this method has high accuracy and reliability. The varying thermal information of the core components in compressors is highly correlated to the working principle. Therefore, the proposed intelligent method is potentially transferred into the industrial applications. To summarize, the infrared camera, as a non-intrusive and non-contact measuring device, incorporating with engineering knowledge-based data processing, has promising applications for the intelligent condition monitoring and fault diagnosis of reciprocating compressors.


The storage of temporally precise spike patterns can be realized by a single neuron. A spiking neural network (SNN) model is utilized to demonstrate the ability to precisely recall a spike pattern after presenting a single input. We show by using a simulation study that the temporal properties of input patterns can be transformed into spatial patterns of local dendritic spikes. The localization of time-points of spikes is facilitated by phase-shift of the subthreshold membrane potential oscillations (SMO) in the dendritic branches, which modifies their excitability. In reference to the points in time of the arriving input, the dendritic spikes are triggered in different branches. To store spatially distributed patterns, two unsupervised learning mechanisms are utilized. Either synaptic weights to the branches, spatial representation of the temporal input pattern, are enhanced by spike-timing-dependent plasticity (STDP) or the oscillation power of SMOs in spiking branches is increased by dendritic spikes. For retrieval, spike bursts activate stored spatiotemporal patterns in dendritic branches, which reactivate the original somatic spike patterns. The simulation of the prototypical model demonstrates the principle, how linking time to space enables the storage of temporal features of an input. Plausibility, advantages, and some variations of the proposed model are also discussed. In daily life, we can distinguish between temporal and spatial properties of our world. In the brain, the temporal as well as spatial properties of the world is largely represented by spatiotemporal patterns of neural spikes. However, an important question currently facing scientists is: How the temporal dimension of the physical world is represented in the brain? The representation of time-dimension is required for a successful interaction of the brain with the four-dimensional physical world  assume that the temporal features of the input come from distinct neurons. The localization of the input time-points at different neurons is a precondition.Several past studies of spatiotemporal patterns are based on the processing the temporal features of an input. Many of these studies address the association between a precise input spike train and a desired output spike train in a temporally specific manner . The input level comprises two neurons, the input neuron I and the attention neuron A. The activity of neuron A is considered a correlate of “attention” serving storage and recall. The output level consists of a single neuron, the extension neuron which comprises several dendritic branches with the particular feature that they all exhibit subthreshold membrane potential oscillations of the same oscillation period but of different phases.The prototypical modelSpike activities of I and A , left arThe core of the presented memory model is the time to space extension process during encoding . It takeFinally, the soma of neuron E accumulates the scaled potentials of dendritic spikes with scaled potentials of arriving inputs from I and A. All three membrane potential changes sum together to produce a somatic spike in neuron E. It is believed that the impact of dendritic oscillations on the soma is balanced out because of the distribution of phases between branches. Therefore, the effect of dendritic oscillations on somatic membrane potential are excluded.For the storage of inputs, a learning mechanism is required. In the proposed model, we consider two “unsupervised” mechanisms for learning, occurring whenever a dendritic spike in E is generated due to the coincidence of spikes from neurons I and A with a depolarization maximum of SMOs in the dendrites of extension neuron E. Gamma-range coincidence detection corresponds to the findings of Das et al. , Das andThe first strategy to store the spike pattern from input neuron I is based on the enhancement of synaptic weights. The choice of the initial synaptic weight is arbitrary, but it is interrelated to other neuronal parameters.The synaptic weights between input neuron I and the dendrites of neuron E, set to W(IE) = 0.15, are kept constant, unaffected by the learning process. The relevant changes have to take place at the synapses from neuron A to neuron E because recall will be initiated via this pathway when the input neuron will be silent. Hence, the EPSP amplitudes from neuron A have to be sufficiently increased to compensate for the lack of the EPSPs from input neuron I.ini = 0.40. In this model a single spike increases the synaptic weight to W(AE)max = 0.55, . This will increase further EPSP amplitudes which shall be sufficient to store the information of the timing of a spike input from neuron I in the corresponding dendrite of neuron E for later recall by neuron A. Such a potentiation seems biologically plausible. Remy and Spruston Spruston  used a sSingle somatic spikes do not significantly change the synaptic weights at dendrites. Since the proposed model is a single trial, involving a single spike burst, the propagation of somatic action potential will have a negligible effect on LTP.In an alternative learning mechanism, the encoding occurs due to the enhancement of the amplitude of the SMO after dendritic spikes. This is consistent with the enhancement of amplitude of theta oscillations from cortical EEG recording in a working memory task in humans  as used during the encoding. During this recall, the input neuron is silent. The impact of the input neuron during encoding is substituted by enhanced synaptic weights from neuron A. Furthermore, during recall, the dendritic branches fire at time-points corresponding to the input spike train that was initially encoded. Persistence of phase-shifted SMOs in the dendritic branches together with enhanced synaptic weights would then result in the recall spikes at dendritic branches. Dendritic spikes generate somatic spikes in E matching the output during the encoding process. It should be noted that the recalled spike train evokes the somatic spike train, generated by the input spike train, not the input spike train itself. The input spike train is recalled only, if it reproduces an equivalent somatic spike train during encoding.A major goal of this study is to show that a single neuron would be able to store temporal properties of an input by the spatial pattern of dendritic branch activation. The current simulation study uses simple mechanisms, such as theta-gamma-oscillations combined with LTP, which is implemented in a small SNN.Gamma oscillations are represented in the rhythm of spike inputs. Theta oscillations are explicitly modeled by phase-shifted sine functions f(OP):h = 6.0 mV is the oscillation amplitude, fq = 8.33 is the oscillation frequency, t is the time in ms and ph = 10 ms is the steps size of phases, multiplied by j = −2 to 10.where Referring to the model of Legenstein and Maas , each spk = 39, g = 2.0, and τ = 1 are constants and W is the synaptic weight ini = 0.15 and initial W(AE)ini = 0.40. δt is the time difference to the EPSP onset.where dend = −48.7 mV is reached. The branch spike potential (BsP) is again modeled in form of an alpha function:SMOs and EPSPs sum up, which generates a dendritic spike whenever a threshold φk = 40, g = 2.0, and τ = 1 are constants with δt as the time difference to the dendritic spike onset.where The total branch potentials are summed up by the SMO potentials, the local EPSPs and spike potentials.B = 1.0) and passive, electrotonic propagation of EPSPs with decay to upass = 0.08.EPSPs and spikes propagate to the soma, scaled by different weighting factors, assuming active conduction of dendritic spikes without attenuation  in Python 3.7 .Parameters for somatic, dendritic, synaptic and oscillation properties are shown in The model has been tested with input trains of random numbers of input spikes at randomized times, however, adjusted to the 100 Hz gamma rhythm of the spiking of the attention neuron. Test runs have been made with spike inputs and recall during a full cycle period of the dendritic SMOs (120 ms). In this period the attention neuron fires a continuous sequence of 12 spikes.For encoding and learning, additionally a randomly chosen number of up to 11 spikes from the input neuron arrive at the dendrites of the extension neuron. These input spikes are synchronized with the spike times of the attention neuron but appear at random positions. In any case, due to the above described mechanisms, , .Recall is initialized by the application of a spike burst from the attention neuron A while the input neuron I is silent. Due to the learning mechanism, .This was confirmed by numerous (100) trials with randomized input patterns each showing an exact reproduction of the initial input pattern in the extension neuron after learning and recall. An example is shown in Details of functionally relevant alterations of dendritic and somatic mechanisms during encoding and learning and how these are enabling correct recall are shown in the following sections.dend = −48.7 mV for spike generation.Dendritic processes: The net membrane potential of the branch E2 is the sum of the intrinsic potentials of SMO and EPSPs from local synaptic activation by neuron I and neuron A, eventually superimposed by a dendritic spike. The time course of all these potentials is shown in In other branches, dendritic spikes will not be generated at this time, even when the EPSPs are the same. The reason is that potential of the phase shifted oscillations is too low. Spikes in other branches will be generated when their oscillations are in their maxima, provided that there is again spike input from both neurons at the input level, . In this example, already shown in Somatic processes: The dendritic spikes propagate to the soma together with EPSPs from all branches. However, as it needs the combination of both to exceed the threshold of spike generation the same spike pattern will appear at the extension neuron as received from the input neuron .Learning is implemented at the extension neuron either by enhanced EPSPs from the attention neuron or by increased oscillation amplitudes of the dendritic branches . These aThe previous spike pattern from the input neuron can be recalled by a burst of spikes from the attention neuron while the input neuron is silent. The attention neuron thereby has to fire at the same gamma rhythm as during encoding and the spike burst should have in minimum a duration of the length of the theta cycle over which the maxima of the SMOs are distributed. This guarantees that each dendrite will receive a spike input from the attention neuron when its SMO is close its maximum. In this case, a single spike from the attention neuron can elicit again a dendritic spike, however, only in those branches in which the learning mechanism, as described before, have either increased the oscillation amplitude or enhanced the synaptic efficacy due to enhanced EPSPs. The one like the other is sufficient to compensate for the lack of spikes from the input neuron. Dendritic branches without these learning effects will remain subthreshold even when a spike from the attention neuron input hits the oscillation maxima.Nothing has changed with spike generation at the soma due to the propagation of dendritic EPSPs and spikes. Propagation of dendritic EPSPs alone cannot generate somatic spikes. This needs the contribution of dendritic spikes. Hence, dendritic spikes, together with dendritic EPSP will propagate to the neuron's soma, there generating a sequence of spikes which exactly reflects the previous input pattern , just reThe simulation of the current model demonstrates how temporally precise spike trains can be stored by the spatial pattern of dendritic branches even in a single neuron. This mechanism can precisely store temporal information about onetime presented single spikes via the interaction of neural oscillations. The spike trains can be just recalled by a continuous spike burst. The key feature of this model is the spatial localization of spike-timing, which is established by phase-shifted theta oscillations of excitability in dendritic branches in combination with gamma-aligned input patterns. This mechanism links the temporal order of spikes of the input to different dendritic branches, allowing the transformation of temporal properties into the spatial pattern of dendritic activation. The spatial localization of the timing information enables the storage of temporal properties by learning mechanisms.Our prototypical model illustrates within a single neuron the general mechanism of transformation of time into spatial dimensions using cross frequency coupling of neuronal oscillations. The same approach can be adapted to ensembles of neurons. Analogous to the dendrites of a single neuron, an ensemble of neurons can work together as canonical microcircuits based on same principles. However, instead of the SMOs of dendrites, the somatic SMOs of ensembles vary in phases. The somatic spikes of single neurons correspond to the dendritic spikes in our prototypical model. The somatic outputs of the ensemble neurons have to be propagated to an additional output neuron, corresponding to the somatic output of the single neuron. A simulation trial of this alternative model-version has produced identical results. A bursting input by neuron A to the ensemble neurons reproduced an equal temporal pattern in the output neuron as was produced during the encoding process. Future simulation studies could reveal the effect of randomness in the spike trains or various oscillation parameters in neuronal ensembles on the spatiotemporal patterns.Some further assumptions of our model can be changed without destroying its functionality. For example, the regular distribution of phases of SMO within spatial units can be replaced by random phases, if the number of dendritic branches was enhanced from 13 to 48. Preliminary simulations led to 96% right recalled spikes and 85% completely right recalled spike trains, using 1,000 random input patterns.Moreover, oscillation-based conversion from temporal to spatiotemporal neural patterns can be also useful in the reverse direction. Since a temporally precise spike train can be represented by a more complex spatiotemporal neural pattern, a spatiotemporal pattern can be propagated to neuronal areas far apart from the origin by the corresponding simpler temporal code. The spatial part of the information can be rebuilt by synchronized oscillations between the area of origin and a target area. In summary, via oscillations spatial information can be propagated by temporal sequences, and these temporal sequences can be efficiently stored by transformation into neural spatial patterns.The assumption of dendritic branches acting as independent subunits to process the memory of spike trains appears to be feasible . Nevertheless, the additional part of a longer lasting input spike train can be stored by a second bursting neuron via its connections to the same (or a new) extension neuron. A temporal driven sequence of bursting neurons [e.g., by time cells , which was active throughout the encoding process, a further interesting option presents itself: The continuity of bursting yields original output of the input spike train, even if the input train itself is finished. Thus, the output of the terminated input is replicated as long as the bursting lasts. Even a complex input pattern within a group of neurons can repeat oneself as long as the burst continues. In addition, using SMO-learning instead of LTP, a properly tuned bursting input from any neuronal source can recall the stored input patterns and realize the working memory.—the three main characteristics of oscillations—can work together for this purpose. Significantly, the simulations presented in this study show that neural oscillations can allow time-dimension to be linked with the spatial dimensions in the brain circuits, which is important for the cognitive functions in interacting with the four-dimensional physical world. Moreover, simple spike bursts frequency aligned to the input events can serve as the trigger for retrieval as well as correlate for attention. The utility of the proposed memory mechanism for various brain functions, such as working memory is also evident. Preliminarily, our model is for now a theoretical hypothesis supported by computational simulations using physiologically plausible range of parameters. As a prototype model, it will be further enhanced by additional biological findings.Authors suggest that the realization of neuronal memory for temporal events is restricted to distributing temporal properties of events over spatially different units. The ubiquitous brain oscillations combined with the synchronization of neural activities can store temporal information as spatial patterns. Frequency, phase and amplitudeThe datasets generated for this study are available on request to the corresponding author.HL and DG contributed conception and design of the study. HL simulated the model and wrote the first draft of the manuscript. DG wrote sections of the manuscript. Both authors contributed to manuscript revision, read and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Visual analysis is the most commonly used method for interpreting data from single-case designs, but levels of interrater agreement remain a concern. Although structured aids to visual analysis such as the dual-criteria (DC) method may increase interrater agreement, the accuracy of the analyses may still benefit from improvements. Thus, the purpose of our study was to (a) examine correspondence between visual analysis and models derived from different machine learning algorithms, and (b) compare the accuracy, Type I error rate and power of each of our models with those produced by the DC method. We trained our models on a previously published dataset and then conducted analyses on both nonsimulated and simulated graphs. All our models derived from machine learning algorithms matched the interpretation of the visual analysts more frequently than the DC method. Furthermore, the machine learning algorithms outperformed the DC method on accuracy, Type I error rate, and power. Our results support the somewhat unorthodox proposition that behavior analysts may use machine learning algorithms to supplement their visual analysis of single-case data, but more research is needed to examine the potential benefits and drawbacks of such an approach. Visual analysis is the most commonly recommended method for analyzing single-case data  to the data that will best predict the labels of the dataset. The values of the parameters of the model determine its accuracy whereas the hyperparameters provide information as how the algorithm should process the data. Table Stochastic gradient descent in machine learning is analogous to shaping in behavior analysis. That is, the algorithm produces successive predictions (approximations) of the labels until these predictions best matches the true labels . During training, stochastic gradient involves only keeping the model that produces the best accuracy while discarding the others . In mathematical terms, the algorithm 1) multiplies the input (features) by weights (parameters), 2) transforms the previous products using a function that produces predictions, and 3) updates the weights by applying a correction related to the gradient of the error on the predictions for the next iteration .For our initial study, we limited our analyses to graphs with three points in phase A and five points in phase B because (1) graphs with fewer points are typically more difficult to analyze visually and statistically  and recorded the expected direction of change  for each graph. Albeit theoretically a BA graph, we also refer to the middle graph  as an AB graph; however, we reversed the expected direction of change when compared to the two other AB graphs extracted from the same ABAB design. The program rejected AB graphs that contained fewer than five points in the second phase. When graphs contained more data points than the required number , we used only the last three points of Phase A and the first five points of Phase B so that the points on either side of the phase change lines remained consistent. In total, our extraction yielded 1,070 AB graphs for training, validation, and testing.As described below, two observers independently categorized the graphs as either showing a clear change or no clear change. In the dataset with agreements only, we only kept graphs for which the two raters agreed on the labels to examine whether a consensus-based approach would train more accurate models. Because the interrater agreement was 91%, this dataset contained 970 graphs.The first author, a professor and doctoral-level certified behavior analyst who conducts single-case research, independently visually analyzed each of the 1,070 nonsimulated AB graphs by conducting a qualitative analysis of trend, immediacy of change, level change, and variability , which were used to adjust the parameters of the model during training. We provided these data to each algorithm along with the labels established by the first author  or the labels on which both raters agreed. The validation sets involved 25% of the datasets (267 graphs for the complete dataset and 242 for the dataset containing agreements only). Each algorithm computed accuracy on the validation sets for different combinations of hyperparameters  the complete dataset and (2) the dataset containing agreements only. In particular, our Python code computed the proportion of time the prediction of each model matched the result of the visual analysis for each dataset by adding the number of agreements between the visual analyst and the model, and dividing this sum by the total number of graphs. Our program computed three values: one for all test graphs, one for test graphs showing a clear change only, and one for test graphs showing no clear change only .both lines. In this case, all points fall below both lines for the graph in the left panel and three points fall below both lines in the right panel. The cut-off value based on the binomial distribution is five. Therefore, only the left graph shows a clear change according to the DC method. In contrast with all other models, the DC method was applied to the original data . Therefore, we kept the models derived from both datasets for our subsequent analyses to conduct additional comparisons.One issue with examining control with nonsimulated data is that the labels need to be established by an expert human rater. However, one can never be certain that the effects detected on the nonsimulated data represent true effects. Therefore, we conducted further analyses with simulated datasets, which were never used during training.x was the univariate times series, t an index of time , a the autocorrelation value, ε a normally distributed error term with a mean of 0 and a standard deviation of 1, and c a constant of 10 to avoid graphs with negative values. The autocorrelation value varied across graphs according to a normal distribution with a mean of .20, a standard deviation of .15, and a maximum of .80, which are based on the values reported by Shadish and Sullivan  between 3 and 6 for Phase A, and between 5 to 10 for Phase B.To examine generalization to simulated datasets, we used the best models as trained and validated using the nonsimulated data  to make our predictions. In particular, our analyses monitored accuracy, Type I error rate, and power. Accuracy represents to what extent the predictions match the true labels of the datasets . Computing accuracy involved adding the number of agreements between the true labels and the predictions of the models  and dividing the sum by 20,000. Type I error rate represents false positives. In the analysis of single-case design, a false positive occurs when a behavior analyst erroneously concludes that a graph shows a clear change when no real change occurred . Computing Type I error rate consisted of dividing the number of disagreements between the true labels and the predictions of the models on the graphs showing no clear change (according to the true labels) by 10,000. Power represents the extent to which a prediction identifies an effect that is present. In our case, power is the proportion of time the model detects a clear change when a real change did occur. To compute power, our program divided the number of agreements between the true labels and the predictions on the graphs showing a clear change (according to the true labels) by 10,000.Table Overall, our proof of concept indicates that different models developed using machine-learning algorithms may produce higher accuracy than the commonly used DC method. On the nonsimulated datasets, the models derived from machine-learning algorithms consistently showed better correspondence with visual analysis than the DC method. All our models yielded higher accuracy and power than the DC method on nonsimulated datasets. Moreover, the models produced their best outcomes on all measures when analyzing graphs with varying phase lengths. Each machine-learning algorithm performed differently across datasets with no one consistently outperforming the others.To our knowledge, our study is the first to examine the validity of using machine learning to analyze single-case data. The analyses clearly showed that the learned models produced generally acceptable control over potential errors in interpretation. It is interesting that our models produced the best accuracy and lowest Type I error rate on the simulated dataset with phases of variable lengths. This observation is consistent with other studies that have shown that graphs with more points per phase produce more accurate interpretations (e.g., Lanovaz et al., As more research is conducted, practitioners and researchers may eventually use machine learning to supplement visual analysis. As observed in the current study, machine-learning models may produce more accurate interpretations of data than the DC method. If the labels were established by a group of experts using consensus, the models could potentially be used as tools to train future practitioners and researchers on analyzing single-case data. Another potential use for our machine-learning models is as decision-making tools in mobile health apps designed to teach or reduce behavior (Knight, McKissick, & Saunders, Despite the promising nature of our results, our study has some limitations that should be noted. First, we restricted our algorithms to basic machine-learning models and manipulated only a limited number of hyperparameters as the purpose of our study was to examine whether machine learning could produce higher accuracy than a well-established structured aid to visual analysis. Future research should examine how accuracy can be further improved by manipulating other hyperparameters or using more advanced neural networks (e.g., recurrent neural networks, convolutional networks; Chung, Gulcehre, Cho, & Bengio,


Recent demonstrations have shown that frequency-diverse computational imaging systems can greatly simplify conventional architectures developed for imaging by transferring constraints into the digital layer. Here, in order to limit the latency and processing burden involved in image reconstruction, we propose to truncate insignificant principal components of the sensing matrix that links the measurements to the scene to be imaged. In contrast to recent work using principle component analysis to synthesize scene illuminations, our generic approach is fully unsupervised and is applied directly to the sensing matrix. We impose no restrictions on the type of imageable scene, no training data is required, and no actively reconfigurable radiating apertures are employed. This paper paves the way to the constitution of a new degree of freedom in image reconstructions, allowing one to place the performance emphasis either on image quality or latency and computational burden. The application of such relaxations will be essential for widespread deployment of computational microwave and millimeter wave imagers in scenarios such as security screening. We show in this specific context that it is possible to reduce both the processing time and memory consumption with a minor impact on the quality of the reconstructed images. Despite these advantages, conventional microwave imaging systems suffer from a number of challenges. In general, microwave imaging requires the synthesis of composite apertures to raster-scan the scene, either mechanically, such as in synthetic aperture radar16, or electronically, such as in phased arrays21. Mechanical scanning is not desirable in that data acquisition time can be significant, posing a major challenge for real-time operation. All-electronic operation can be achieved using phased arrays. However, beam synthesis using phased arrays requires that each antenna within the composite aperture has a phase-control circuit, or a phase-shifter, and preferably a power amplifier (to compensate for the insertion losses of the phase shifters). As a result, using phased arrays, all-electronic operation can require a large number of phase-shifting circuits and power amplifiers, significantly increasing the complexity and power-consumption of such apertures. Applications requiring high depth resolution will also be particularly constrained by the fractional bandwidth limitations of the active components of each transmission and reception chain.Imaging with microwaves brings about several advantages, such as all-weather operation, the use of non-ionizing radiation, the ability to see through most optically opaque materials and the availability of mature component technology. Due to these advantages, microwave imaging has proven to be useful in a wide range of applications ranging from security-screening27. Numerous proofs of concept of frequency-diverse computational imaging have recently been proposed35, where the scene information is encoded onto a set of measurement modes that exhibit quasi-randomness across the operating frequency band. In other words, the scene information is sampled on quasi-random bases by stepping through a number of frequency points across the operating frequency band (frequency sweep)36. Leveraging this quasi-random set of modes offered by engineered frequency-diverse systems eliminates the need for beam synthesis and requires a simple frequency sweep to achieve imaging. A three-dimensional (3D) microwave image of the scene can then be reconstructed by interacting the measurements of the scene with the transfer function of the frequency-diverse imaging system using computational reconstruction techniques, such as direct algorithms, including the matched-filter technique, or iterative algorithms, including the least-square technique37.To address these challenges, unusual modalities have been proposed that leverage ideas related to computational imaging32. It is reported in38 that adopting such a prior constraint on the imaging problem can reduce the number of voxels to be reconstructed, and hence the complexity of the imaging problem, by more than 90%. Another approach includes the parallelization of the image reconstruction algorithm using general-purpose graphics processing units (GPGPUs)39 and field-programmable-gate-array (FPGA) structures4. However, despite the significant reduction in the size of the computational problem enabled by applying the aforementioned techniques, the reconstruction time can be on the order of several seconds32.Using such frequency-diverse schemes for computational imaging of large scenes at microwave frequencies, such as in security-screening, can be a computationally demanding task. For a 3D scene discretised at the resolution limit of a finite-size aperture, the number of voxels to be reconstructed can be significant. Several techniques can be adopted to reduce the computational complexity of the imaging problem. For example, additionally using data from optical sensors (sensor fusion), the field-of-view for a given (microwave) imaging problem can be constrained to a volume bounded by the surface of the imaged object40. In the considered imaging problem, the sensing matrix links the space to be imaged to the measured signals; by decomposing the sensing matrix into its principle components, we can identify independent structures of which the sensing matrix is composed and their respective contribution levels. We then select only the most significant principal components in order to compute a model of the sensing matrix with reduced dimensions.In view of this, it is evident that additional techniques are needed to further reduce the computational complexity of the imaging problem for real-time operation. Inspired by well-known lossy compression schemes such as JPEG or MP3, we hypothesize that a moderate loss of information can be tolerated in order to substantially reduce the memory and processing burden. Principal component analysis (PCA) is a mathematical tool that can be used in this perspective to decompose a matrix on a set of orthogonal bases41 and42, is based on the radiation of illumination patterns specific to each type of scene to be imaged, necessarily requiring the use of reconfigurable systems for scene-dependent beam-synthesis. Provided a prior knowledge about the nature of the scene to be imaged is available, these authors have shown that it is possible to limit the number of sequential captures necessary for image reconstruction compared to the use of purely random patterns. Recent work in43 took this idea even further by directly integrating a model of the physical layer into an artificial neural network in order to jointly learn optimal measurement and processing strategies based on a priori knowledge of scene, task and measurement constraints. Since this "learned sensing" strategy enables one to minimize the acquisition of task-irrelevant information, it is highly task-specific and requires a supervised learning technique. In contrast to these prior works, the approach we propose here applies PCA to the sensing matrix rather than the expected scene. Consequently, our approach is not scene dependent and does not require the use of sequential measurements relying on active reconfigurable antennas. We propose a system-dependent but scene-independent method relying on a frequency-sweep to generate a succession of random illumination patterns that interrogate the scene to be imaged; via PCA we can limit the dimensionality of the sensing matrix and thereby the computational complexity of the image reconstruction.Our proposal is distinct from recent ideas in the literature to optimize microwave imaging systems using PCA. One such approach, proposed inThis paper demonstrates that our approach makes it possible to reduce computation times and memory consumption for imaging applications while having only a minor impact on the quality of the reconstructed images. Our approach allows each application to dispose of a new degree of freedom in image reconstruction by moving the constraints to either image quality or numerical performance. The outline of this paper is as follows: In “In frequency-diverse computational imaging, a set of frequency-dependent spatially-varying field patterns are used to interrogate the scene to be imaged. Using the first Born approximation, the measured signal is linked to the imaged object by means of the sensing matrix (or the transfer function) of the imaging system as follows:g denotes the measurement signal, i and j are the indices of the transmit and receive frequency-diverse antennas, and E-field patterns of the transmit and receive antennas propagated to the scene using the free-space Green’s function44. Finally, 45. Under the first Born approximation, the sensing matrix is proportional to the dot product of the transmit and receive frequency-diverse antenna fields: Equation  is the imentclass2pt{minimT transmit antennas and R receive antennas, and sampling the frequency band at F frequency points, the total number of measurement modes is D, target distance d, and wavelength N voxels to the M measurement modes. The discretized forward model is the following:Retrieving an estimate of the reflectivity of the scene mentclass2pt{minimIn , althougThe simplest techniques, known as matched filtering, consists in using the conjugate transpose of the sensing matrix, denoted 38. Therefore, it is evident that additional techniques are needed to solve the inverse problem of Eq.  source of information and that the sampling of the reconstructed image perfectly matches the resolution limit of this system. Under such ideal conditions, the SVD of the sensing matrix yields a decomposition with singular values of identical levels48, reflecting that 49.In , \documeentclass1pt{minimaN is lower than the number of measurement modes M, there are N non-zero singular values 50:N voxels of the scene, it is then possible to try to reconstruct only K coefficients corresponding to spatial correlations with the masks formed by the most significant singular vectors Since the number of reconstructed voxels K singular vectors, forming K most significant singular values. The measurement then takes the following form, revealing the principal components K coefficients corresponding to independent linear combinations of the information of the scene to be imaged:We now define a new matrix formalism of the imaging problem by selecting only the first The matrix Finally, taking advantage of the orthonormal properties of the Rather than storing the matrix 51 has shown that based on optical sensors the surface boundaries of the object under consideration can be detected to define a volume around this surface that constrains the domain for microwave imaging  factor can provide36, which is given by Figure The gains achievable in numerical performance by applying the dimensionality reduction of the sensing matrix that we propose inherently depend on the correlations between the utilized measurement modes. The considered imaging hardware has been optimized to generate a quasi-random set of measurement modes, meaning that an effort was made to keep the correlations low. This is evidenced by the singular value spectrum of the sensing matrix shown in Fig. In the following, two data processing scenarios are studied to illustrate the impact of the proposed technique on the quality of the reconstructed images and on the associated computation times. First, we consider a matched filter as example of a direct reconstruction technique; second, we consider the generalized minimal residual method (GMRES) as example of an iterative reconstruction technique.The first approach using a matched filter is based on a simple phase compensation of the sensing matrix. In the absence of any dimensionality reduction, image reconstruction by matched filtering takes the following form:N voxels or only reconstructing the selected subset of For reference, we show reconstruction results with matched filtering without the proposed PCA-based dimensionality reduction in Fig. Matched filtering not only has the advantage of reducing the dimensions of the matrices manipulated for image reconstruction (in comparison to other techniques), but also of limiting the impact of clutter related to imperfect estimations outside of the reduced region of interest. Using this first simple reconstruction method as a benchmark, the advantages and disadvantages associated with PCA-based dimensionality reduction are now investigated.A selection of four reconstruction examples with matched filtering is presented in Fig. Example B in Fig. 32.An analysis of the achieved reduction of the computational burden for the considered matched-filtering technique is shown in Fig. For direct reconstruction techniques like matched filtering, the PCA-based dimensionality reduction also yields a much more direct benefit related to memory usage. Real-time applications require the ability to store all matrices needed for computations in a fast memory; body-scanning applications are then particularly constrained by the volume of data used to calculate each image. The memory saving factor is defined by the following equation:M corresponds to the number of utilized measurement modes, N to the total number of voxels and K to the number of selected principal components. We emphasizee that the total number of voxels N is taken into account in Eq.  is applied, having yielded good performances in previous workember of :18\documBy minimizing ges Fig. .Figure 853.Compared to reconstructions performed only by matched filtering, magnitude-term compensation provides more accurate estimates of the imposed constant reflectivity function over the entire target. However, the specular nature of the interaction of waves with the target tends to create shadows that cannot be removed unless the target is illuminated from other incident angles. This physical limitation, also affecting conventional imaging systems, can nevertheless be mitigated by merging a set of reconstructions from different perspectivesAs in the previous section, we plot the convergence of the reconstruction quality in terms of the MSE for the use of different amounts of leading principle components in Fig. The effect of the PCA-based dimensionality reduction on the calculation time depends again on the number of selected principal components. Although the results obtained with 100% of the principal components are identical to those obtained by applying the same technique with the As before, we also compare the memory consumption with and without dimensionality reduction. For this purpose, the ratio of matrix sizes required to reconstruct images from frequency measurements alone is calculated for the iterative GMRES technique. If the full sensing matrix is used by GMRES, the quantity Our dimensionality-reduction technique hence enables us to limit memory usage as long as Having considered a more computationally intensive image reconstruction based on an iterative technique in this section, the utility of the proposed approach was even more evident. For example C . The elimination of these weakest components thus leads to the suppression of noise subspaces, implicitly realizing a regularization by truncated singular value decomposition58.This study concludes with an analysis of the impact of thermal noise usually associated with the use of active receivers. Previous work has highlighted the usefulness of PCA in the identificationThe previous sections have highlighted the beneficial effects of dimensionality reduction of sensing matrices on image computation times and memory consumption. As a continuation of these efforts, it is finally proposed to study the effects of additive measurement noise on the proposed PCA-based technique. Following the model given by Eq. , thermalFor a series of SNRs of noise-corrupted measured signals, a matched-filtering reconstruction is performed. The figure of merit is again defined as the MSE for each estimate, considering the noise-free reconstruction with As expected, the quality of the PCA-based reconstructions eventually converges to that of reconstructions based on the full sensing matrix 59. Finally, we note that the reconstruction techniques used in this paper are agnostic to the scene to be imaged, thus imposing no restrictions on the nature or on the sparsity of the region of interest. Future efforts may investigate whether the presented dimensionality reduction technique can be combined with structure-aware imaging algorithms60.Since computational imaging systems typically cannot ensure perfect spatial and spectral orthogonality of the utilized measurement modes, there is usually a non-negligible level of correlation between the obtained measurements. These correlations imply that upon a change of basis via singular value decomposition, many components of the sensing matrix contribute only weakly; neglecting such insignificant components thus entails only a minor deterioration of the imaging quality but offers significant reductions in memory usage and enhancements in computation speed. Although the suppression of the least contributing principal components generally results in a loss of quality of the reconstructed images, this study also highlighted that many of these suppressed components cannot contribute to the reconstruction under realistic operating conditions due to the presence of noise anyway. Our study illustrated the proposed dimensionality reduction of the sensing matrix with a case study of a specific computational imaging system based on a Mills-Cross aperture composed of metasurface antennas and a complex target (a human), considering both direct and iterative image reconstruction techniques. The finite spatial extent and intrinsic losses of the considered metasurface antennas result in notable correlations between the utilized measurement modes. Our results offer the possibility to trade-off memory needs and computation time with the image reconstruction quality, offering a new degree of freedom in computational imaging to place the emphasis on either reconstruction quality or numerical performance. Our method can readily be applied to other imaging systems and contexts that rely on sensing matrices with inevitable correlationsSupplementary Information 1.Supplementary Information 2.Supplementary Information 3.


Autonomous driving systems tightly rely on the quality of the data from sensors for tasks such as localization and navigation. In this work, we present an integrity monitoring framework that can assess the quality of multimodal data from exteroceptive sensors. The proposed multisource coherence-based integrity assessment framework is capable of handling highway as well as complex semi-urban and urban scenarios. To achieve such generalization and scalability, we employ a semantic-grid data representation, which can efficiently represent the surroundings of the vehicle. The proposed method is used to evaluate the integrity of sources in several scenarios, and the integrity markers generated are used for identifying and quantifying unreliable data. A particular focus is given to real-world complex scenarios obtained from publicly available datasets where integrity localization requirements are of high importance. Those scenarios are examined to evaluate the performance of the framework and to provide proof-of-concept. We also establish the importance of the proposed integrity assessment framework in context-based localization applications for autonomous vehicles. The proposed method applies the integrity assessment concepts in the field of aviation to ground vehicles and provides the Protection Level markers  for perception systems used for vehicle localization. The second half of the last decade has seen a significant emergence of commercially available vehicles with autonomous driving capabilities. We can confidently say that the status of autonomy in vehicles is well into the realm of Society of Automotive Engineers (SAE) level 2 . While tSemi-urban and urban environments often contain a multitude of intersections, roundabouts, road-splits, and merges compared to highway scenarios. As discussed in Traditionally, vision data is used to detect ego lane markings and/or lanes parallel to the ego lane using a curvature-based model. In urban scenarios, such lane detection models fail due to different types of lane markings , orientation  and complex curvatures . Another approach using visual data is to detect the drivable road region in front of vehicle. But due to the unforeseeable shapes of possible road segment detections, modeling of such output with a geometrical model is difficult. Intersections with multilane branch roads can have a large common region at the center, which can limit the observability of other road branches through visual inputs.It is reasonable to assume that vehicles travel slowly and stop more often in semi-urban and urban scenarios than highways. GPS receivers are proven to have poor performance in slow-moving vehicles . CombineWith the exception of a few advanced and proprietary Geographic Information Systems , publicly available GIS sources lack accurate road properties  and strongly depend on rule-based rendering to display maps. The discrepancies observed while overlapping the satellite view and rendered map structures from different GISs as shown in While data from vision, GPS, and maps add complexities and impose limitations, LiDAR, on the other hand, can provide useful data in urban and semi-urban environments. It can observe the ego road and other road branches efficiently. By using the reflectivity information available in LiDAR data, we can detect bright surfaces like lane markings and curbs . Though The integrity monitoring method in  providesThe paper presents the following contributions based on the problem statement outlined above.Defining a common reference frame and formalizing a common model to represent all data sources in all scenarios.Prototyping an integrity assessment framework using the common model and providing proof of concept.Analyzing the performance of the proposed framework using publicly available datasets and comparison with other state-of-the-art integrity monitoring solutions from the literature.The framework proposed for the integrity assessment developed in this work is given in The purpose of the Detection Block is to extract the same information (features) from each data source. From the literature review, we identify three features that are most commonly used in state-of-the-art localization methods in urban scenarios—lane markings, drivable roads, and the structure of the surroundings of the vehicle. Here, we limit the surrounding structures to grass patches/vegetation and curbs and avoid building facades and other objects due to the complexities of their detection. Indeed, any feature can be used in this process if it is detectable from every data source considered. The methods used to detect these features from each source are explained here.To accommodate varieties of lane markings present in different scenarios, all possible markings are detected. Images from cameras are transformed to bird’s-eye view (BEV) using camera calibration. Intensity-based segmentation is used to detect all possible white lane markings. After detection of all the candidate lane markings, blob analysis is used to reject poor detections . Seed-baA subset of LiDAR data containing points that lie inside a 3D region of interest (ROI) is selected. Points on the road and on the edges of the road are classified using 3D gradients. The ROI is divided into smaller patches in the XY plane, and the points belonging to each patch are examined for their Z values. This helps to differentiate between road segments, curbs, dividers, vegetation, etc. using the technique presented in . Points OpenStreetMap (OSM) is used in this work as a GIS source. OSM provides nodes corresponding to ways, grass patches, and railings, etc. However, finding relevant geometrical information in a vehicle’s surroundings from maps involves two key components: location and orientation of the vehicle . Using li represents a 20 × 20 × 100 cm block in the real world. Four feature labels (LBs) are assigned to cells in the FG according to the type of feature: (1) LBr—road, (2) LBl—lane marking, (3) LBo—other surfaces, (4) LBu—unclassified/unidentifiable. The blocks corresponding to each of the cells are examined for the information they contain. The type of feature with the highest ratio inside a block is used to assign the respective label to the cell. Each data source produces an FG following this criterion, as shown in To be able to deal with the features and geometries of different types and shapes, a 2D feature grid (FG) is proposed as the model. FG consists of an array of cells where each cell cAlong with labels, it is important to model the confidence of data provided by each sensor. The accuracy of LiDAR data decreases as the distance from the sensor to the measurement location increases . On the Hence, the total representation of data from sources will have two components: the labels and their importance, which is denoted by the weights. Other source-specific weighting functions using homography of image transformations and LiDAR data acquisition models can also be used for this purpose. However, data sources like maps use uniform weights for all the cells in their FGs due to the fact that they are not measured but just extracted.The treatment of different sensors as multimodal data sources with a common frame of representation and the same dimensionality allows us to use the definitions of integrity presented in the domain of data sciences. Integrity measures overall accuracy and consistency of data sources . While aiFG and u. After computing the matches between all of the possible combinations, the integrity associated with a source is computed asLet The integrity analysis mentioned in t on map FG  at first by iteratively distributing particles around the best match localizations. The lateral offset estimation x  and In the sequential localization optimization, coherence between the map  than loThis work presents a framework for integrity monitoring of sources used in the localization of autonomous vehicles. The limitations of common geometrical models in representing multimodal data sources are identified in this work. To overcome these issues, a semantic feature grid model is proposed that can geometrically represent different features using labels. A function for coherence evaluation between feature grids is formalized to iteratively optimize the localization as well as to assess the integrity of data sources. The framework is tested using different scenarios from datasets, and the results show the versatility of the proposed model, which is able to provide reliable and consistent integrity estimation in highway as well as semi-urban and urban environments. This method is proven robust against inconsistencies in feature detections such as partial detections, occlusions, and poor map rendering. The method presented claims scalability since it can be implemented with any number of sensors and digital map sources. The only requirement for the applicability of this framework is the ability to detect common features from all of the data sources and represent them geometrically in the proposed feature grid representations. This work also illustrates how classical integrity markers like protection levels can be transposed for perception data sources used in autonomous vehicles.The rule-based map rendering technique used in this method is observed to be contributing several inconsistencies, which makes it difficult to isolate map rendering errors from GPS positioning errors. We propose the use of high-definition maps, which are enriched with globally localized lane-level information, to address this issue. Accurate maps will improve the coherence estimation between features detected from other data sources such as stop lines, pedestrian crossings, lane merging information, road structure information, etc. It will also be important to study map-rendering techniques that improve integrity multi-source perception analysis by including precise building footprints, road width information, lane markings, and traffic sign localization.