In this paper, a novel approach to Non-Destructive Testing (NDT) of defective materials for the aircraft industry is proposed, which utilizes an approach based on multifrequency and spectrogram eddy current method combined with an image analysis method previously applied for general-purpose full-reference image quality assessment (FR IQA). The proposed defect identification method is based on the use of the modified SSIM4 image quality metric. The developed method was thoroughly tested for various locations, sizes, and configurations of defects in the examined structure. Its application makes it possible to not only determine the presence of cracks but also estimate their size. Structure diagnostics in terms of damage presence determination is a vital factor in aircraft maintenance systems. At present, most of the used diagnostic techniques include mainly visual inspections and those using Non-Destructive Testing–NDT . Such meThe main phenomena that may lead to catastrophic aircraft damage are corrosion, stress corrosion cracking, and fatigue cycle cracking . In mostTesting with the eddy current method has a comparative character and requires calibration. It constitutes a significant limitation, especially in diagnosing complex geometry elements and multilayer ones . On the According to the industry survey of structural health monitoring technology and NDT usage , crack dAs Mottl proposed using the Dodd and Deeds model , detectiSeveral types of probes can be used for damage position, size, and orientation determination: manual , rotating and sliding .Several constraints in selecting the proper inspection approach should be considered, such as time of inspection, geometrical implications, surface treatment conditions, the possibility of damage detection, and sensitivity ,15.Therefore, an application of the NDT methods for materials used in the aircraft industry is one of the most demanding areas of research in the era of Industry 4.0, integrating the knowledge of electromagnetic measurements, particularly the eddy current method, with data analysis methods originating from signal and image processing domain.In many cases, the results of non-destructive measurements take the form of 2D or 3D images, and therefore image analysis methods are of particular interest. Regardless of the potential use of image quality assessment (IQA) methods for an automatic evaluation of the usefulness of individual images or measurements for further analysis, some of the IQA metrics might also be applied as tools for detecting the presence of material defects or even for their classification.Unfortunately, the applicability of many full-reference (FR) metrics, considered the most popular and the most universal, may be limited by assuming the necessary knowledge of the reference image, which may be unavailable in many practical applications ,17,18,19The rest of the paper is organized as follows: In An example of non-destructive technologies for structural integrity monitoring of the aerial components considered in this paper is the eddy current inspection performance on the PZL-130 Orlik aircraft structure. Within the conducted Full-Scale Fatigue Test (FSFT) for that aircraft, a program of detailed NDT inspections was scheduled. A part of that program was the Eddy Current inspection delivery, especially for critical parts of the wing structure. Inspection delivered during the FSFT revealed the existence and development of more than 100 failure modes, which were detected and described.According to the best practice standards, the exemplary failure modes that can be found during the aircraft inspection are as follows : hole pe® mobile system and C-scan data presentation. Such presentation mode shows the signal value (impedance module) as a function of the sensor location.Several eddy current techniques were implemented for damage detection, such as high-frequency reflection probes, high- and low-frequency automated scanning, and the magneto-optic imaging MOI™ system for the general and detailed structure visualization. The inspection was delivered with the 20 kHz for surface-breaking cracks and 1 kHz for the subsurface, consecutively. Scan resolution (pixel-to-pixel size) was equal to 1 mm. Based on collected data, the NDT operator provides data analysis using visual recognition of the anomaly, which may exist on the provided image. That analysis is being supported by signal classification and description tools embedded in the system. Moreover, the data export function enables more advanced analysis based on signal processing methods .Such an inspection method has also benefited from data comparison possibility, a valuable function for the FSFT to monitor damage nucleation and damage progression. A more detailed analysis is possible for specific regions of interest (ROI) with the complex impedance plane analysis, signal evaluation, and size quantification tool.Once the FSFT inspection was completed, the need for an increase in the damage detection sensitivity, especially in the second or third layer, has arisen.The approach to the detection and classification of damage considered in this article is based on using a probe adapted to detect damage in multilayer aluminum aircraft structures, an advanced multifrequency inspection method, and the mutual image similarity assessment.From the observations and practical inspection, the use of a dedicated probe with the proper signal-to-noise ratio and efficient penetration depth is a required tool for detailed aircraft inspection. In the case of structures with multiple layers, it is also crucial to utilize many frequencies.Analysis of the various signals achieved from all the frequencies is a complicated and time-consuming procedure. For this reason, the automation of the defect detection and identification process is essential for the proper and effective functioning of the inspection system.In the classical ECT systems, the use of several frequencies requires the application of several probes with valid central frequencies to obtain the estimated depth of penetration. Additional limitations also come from a decrease in the current density with an increase in depth of penetration. Such phenomena reduce inspection reliability and affect the probability of damage detection. Thus, the inspection of multilayer structures and geometry constraints requires specialized probes and signal processing to keep linearity of damage detection and thickness change. In the proposed multifrequency ECT system, many of these limitations were overcome.In the eddy current method, the measurement sensitivity depends on several factors, such as the transducer dimensions, an excitation frequency, and properties of a tested structure (permeability and conductivity). Unfortunately, high sensitivity and a high resolution usually cannot be achieved at the same time. Enlargement of the probe dimensions will result in higher sensitivity (for deeply loaded defects) and a lower spatial resolution. Decreasing the excitation frequency will cause a similar effect. In testing multilayer aircraft structures, high and possibly similar sensitivity for detecting defects in different layers and at different depths is required. At the same time, it is necessary to maintain the appropriate spatial resolution. It is crucial to detect even the most minor cracks in the vicinity of holes after riveted connections. In this case, it is particularly advantageous to use a measuring system based on the Massive Multifrequency Excitation and Spectrogram Method (MMFES), proposed in . This meEs–waveform of the excitation signal, n–number of harmonics, iU–amplitude of the i-th component, if–amplitude of the i-th component, iϕ–phase of the i-th component.The measurements were carried out using the computer system implementing the Massive Multi-Frequency Excitation and Spectrogram (MMFES) method. In the MMFES system, a complex waveform is used as the excitation, created by adding sinusoidal components with precisely defined frequencies and amplitudes. The formula which describes the function defining the excitation voltage is as follows:n is selected in such a way as to achieve the highest possible signal to noise ratio at the eddy current transducer output. The relation of individual harmonics amplitude to the expected noise level and the possibility of filtering signals in the frequency domain are considered. The initial phases of individual harmonics do not significantly affect the ability to detect defects, but it is possible to limit the maximum value achieved by the excitation signal by appropriate selection. One of the critical issues is the selection of the excitation harmonic amplitudes. They are experimentally determined so that the harmonic components of the signal measured at the transducer output have the same value when it is located over the tested material that does not contain heterogeneity. The amplitude equalization process usually requires two to three iterations until the differences are satisfactorily reduced . The experimental process allows eliminating the influence of the transducer frequency characteristic, lead wires, and other factors such as a lift-off and the inclination of the transducer concerning the tested material surface. The plot of excitation voltage measured on the output of the D/A converter as a function of time is shown in The frequency range is selected in such a way as to include the values for which the maximum interaction of eddy currents with defects occurs at possible depths of the tested structure. The number of harmonic components The excitation driving voltage is generated in the D/A converter  and then fed through the high-frequency power amplifier  to the excitation coils of the eddy current transducer.The voltage generated in the measuring coil of the eddy current transducer is amplified and filtered with the Kronhite 3988 and then digitized in the NI PXI 5922 A/D converter . Due to the high resolution and accuracy of the A/D converter, which is achieved while maintaining a relatively high conversion rate, it is possible to obtain precise information about minor changes of amplitudes of the harmonic components in the processed signals. In these measurements, the sampling rate was set to 315 kS/s, which allows a resolution of 24 bits to be achieved. One of the critical issues in the MMFES system is to minimize spectral leakage. Therefore, the sampling frequency is precisely selected, considering the harmonic frequencies present in the excitation signal. Additionally, a clock that controls the A/D converter is used simultaneously to control the D/A converter generating the excitation. The block scheme of the MMFES system is shown in Signals obtained in the Ultrasonic Testing (UT) are commonly displayed in three different formats. The formats are known in the NDT terminology as A-scan, B-scan, and C-scan presentations. Each presentation format provides different information about the region of material being inspected. The A-scan presents the amount of received ultrasonic energy as a function of time for a single position of the transducer. Peaks visible in the A-scan presentation allow to estimate the location and size of the discontinuity in the inspected material. Similar information can be achieved from the frequency characteristic obtained in the MMFES ECT system. Therefore, we will use the term A-scan for such kind of data presentation despite some significant differences. An example of the MMFES ECT A-scan in the 300 Hz–2 kHz range is shown in x, y) of the transducer and showing values for a selected time of acquisition. Finally, D-scan is a three-dimensional representation of the signals measured during  scanning.In the UT method, the image composed of A-scans captured during the transducer’s linear scanning is called a B-scan. From the B-scan, it is possible to identify the depth of the discontinuity and its linear dimension in the scanning direction. A C-scan is an image obtained during two-dimensional scanning . The plot of the signal value for a single frequency as a function of the transducer x and y positions  corresponds to a D-scan.Finally, the signal value three-dimensional plot against frequency The C-scans taken for the selected frequencies will be the input images to the automatic defect recognition system using image quality assessment based on similarity.Inspection of the aluminum multilayer structures required a transducer characterized by high sensitivity, deep penetration, and the ability to detect defects in all directions with similar sensitivity. The eddy current differential transducer a consistAE, BE, CE, DE) with 25 turns each are wound on the remaining columns. The excitation coils (AE and BE) are connected in series, as are the CE and DE coils . There were also long cracks in the second layer, but it was observed that some of the cracks were less than 2 mm long.This research aimed to develop a universal identification algorithm that works appropriately with defects of various sizes and orientations. However, as mentioned earlier, aircraft specimens contained a relatively limited set of naturally formed defects. For this reason, it was decided to produce aluminum specimens with a similar structure but with defects of a much more different nature, both in configuration and length.Each sample consists of three 1 mm thick aluminum alloy plates joined together . The tesIn the research, we used 40 different samples, which were created from the assembly of the presented four plates with artificial defects in various configurations . The use of various variants of the arrangement of basic defects creates a chance to test the system in a fairly wide range. However, it is planned to further expand the database of samples with artificial defects to verify responses in less frequent situations. At the same time, work is underway to collect a set of samples taken from real objects, but unfortunately, the number of experimental samples is limited by their availability. The defects found there are not very diverse and most often have larger dimensions than those that we would like to detect during routine inspections. For this reason, it is crucial to operating in two ways, collecting actual samples and using samples with artificially generated defects.x and y-direction. The measurements were made using the multifrequency excitation signal consisting of sinusoidal components with frequencies from 315 Hz up to 1995 Hz (17 frequency components). The lift-off (the gap between the specimen and the sensor) was measured to be 0.4 mm.The measurements were performed by scanning the probe over the area surrounding the hole in steps of 0.5 mm in The resulting C-scans for the lowest excitation frequency are presented in y axis  achieved for the other frequencies were not significantly different. Therefore, only one frequency component was selected for the presentation.Examples of the eddy current C-scans for a specified ROI using a single frequency are shown in the previous subsection as color plots. Nevertheless, the type of measurement data obtained as the C-scans is single-channel, similarly as in the case of greyscale images, although the data range is different. Hence, their color representation is unsuitable from the image processing point of view, where various three-channel color spaces are applied to analyze color images. Additionally, most state-of-the-art image quality assessment (IQA) methods, described further, were developed for monochrome images. Therefore, the obtained results were converted into grayscale 2D images, using 8-bit values in the range 0–255, for which the considered IQA methods may be directly applied.Since the expected features of such obtained images are their symmetry and self-similarity, as it may be observed in General-purpose image quality assessment methods, including those used in our experiments, may be considered as useful approaches for many applications where the image or video quality plays an important role, e.g., making it possible to skip the degraded frames during the analysis of video sequences or to optimize some newly developed image filtering or lossy compression algorithms.On the other hand, the universal full-reference IQA methods, based on the comparison of an undistorted reference (“pristine”) image with the degraded one, may also be used for the detection of some types of contamination or image distortions as well as the estimation of their amount or level.Although in many applications, where the reference image might be unavailable, the most desired approach would be the no-reference (NR) IQA, also referred to as “blind”, such methods are usually quite complex, and their performance is lower than state-of-the-art FR IQA metrics. Usually, such NR metrics are sensitive only to a limited number of distortion types, hence their universality is relatively low.Nevertheless, the development of new IQA methods during recent years ,18,19,24The most popular FR metric, known as Structural Similarity (SSIM), was proposed in , being tx and y, representing two fragments of the compared images for the specified location of the sliding window, may be expressed as the product of three components:x and y, respectively, and L is the dynamic range of pixels values. The final SSIM result is obtained as the average of local values calculated for all window positions.The simplified idea of the calculation of the local SSIM value between the blocks Following the first success of the SSIM, many similar approaches and extensions of this metric were proposed by various researchers during recent years, starting from its multi-scale version, known as MS-SSIM , making The same general approach is also used in the Edge Strength Similarity metric , as wellNevertheless, although the performance of most of these improved metrics is significantly better in comparison to the “original” SSIM for many IQA benchmarking datasets, their application for some other types of images, e.g., containing multiple distortions  or for tA and its neighboring blocks B according toB are limited by a specified distance (the default parameters are search area 19 × 19 pixels and the block size M = N = 9). Nevertheless, the blocks that contain the whole central part consisting of 5 × 5 pixels block are excluded from the calculations of the predictability measure A remarkable extension of the SSIM formula was proposed by Ponomarenko et al. , where tsure εA2  as illusAlthough all SSIM-based metrics utilize the estimation of image similarity, they cannot be applied directly for the analysis of images obtained as the results of measurements considered in this paper. Hence, their modification related to the calculation of the mutual  similarity between the image fragments is postulated.The concept of similarity calculation between two images without using reference images was also used in blind image quality assessment, although in this case, the use of the pseudo-reference image (PRI) is necessary ,42 is baNevertheless, the approach proposed in this paper is different since it does not require any comparisons with any other images, and the only required data for the mutual similarity calculation is the image that is the result of eddy current measurements.As the samples without any defects should provide symmetrical images obtained as measurements, the first necessary task is to detect the origin as the measurement results are not necessarily centered on the image plane. Unfortunately, considering the influence of defects and the presence of the measurement noise, the application of moment-based methods, e.g., the Centre of Gravity, has not led to satisfactory results. Nevertheless, the proper results for all considered samples, i.e., the detection of an individual pixel representing the center of symmetry of measurement data, were obtained employing the correlation between the original image and its horizontally or vertically flipped equivalents. After determining the locations of the maximum correlation values, the images are shifted and cropped, forming four quadrant images, which may be used as inputs for mutual similarity calculations.Before the symmetry detection, the grayscale images were initially normalized to the range 0–255, where 0 corresponds to the lowest measured value, and 255 represents the highest peak value. To decrease the influence of the highest peaks on the obtained results and make it possible to detect the presence of smaller defects, which influence the lower values the additional thresholding and normalization were conducted, equivalent to image saturation, assuming the preservation of only the lower 20% of the original range for simplicity the threshold value equal to 50 was chosen for 8-bit grayscale images). Such obtained images, divided into four quadrants expected to be symmetrical, form the input data for the mutual similarity calculations.Since most of the FR IQA metrics were proposed assuming that the compared images represent the same scene and only one of these images is corrupted, they might also be applied for comparison purposes, assuming that both images are highly similar. Another essential requirement is the proper registration of both images; hence the precise determination of the center of symmetry is a crucial issue, influencing the obtained results.Studying the obtained experimentally images, both horizontal and vertical symmetry may be easily observed as expected for the images without defects. Hence, the division of images into four fragments was proposed making it possible to calculate four similarity factors between each part and the flipped neighboring fragment, as shown in A similar approach to mutual similarity calculation was also used for the aesthetic quality evaluation of the 3D printed surfaces . HoweverDuring the conducted experiments, several similarity-based IQA metrics discussed in Nevertheless, it should be noted that the direct application of the FR IQA would be impossible without the knowledge of the reference images of metal structures without cracks which should be precisely registered with the analyzed images. Therefore, we proposed a novel approach assuming the calculation of the mutual similarities reflecting the expected symmetry of the images obtained using the multifrequency eddy current method. As the typical methods used for this purpose failed in the initial experiments, we adapted some metrics that are well-known and commonly accepted for general-purpose IQA, although the application of the Structural Similarity and the other IQA metrics for the classification of samples required the use of the average and maximum values of the mutual similarities obtained for 17 frequencies, not just the simple calculation of their values.During the conducted experiments, several similarity-based IQA methods were analyzed and verified towards their potential application as the similarity measures in the investigated problem. To illustrate the non-applicability of most of the considered metrics the results achieved for the PSIM, assuming the use of six mutual similarities, are presented in Similarly, the application of some other metrics considered in the paper, except the proposed application of the SSIM4, has not led to satisfactory results, regardless of the analysis of the maximum, minimum, average, or median values of the mutual similarities.Much better results, presented in Even better discrimination may be observed using six mutual similarities, as shown in Since some samples used in experiments contain multiple cracks, they were marked with black hexagons. As it may be observed, there is a relation between the size of the crack (as well as its depth) and the values of both considered features, as well.Nevertheless, correct classification of samples into high-quality ones and those containing some cracks may be conducted using the product of two considered features (as well as their sum), particularly assuming the calculation of six mutual comparisons.f = 630 Hz) than in the case of the defect located in the second layer (f = 1050 Hz). This effect can be further emphasized by using a wider range of excitation frequencies. It is crucial in the case of surface defects where the applied frequencies should be much higher. Nevertheless, due to the limited maximum excitation frequency, the plot of the frequency characteristics for the defect in the first layer did not contain the extreme point. This curve is not shown in The applied multifrequency method allows for even more detailed identification of defects. Thanks to the access to information obtained at many frequencies, it is possible to determine the layer of the structure containing the damage more precisely. The broader range of excitation frequencies (and higher maximum excitation frequency) may cause lower sensitivity to defects occurring in the third layer or below. The provision of the ability to detect deeply located defects has resulted from specialists’ requirements from the aviation industry.Currently, the eddy current techniques used for damage detection in the aerospace industry require impedance plane analysis or C-scan data visualization to evaluate damage existence. Each data collected must be evaluated by the qualified operator. Such an approach requires a relatively large amount of time for analysis and data comparison. Moreover, the sensitivity of inspection is decreased with the use of lower frequencies. Therefore, the application of the proposed method is interesting for the further development of an efficient damage detection methodology in the aerospace industry.The solution proposed in the paper makes it possible to determine the presence of cracks in the aircraft structures using the mutual image similarity analysis of images obtained as the result of the multifrequency eddy current measurements. After validating the usefulness of several image quality assessment metrics, the results achieved by the application of the modified SSIM4 method were encouraging, allowing not only for the detection of structural defects but also for the estimation of their size.A relatively large number of experiments with defects of various sizes, configurations, and locations in the studied structure were carried out. The possibility of both single cracks and complicated, complex damages with cracks propagating in many directions was considered. Such a complementary set of data made it possible to thoroughly test the proposed solution.2 of the specimen is around 150, the analysis using non-optimized Matlab code will take about 15 min. Analysis can partially overlap with the measurement process. The time can be considerably reduced using GPU and parallel computing for data acquisition  and defect detection. Another aspect is the time needed to perform the measurements. A significant limitation, in this case, is the lowest excitation frequency, as it is necessary to measure at least half of the period. In the case of the excitation frequency as low as 300 Hz, the measurement of 1 m2 with a resolution of 0.5 mm will take at least 50 min.The experimental verification of the proposed approach confirms its potential industrial application, mainly due to a fast calculation of the quality metric compared to the total measurement time. The computation of the final combination of two features based on six mutual similarities using the SSIM4 metric together with visualization takes a few seconds. The calculations of the mutual similarities for different layers and areas may be easily parallelized as well. Assuming that the number of rivets in 1 mAccording to our best knowledge, the proposed method is the first attempt to use image quality assessment methods to analyze eddy current measurements, particularly concerning the application of the multifrequency eddy current method to the inspection of riveted structures. In contrast to many popular deep learning approaches, our method is explainable as its main assumption is based on detecting the expected symmetry and its numerical estimation related to the size of the cracks.


Automatic detection of low-magnitude earthquakes has become an increasingly important research topic in recent years due to a sharp increase in induced seismicity around the globe. The detection of low-magnitude seismic events is essential for microseismic monitoring of hydraulic fracturing, carbon capture and storage, and geothermal operations for hazard detection and mitigation. Moreover, the detection of micro-earthquakes is crucial to understanding the underlying mechanisms of larger earthquakes. Various algorithms, including deep learning methods, have been proposed over the years to detect such low-magnitude events. However, there is still a need for improving the robustness of these methods in discriminating between local sources of noise and weak seismic events. In this study, we propose a convolutional neural network (CNN) to detect seismic events from shallow borehole stations in Groningen, the Netherlands. We train a CNN model to detect low-magnitude earthquakes, harnessing the multi-level sensor configuration of the G-network in Groningen. Each G-network station consists of four geophones at depths of 50, 100, 150, and 200 m. Unlike prior deep learning approaches that use 3-component seismic records only at a single sensor level, we use records from the entire borehole as one training example. This allows us to train the CNN model using moveout patterns of the energy traveling across the borehole sensors to discriminate between events originating in the subsurface and local noise arriving from the surface. We compare the prediction accuracy of our trained CNN model to that of the STA/LTA and template matching algorithms on a two-month continuous record. We demonstrate that the CNN model shows significantly better performance than STA/LTA and template matching in detecting new events missing from the catalog and minimizing false detections. Moreover, we find that using the moveout feature allows us to effectively train our CNN model using only a fraction of the data that would be needed otherwise, saving plenty of manual labor in preparing training labels. The proposed approach can be easily applied to other microseismic monitoring networks with multi-level sensors. Automatic detection of low-magnitude earthquakes has been a longstanding research problem in seismology. Although earthquakes are known to be clustered in time and space, the underlying processes that connect one earthquake to the next are still poorly understood. The challenge comes from our limited capability to detect small earthquakes reliably . Given tOne of the earliest approaches to automate seismic event detection was the STA/LTA algorithm . The algTo enhance the detection of events for low SNR waveforms, the template matching technique  has beenSeveral other techniques have been proposed over the years to tackle the problem of automatic event detection. Yoon et al.  proposedDeep learning has emerged as a powerful technique to tackle longstanding problems in various data-intensive fields such as computer vision and speech recognition. The ability of deep learning to recognize complex structures in high-dimensional data makes it attractive over other conventional algorithms . In imagImproved performance over conventional methods and ease of implementation have encouraged applications of deep learning on various seismological problems, including passive seismic event detection . Perol eDetection and study of these, predominantly low-magnitude, events caused by induced seismicity help in understanding subsurface mechanisms and assessing hazard probabilities. Another region that suffers from the problem of induced seismicity is Groningen, the Netherlands. Groningen has suffered from induced seismicity over the past three decades, caused by production from the Groningen gas field—the largest gas field in Western Europe. Continuous gas production since the 1960s led to reservoir compaction, which has been causing induced seismic events since 1991 .Since the turn of the century, these events have significantly increased in both the frequency of occurrence and magnitude. These seismic events pose a danger to the lives and properties of residents in Groningen and neighboring areas. The recurrence of these events forced the Dutch government to announce the stopping of production by 2022 , causingIn this study, we use data recorded by the shallow borehole network in Groningen, known as the G-network. The network consists of borehole stations that contain multi-level geophones at increasing depths of 50 m. Since CNNs are well-known for identifying spatial features in data, we use the moveout pattern of the energy traveling across the borehole sensors at a single station as the main distinguishing factor between events originating in the subsurface and noise coming from the surface. This is often the most challenging problem in detecting low-magnitude earthquakes. On a single sensor, earthquake energy coming from below or energy coming from a local noise source on the surface may look the same, leading to difficulty in classification. In contrast, the two sources of energy will show opposite moveout patterns on a multi-level sensor network. Just as a human interpreter could visually separate true events from coherent noise by picking the moveout, i.e., up-down (noise) or down-up (event), we train a CNN for this classification task. Recognizing these moveout patterns increases the network’s resolution in detecting events of lower magnitudes and in minimizing false detections.https://github.com/AhmedShaheen01/GroningenNet (accessed on 27 October 2021).We test the trained CNN model on two months (December 2017–January 2018) of continuous records at five stations near Zeerijp—the site of a major event  at its seismic and acoustic data portal . In partAll time-windows undergo pre-processing before being used to train the CNN model. The time windows are detrended, demeaned, and bandpass filtered between 5–25 Hz. In agreement with the analyses of Wyer et al.  and MukuFurthermore, to help the CNN model generalize better to variations in the test data, we augment our training event windows by adding time shifts such that the first-break time is varied with respect to the start of the window. This step is necessary to avoid any potential bias towards events that have first-breaks at particular time-samples within an event window. For each event window, we generate an additional sixteen windows by moving the window back and forth with a randomly generated time-shift. The output after the afore-mentioned pre-processing workflow is a 30 s time-window labeled into either signal or noise. The shape of each training data is a 3D matrix whose dimensions are: 4 (number of velocity geophones) × 3001 (time-window length × sampling rate) × 3 (the 3-components of each sensor). This is an analogous set up to the one used in the classification of color images, where the first two dimensions are the image length and width, and the third dimension represents the three color channels . The total number of labeled examples is 67,847, with 21,624 belonging to the event class and 46,223 belonging to the noise class. These labeled examples are split according to the following distribution: 60% for training, 20% for validation, and 20% for testing.Convolutional neural networks (CNNs) are inspired by studies on the visual cortex of cats and monkeys ,24. A CNA convolutional layer is a key part of a CNN, where feature extraction takes place. The layer extracts features from data by convolving filters of small dimensions with the input data. Then, the result of this convolution process is passed through a non-linear activation function. The output after the activation function is referred to as a feature map. A feature map carries important information from the input data after being filtered. A convolution layer can produce tens of feature maps, each containing a certain trait of the data that helps in efficiently performing the task at hand, be it a classification or a regression problem.The convolutional layer produces a huge amount of data that need to be scaled down for computational efficiency without losing important information before proceeding further. Therefore, each convolutional layer is followed by a max-pooling layer. This layer reduces data dimension by taking the maximum value of neighboring inputs in the feature maps. Getting the maximum ensures that the most important values are passed on. The reduced output of the pooling layer is passed to the subsequent convolutional layer to obtain more abstract features, followed by another pooling layer, and so on. This sequence of convolutional and pooling layers continues as many times as necessary to build highly complex features. Then, the output of the final pooling layer is flattened and passed into fully connected layers before passing on to the final layer that outputs the result. The convolutional filter coefficients and network weights are learned through the process of backpropagation .Our proposed CNN architecture consists of 3 convolutional layers. Each of them is followed by a max-pooling layer. The output of the last pooling layer is flattened and passed into two fully connected layers, followed by the output layer, which consists of a single neuron outputting whether the input is an event (1) or noise (0). The activation function used in all layers, except for the final layer, is the rectified linear unit function (ReLU). The final layer uses the sigmoid activation function. We use the Adam optimizer, and the loss function used to compute the error is the binary cross-entropy. The CNN model is implemented using the Tensorflow library  in PythoTo evaluate the performance of the trained model on continuous data, the model is tested on recordings from five stations  during the months of December 2017 and January 2018. In these two months, 23 events are recorded in the catalog, including a major event near Zeerijp . We require a given time-window to be flagged as an event class on at least two of the five stations for it to be classified as an event. This helps us reduce the possibility of false alarms and ensures the robustness of the CNN predictions. We use the available event catalog and manual verification to analyze the accuracy of these predictions.For comparison, we run STA/LTA with several detection thresholds on the same continuous data coming from the selected five stations. The four velocity geophones at each station are stacked to enhance the SNR. This stacking is done to improve the STA/LTA’s detection accuracy. Similar to CNN, to be classified as an event, we require at least two of the five stations to cross a given threshold. The findings of the STA/LTA are classified into true events, uncertain events, and false detections. The uncertain class refers to time-windows that are categorized as events by STA/LTA but are unverifiable visually due to low SNR data.Template matching is also run on the same stations for the two-month period. It uses the 23 cataloged events during the two-month period as master events to search for undetected events. Template matching results are also compared to the CNN’s results in the following section.In this section, we provide training details of the CNN architecture, outlined in The network is trained with an Adam optimizer using mini-batch optimization with a batch size of 64. The training stops after 14 epochs as validation accuracy ceases to improve for eight consecutive epochs. The network is then tested on two months of continuous data at the five stations near Zeerijp, indicated with red triangles in The trained CNN model picks 20 of the 23 cataloged events that were reported during the two-month period. The three unpicked cataloged events are actually very low-magnitude events .We developed a CNN-based algorithm for detecting low-magnitude events in Groningen using a multi-level sensor network. We find the performance of the trained CNN model to be considerably superior in comparison with STA/LTA and template matching. We test the performance of these methods on a two-month dataset around the relatively large Prior deep learning studies on the problem of seismic event detection ,15,27 usHowever, one downside of our proposed CNN model is its lack of ability to detect events buried under noise. This characteristic is due to the data used to train the network. We only used data that a human interpreter could visually confirm as events and, therefore, it did not include events that were masked by noise. One approach to overcome this limitation is to prepare training data using template matching since it has the capability to detect events even in a negative SNR regime. This also highlights an important direction for deep learning practitioners where existing tools and methodologies can be used to further improve the capabilities of these modern learning algorithms instead of viewing them solely as competing approaches.


This paper studies the problem of the dynamic scaling and load balancing of transparent virtualized network functions (VNFs). It analyzes different particularities of this problem, such as loop avoidance when performing scaling-out actions, and bidirectional flow affinity. To address this problem, a software-defined networking (SDN)-based solution is implemented consisting of two SDN controllers and two OpenFlow switches (OFSs). In this approach, the SDN controllers run the solution logic . According to the SDN controllers instructions, the OFSs are responsible for redirecting traffic to and from the VNF clusters . Several experiments were conducted to validate the feasibility of this proposed solution on a real testbed. Through connectivity tests, not only could end-to-end (E2E) traffic be successfully achieved through the VNF cluster, but the bidirectional flow affinity strategy was also found to perform well because it could simultaneously create flow rules in both switches. Moreover, the selected CPU-based load-balancing method guaranteed an average imbalance below 10% while ensuring that new incoming traffic was redirected to the least loaded instance without requiring packet modification. Additionally, the designed monitoring function was able to detect failures in the set of active members in near real-time and active new instances in less than a minute. Likewise, the proposed auto-scaling module had a quick response to traffic changes. Our solution showed that the use of SDN controllers along with OFS provides great flexibility to implement different load-balancing, scaling, and monitoring strategies. Technologies such as network function virtualization (NFV)  and multThe placement of applications and NFs, such as user plane functions (UPFs), at the network edge, provides significant improvements in the end-to-end (E2E) response time and bandwidth consumption, thus reducing the occurrence of bottlenecks in the network backhaul . HoweverDespite this potential solution, the deployment of virtual network functions (VNFs) at the edge poses several challenges due to the fact that they are more sensitive to traffic variations and the limited resources of the MEC nodes . When thThe virtualization of some traditional NFs such as traffic accelerators or firewalls, which can be deployed in a bump-in-the-wire (BITW) manner to avoid altering the communication endpoints, would bring about extra difficulties. No modification in the packet headers is allowed when redirecting traffic to guarantee the standard functionality of a service  due to their transparent deployment. Scaling out BITW VNFs may cause occasional loops in the network and subsequent problems . Additionally, bidirectional flow affinity during the flows’ lifetime may also be required.The design, implementation, and evaluation of a solution to make load balancing and auto-scaling decisions to manage a cluster of transparent VNFs.The proposed solution can not only distribute traffic according to the selected LB strategy but also guarantee bidirectional flow affinity without packet modification.An auto-scaling strategy is presented to manage the size of a transparent VNF cluster.The testbed of the proposed strategy in this research utilized real hardware unlike most works on this topic, which use simulation tools  to assess the performances of their proposed strategies.Most of the literature addressing the problem of load balancing (LB) and auto-scaling of VNF clusters ,11 negleThe remainder of this paper is organized as follows. This section provides a review of the literature related to load-balancing and auto-scaling solutions in virtual environments.Most SDN-based load-balancing studies focus their solutions on the designing of new methods to decide on the most suitable destination for a given traffic flow. In this manner, the authors of ,13 propoIn , two OpeThe authors of  propose Abdelltif et al.  propose The authors of  provide In , an autoThe authors of  propose Dutta et al.  present Lange et al.  proposesDespite the proposed solutions, none of the aforementioned works address the issue of load balancing and auto-scaling transparent VNF clusters. Of these works, the most similar to this paper is , as it pThis section presents some background on NFV management and orchestrators, SDN, and OpenFlow switches. These elements are fundamental pillars for our solution implementation.NFV is considered a key enabler technology for addressing the stringent requirements of 5G and beyond networks ,25,26. IThis paper adopts OSM as the NFV orchestrator (NFVO) and virtual network function manager (VNFM). It is responsible for the deployment of NSs and VNFs and monitoring the life-cycle management of VNFs. The selection of OSM was based on its maturity, performance, and comprehensive utilization . AnotherAlong with the NFV paradigm, SDN is recognized as a crucial pillar in the development of 5G and beyond networks to fulfill their network requirements ,25,26. TThe SDN concept is applied in a wide variety of solutions in which the controller is a key element in the control operations. In this regard, the architectural deployment of the controllers is crucial to guarantee that the performance of the overall network is adequate. Specifically, from an architectural point of view, SDN controllers can be classified into two groups: centralized and distributed. The centralized architecture is formed by a unique controller for the sake of simplicity. However, this design represents a bottleneck in the network and has scalability limitations when the network traffic increases. By contrast, a distributed architecture improves network scalability, flexibility, and reliability. For instance, this architecture avoids having a single point of failure since when one of its distributed controllers fails, another(s) can assume its functions and devices. Therefore, this design is more resilient to different kinds of disruptions. Furthermore, with the programmability of SDN, this process can be automated and configured according to the design and requirements of the network. Thus, the distributed architecture is capable of responding and adapting to new requirements and conditions.An OFS consists of one or more flow tables and a group table, which perform packet lookup and forwarding, and one or more OpenFlow channels to communicate with an external controller. The switch communicates with the controller, and the controller manages the switch via the OpenFlow protocol . The conWhen a flow arrives at an OFS it verifies, in its flow tables, if there is already a defined rule that matches the incoming flow. If such a rule exists, then the specified actions are executed. A wide variety of instructions can be applied to flow entries, such as packet forwarding  and packet modification. In the case of several rules matching the flow description, the rule with the highest priority order is applied. In contrast, if no match is found in any flow tables, the switch can be configured to send the packet to its assigned controller or drop it.Utilizing flow affinity implies that packets belonging to the same flow will undergo similar treatments along their paths  ,39. In tIn this section, after a brief overview of the problem of interest, the proposed solution for the management  of a cluster of transparent VNFs is presented.Network operators are virtualizing their physical network functions to align with new services requirements imposed by 5G networks and beyond. These NFs can be grouped into different categories according to their characteristics . Many of the NFs associated with network performance  are usually transparently placed between the access and internet network providers. In other words, they do not have a forwarding IP since they are placed as BITW functions and, thus, work in layer 2 of the OSI model. The virtualization of NFs offers flexibility for the management of network resources since NFs can be scaled as their processed traffic changes. Existing MANO frameworks  offer some scaling policies to automatically adjust VNF capacity to traffic demands. However, the scale-out policy of MANO frameworks is characterized by the placement of new deployed functions in the same subnetwork as the original ones. Moreover, they do not allow for the modification of previously launched NSs, for instance, the adding of new VNF instances or subnetworks. These limitations cause severe problems in the network when scaling transparent functions since network loops may appear.Load balancers are crucial to avoid overload and guarantee efficient traffic distribution to the pool members when more than one instance of a given type is deployed and forms a cluster. Additionally, load balancers enhance the availability and reliability of the network service by redirecting incoming requests to only healthy VNFs. Well-known MANO frameworks, such as OSM and ONAP, lack native load-balancing services. Thereby, they rely on specific virtual functions  that neMoreover, some VNF types, such as firewalls and intrusion detection systems, require that the same instance process all the fragments of a given flow during the flow lifetime. Other VNF types  may need to ensure bidirectional flow affinity to work properly.Under these circumstances, designing a mechanism for the efficient management of transparent VNF clusters is critical. The envisioned solution must be able to dynamically scale cluster resources, balance the flow traffic between transparent VNFs, ensure bidirectional flow affinity, and avoid packet modifications and extra processing.Our proposed solution for the management of transparent clusters is formed by two redirector VNFs, one at each side of the cluster. The reason for placing redirector functions on both sides of the cluster (uplink and downlink directions) is to ensure the bidirectional flow affinity requirement see . Each VNThe envisioned VNF redirector application has a modular design, where each module performs a specific function . The logic of these functions runs on top of the SDN controllers, as shown in To improve the VNF redirector performance and allow faster packet processing in the load-balancing module, our design exploits the parallel processing capacity of multi-cores along with a multi-threaded technique, thus avoiding delay in its procedure due to the performance of other modules. The following subsections provide more insights into the operation mode of the VNF redirector’s modules.Transparent VNFs are not compatible with well-known health monitor methods, such as PING, TLS-HELLO, UDP-CONNECT, and TCP-CONNECT, since they all require that their queries be directed to an IP address. A workaround could be the use of the management network to send the messages associated with one of these health methods. However, this implies additional load in the VNF cluster. To resolve this shortcoming, our solution uses information already available to other components . The VNFM monitors the health status and performance  of VNF instances and network services to manage their life cycles. In the OSM MANO, its monitoring module has a feature (mon-collector) to collect the specified metrics in the VNF descriptors. The mon-collector polls the VIM, where the VNFs are deployed, to gather the desired metrics and stores them in its Prometheus time-series database (TSDB). To make this information available to the redirector application, our solution implements the monitoring module. This module communicates with the Prometheus TSDB and gathers VNF status and metrics. In this manner, the LB can be aware of the healthy and unhealthy instances and their available resources. This approach solves the limitation mentioned above and avoids overloading the cluster with frequent health polling through the management network.This monitor module also gathers information about the deployed NS configuration, for instance, the ID of the virtual deployment units (VDUs) that constitute the transparent cluster and their subnetworks. These data are used to match the VDUs with the OVS switch interfaces to which they are connected. These interfaces are discovered upon the OFS registration on the SDN controllers. In this way, the monitor module helps the redirector application to discover the network topology.Additionally, when the monitoring module detects an unhealthy instance, it communicates the LB to trigger a flow rule updating process. Then, the LB module uses a method to delete the OF rules associated with the unhealthy instance in the switches table. This is achieved by sending a delete flow message to the OF switches, which instructs them to delete the flow rules that contain the specified port . With this action, the monitoring module avoids network disruption because another healthy transparent VNF can be selected by the LB to process the incoming traffic.The automatic scaling of transparent VNFs imposes several challenges to loop avoidance. For instance, when more than one transparent VNF is deployed in the same subnet, loops appear given that they act as a "wire" in the network. Thus, transparent VNFs should be deployed on different subnets to avoid network loops and redirect flows to a specific transparent instance since they do not dispose of routing information. However, the current MANO frameworks do not allow us to specify different subnets when executing auto-scaling policies or to add new VNF instances and subnetworks to already instantiated NSs. Therefore, we diverge from the assumption that the VNF cluster has been already dimensioned to resolve these challenges. In other words, the cluster is launched with all its members, and each VNF instance connects with one interface in the OFS. It should be noted that the OFS interfaces also need to be dimensioned according to the pool size. However, to save energy and computing resources, only the minimum required number of instances remains active, and the rest are on standby. Thus, the main aim of the scaling module is not to create or remove instances, but to manage their status . To this end, the auto-scaling module needs to interact with the VIM. The OpenStack VIM, for instance, through its Nova service , offers For the implementation of the auto-scaling logic, the following parameters need to be defined.Scaling metric: the metric to be monitored  and upon which scaling actions will be taken;Aggregation type: refers to how the scaling metric is gathered ;|VNF|_max: maximum number of active instances in the cluster;|VNF|_min: minimum number of active instances in the cluster;Thresholds: upper and lower bounds of the selected metric upon which scale-out or scale-in actions are triggered, respectively;Threshold time: a minimum amount of time in seconds during which the state of the scaling metric with regard to the threshold values must sustain to trigger a scaling event. Different threshold times can be defined for scale-in and scale-out actions;Cooldown time: the minimum amount of time that the system must wait after triggering an event before activating another.The auto-scaling logic runs during the system lifetime and updates the scaling metric values at each time instance. There are two ways for maintaining flow affinity when load balancing traffic. One approach uses a dedicated load-balancing algorithm based on an IP hashing method. The other uses a stick table in memory along with a non-deterministic LB algorithm . The use of the load-balancing algorithm approach is suitable as long as the number of involved instances does not change. Otherwise, more complex techniques, such as consistent hashing, need to be implemented to diminish the effects of variations in the number of instances on the flow affinity mapping. This method is recommended for load-balancing applications that do not require synchronization among LBs but still use the same hashing function. The second approach requires a global view of the system or synchronization among the LBs to ensure bidirectional affinity. The advantage of this approach is that no session is redirected when a new instance is added to the cluster. Our solution is based on the second technique since flow affinity is guaranteed upon flow entries registered in the OFS tables. Moreover, our solution does not require any exchange of information between the LBs . The reason is that the flow entries associated with a given flow are simultaneously created in both switches by their master controller, which has a global view of the cluster. A drawback of working with flow tables is that flow waiting time may increase with significant traffic. Nevertheless, this effect can be diminished by implementing an efficient mechanism to manage switch tables ,44.Additionally, old flow entries belonging to expired flows can be removed to avoid overloaded flow tables. Using the idle_timeout field in OF FLOW_MOD messages is highly recommended in this process. This parameter allows you to specify a time interval that the switch can use to remove idle flow entries when no packet has matched within the given number of seconds. For this process to be completed successfully, the OFS must keep track of the arrival time of the last packet associated with the flow.The communication process between the OFSs and the master controller for the flow rules creation is described in Once the controller has determined the OF rule, it sends an OF PACKET_OUT message and a FLOW_MOD message to the switch that sent the OF PACKET_IN (steps 5 and 6). These steps instruct the switch to send the packet through the selected port and configure the specified rule in its table. Similarly, a reverse OF rule is proactively configured in the opposite switch by swapping the source and destination fields in the matching pattern and specifying the same action . In this manner, our load-balancing module ensures bidirectional flow affinity by simultaneously creating flow entries in both switches.Currently, the load-balancing module has available three load-balancing strategies: random, round-robin, and least loaded ,45. The The main objective of this section is to validate the proposed solution as a feasible approach to manage traffic distribution and resources assigned to a cluster of transparent VNFs according to dynamic traffic demands.For the validation of the proposed solution, a simplified scenario was considered. The network service consisted of a VNF cluster, two redirector VNFs, four traffic generators (TGs), and one network. The cluster was formed by four generic VNFs that were transparently deployed by configuring their interfaces as a bridge. The TGs’ main function was to inject traffic between the access and data networks. This traffic had to traverse the cluster in a distributed manner. Additionally, subnetting was applied to divide the network domain into smaller subnets. This division allowed each group member to be deployed on a different subnet. It thus ensured that traffic passed only through a single transparent VNF and E2E connectivity between TGs located on different sides of the group. However, adopting the subnetting can introduce several loops in the network when working with BITW VNFS. To account for this, the network service deployment was automated to enable the spanning tree protocol (STP) in the redirectors and transparent VNFs interfaces to avoid network outages due to loops. Once the master controller had taken over the network control, the STP was deactivated to enable interfaces with blocked status. For our implementation, Ryu  was seleAs a proof of concept, we ran several experiments to verify the correct operation of each aspect of the proposed solution. We started by checking the connectivity between the different elements of the system. One essential requirement of our solution was the assurance of bidirectional flow affinity. To validate the solution for this feature, we generated traffic between the access and internet TGs. We configured the redirector application to create ARP and IP traffic flow rules based on their source and destination addresses. Additionally, they also had a default rule configured when they were added to the SDN controller to send packets that did not match any entries in their flow tables to the controller. As new traffic went through the switches, new rules were created according to the established matching criteria. By comparing the flow tables of the switches, we noticed the presence of bidirectional flow affinity, since each pair of source-destination addresses was configured to go through the same port (VNF instance) in both switches. Thus, bidirectional flow affinity was guaranteed by creating the rules in the access and internet side switches simultaneously during the arrival of PACKET_IN events in the SDN controller.A load-balancing strategy based on CPU utilization  was selected to evaluate the traffic distribution among members of the transparent cluster. For this test, 20 TCP flows were generated with a bandwidth of 1 Mbps, a duration of 2100 s and a waiting time of 90 s between each flow. For the traffic generation, a script based on iperf3  was implOne of the main advantages of virtualization is to provide network operators with the capabilities to increase or decrease their NFs according to specific metrics. This section evaluates our solution by studying its behavior during different scaling actions. Similar to the previous experiments, our pool started with two active members, and the rest were on standby. The TG in the access side generated 50 TCP flows with a bandwidth of 20 Mbps, a connection time of 15,000 s, and a waiting time of 300 s between each flow. These conditions guaranteed that there would be enough flows to increase the CPU load of the active cluster’s members and trigger the scaling actions.During the second half of the experiment, the oldest flows assigned to the initially deployed VNFs  started to reach their specified lifetime. Thereby, these VNFs were finished by the TG located on the access side. The termination of these flows produced a decrease in the load of these VNFs. Similar effects were noticed in the two last activated instances  around 03:45 h. The average CPU load in the transparent cluster started decreasing until the minimum established threshold was reached. At this point, the instance represented by the blue line was deactivated, and its existing flows were reassigned to the other VNFs due to a scale-in action. After waiting for the completion of the cooldown time, another scale-in action was triggered since the average CPU usage was still below 20%. This action resulted in the deactivation of the transparent VNF depicted in yellow.During this experiment, we noticed a reduction in the CPU load of the initially deployed VNFs when new instances were activated even though their allocated flows were invariably maintained. This behavior was due to an increase in system congestion levels caused by the establishment of new connections. To avoid congestive collapse, TCP implements a congestion window (CWND). This mechanism limits the number of outstanding unacknowledged packets that can be in transit on the network for a given pair of source and destination. Specifically, when the network traffic increases, each client that injects TCP packets increases its CWND to reduce its forwarding rate . These rThe primary purpose of the health monitoring test was to assess the system’s responsiveness when a member of the transparent VNF cluster fails. To this end, 20 TCP flows were generated with a bandwidth of 20 Mbps, a connection time of 6000 s and a waiting interval of 300 s between each generated flow. Additionally, a low granularity (300 s) was used in OSM and OpenStack to aggregate the collected metrics. This experiment started with two active VNFs to which the incoming TCP traffic was evenly distributed according to their CPU load. After 40 min of the simulation, a failure was emulated in a VNF instance  by putting it on standby. This alteration in the selected VNF state can be observed in Moreover, some significant delays were noted in displaying the CPU metric in the Grafana dashboard. We believe that better performance can be achieved by using higher granularity values . However, this involves modifying various parameters and configuration files at both OSM and OpenStack levels, which is not trivial and lacks detailed documentation.In this paper, we have proposed an SDN-based solution to manage a cluster of transparent VNFs dynamically. Specifically, our solution implements a modular application that runs on an SDN controller to perform load-balancing and auto-scaling decisions. The load-balancing block guarantees bidirectional flow affinity without packet modification by simultaneously configuring OpenFlow rules in the switches comprising the proposed solution. Most of the reviewed literature on this topic conducted experiments by using simulation tools such as Mininet. In contrast, our solution was implemented in a real environment using two well-known frameworks, OSM and OpenStack.The evaluation results validated the feasibility of the proposed solution. The solution was shown to successfully redirect E2E traffic through the transparent cluster without losing any packets. Additionally, its bidirectional flow affinity capability was demonstrated by comparing the switches tables and finding each pair of source-destination addresses attached to the same port. Likewise, the effectiveness of the load-balancing and auto-scaling mechanisms was demonstrated through different experiments. Moreover, the monitoring module’s performance was evaluated through a health test in which we could notice the activation of a new instance to meet the minimum number of active members in the cluster after detecting a failed member.Future work directions for this topic include designing strategies based on traffic forecasting. These strategies could enable scaling and load-balancing decisions to adapt to dynamic variations in traffic proactively. Furthermore, we intend to implement and test the performance of the proposed solutions in other VIM  and MANO technologies to evaluate and compare their performance and extend the evaluation scenario (network service topology and SFC).


Recent advances in artificial intelligence with traditional machine learning algorithms and deep learning architectures solve complex classification problems. This work presents the performance of different artificial intelligence models to classify two-phase flow patterns, showing the best alternatives for this specific classification problem using two-phase flow regimes (liquid and gas) in pipes. Flow patterns are affected by physical variables such as superficial velocity, viscosity, density, and superficial tension. They also depend on the construction characteristics of the pipe, such as the angle of inclination and the diameter. We selected 12 databases  to train and test machine learning models, considering these variables that influence the flow patterns. The primary dataset is Shoham (1982), containing 5,675 samples with six different flow patterns. An extensive set of metrics validated the results obtained. The most relevant characteristics for training the models using Shoham (1982) dataset are gas and liquid superficial velocities, angle of inclination, and diameter. Regarding the algorithms, the Extra Trees model classifies the flow patterns with the highest degree of fidelity, achieving an accuracy of 98.8%. Artificial intelligence has benefited many different disciplines. This phenomenon has impacted a wide swath of fields as diverse as clinical treatment, economics, internet security, and fluid transport . Fluids When gases and liquids flow simultaneously in a pipe, the phases can distribute themselves in various configurations depending on many variables. The interface distribution determines the configuration, which results in different flow characteristics . There iIn vertical or moderately deviated pipes, gas-liquid mixtures’ most common flow regimes are the bubble, slug, mist, churn, and annular flow. In horizontal pipes, there may be stratified or wavy stratified flow in addition to many of the regimes observed in vertical wells. Two-phase flow regimes can be represented as plots or maps, using their phase velocities or functions on each axis. Dispersed Bubble Flow (DB). It has the gaseous phase distributed as discrete bubbles within a continuous liquid phase. In other words, the tube carries a specific liquid that has small bubbles evenly distributed over the entire surface. This flow occurs when the rates of liquid are extremely high compared to those of gas.Stratified Smooth Flow (SS). It presents the gas flows through the upper part and the liquid through the lower part of the pipe. That is, there is a complete gravitational separation of the two phases. One reason is the higher density of the liquid and the force of gravity exerted on fluids.Stratified Wavy Flow (SW). It arises when the velocity of the gas increases in the SS flow, producing large ripples in the liquid. The SS and SW flows are stratified.Annular Flow (A). In this regime the gas flows in the center of the tube, as the gas velocity increases, the liquid flows through the tube wall or as scattered droplets.Intermittent Flow (I). The distribution of liquid and gas is not uniform. It changes from filling the entire diameter of the tube with liquid to having long spaces with gas. In horizontal flows, plugs or slugs of liquid separated by gas zones fill the whole cross-section of the pipe with a stratified liquid layer flowing along the bottom.Bubble Flow (B). It occurs when the gas phase is uniformly distributed and flows as discrete bubbles in a continuous liquid phase. The presence of slip in the bubbles flow allows them to move faster than the liquid phase; that is, there is a relative movement between fluids.Below is the description of each flow pattern as well as its abbreviation:Flow patterns change according to different physical properties and characteristics of the medium. The most notable physical properties of liquid and gas are superficial velocity, viscosity, density, and surface tension. Furthermore, the influence properties of the medium are the angle of inclination and the diameter of the pipe . Given tIn the study of two-phase flows, the correct estimation is vital due to its relationship with the design variables such as phase holdup, pressure drop, and chemical reaction rate . The preIn recent years, researchers have developed methodologies to identify multiphase flow regimes. The techniques based on machine learning are fundamental for classification with databases generated in laboratories. The ranking precision they have achieved shows that these methodologies work. The following are some of the studies highlighted for this problem: Previous work on the classification of two-phase flow pattern using the 12 databases to be used in the experiments of this paper reported an accuracy of: 74.71% for the classification of six flow patterns, 79.02% for the classification of five flow patterns, and 82.29% for the classification of three flow patterns, using a mechanistic model ; 74.84% This research implements machine learning (ML) techniques in the problem of predicting two-phase flow pattern, it presents machine learning models with their optimized parameters for better classification. This makes possible to determine which are the best alternatives for classifying flow patterns. Extra Trees stands out for its capacity for classification, training, and prediction times, among other models. It was shown that the machine learning techniques implemented are a viable alternative to classify flow patterns where fluids interact in liquid and gas phases. Due to the geometric distribution of the flow patterns, there are combinations of them ; therefoTo summarize, the significant contributions of this paper are the exploration of multiple machine learning models and the identification of the best performing ones to classify flow patterns using two-phase flow regimes (liquid and gas) in pipes. In addition, this work presents a classification system design made up of dataset partitioning, importance and selection of features, hyperparameter adjustment, construction of deep learning architecture, and evaluation of metrics.Materials and Methods” showing the Data and all the extensive treatment done for the problem, including the unions of patterns with similarities, the importance and feature selection, and unbalanced data troubleshooting. The Models presents and describes the models used in the investigation. It also shows the Training model for the Performance Evaluation, which shows the metrics used to validate the performance of the model. The section “Results” shows the main results obtained for the classification of flow patterns. “Discussion” present different analyzes and discusses the results obtained. Finally, the last section presents the “Conclusions” of the article.The remaining sections of this article are in the following order: “The The The flow patterns in the database depend on superficial liquid and gas velocities (Vsl and Vsg), the fluid and gas viscosity (VisL and VisG), the liquid and gas density (DenL and DenG), the surface tension (ST), the angle of inclination of the pipe (Ang), and the pipeline’s diameter (ID).This paper also involves another dataset, based on For further experimentation, the 12 DB produces two new datasets. These are the combination of flow patterns data with similar physical characteristics. The first dataset has a new label named Stratified Flow (ST), which combines instances from labels SS and SW, resulting in five labels. The second dataset has three labels resulting from the combination of Dispersed Flow (DB + B), Segregated Flow (ST + A), and I.Data preparation has the following operations: assigning each label a numerical value see , splittiθ < 90. In The 12 databases consist of a total of 9,029 experimental data points. Given the nature of the data the human error in some data samples is possible, because each researcher conducting experiments had to make visual observations and judgments to identify each experiment into a flow pattern regime, increasing the possibility of some bias. However, for a given set of pipeline flow conditions, including the operating superficial gas and liquid velocities, fluid properties, and pipe geometry, it is possible to predict the flow pattern.A combination of Z-score, also known as standard-score normalization, and Principal Component Analysis (PCA), is applied to the databases employed in this work to compute feature importance and perform feature selection. Keeping the most relevant while eliminating redundant features may decrease noise in the training procedure and improve performance. Based on these results, only the features Ang, Vsg, Vsl, and ID are selected to train models using the Addressing the class imbalance problem, There are different methods to deal with unbalanced datasets:Up-sample Minority Class. (y Class. .Down-sample Majority Class. (y Class. .Synthetic Minority Over-sampling Technique (SMOTE) and Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN) to balance the data  and learns the values of the parameters, or weights, θ that achieve the best approximation from input features x to the output y  , ConvoluGrid search and K fold cross-validation allow this work to improve the machine learning algorithms. After selecting a range of hyperparameter possible values, the grid search uses different combinations of them. With this process, the model uses some data to train and another to do testing. After that, the best hyperparameters allow saying what the best result is. Also, the cross-validation technique allows dividing the data in K folds, where K-1 allows training the model, and the other part is for testing. Then, the algorithm trains K times using different distributions, changing the test data, and the K results of test data produce the average result. Using this strategy allows searching the optimal hyperparameters .The first step to find the model with high performance is to determine the best set of hyper-parameters for each one. The hyper-parameter selection is performed training each model with different parameter values and using accuracy metric as a performance indicator. The following classifiers use 12 DB: Extra Trees, Support Vector Machine, Random Forest, Gradient Boosting, and a fully connected neural network. The first four models were subject to hyper-parameter tuning to optimize their performance for the task. The parameters and ranges evaluated are presented on These experiments are different models trained using different balance techniques for the data, like those carried out without class balance. The best models with their hyperparameters are in For the 12 DB the ANN see  is used An important choice when designing artificial intelligence systems is the metrics used to evaluate them . Below aThe accuracy is the fraction of correct predictions. In some cases, it is represented as a percentage or in values from 0 to 1.Precision: it measures a classifier’s ability to identify only the correct instances for each class.Recall: also known as sensitivity, it shows the classifier’s ability to find all correct instances by class.F1: is a weighted harmonic mean of precision and recall normalized between 0 and 1. F1 score of 1 has a perfect balance due to the relationship between precision and recall. This metric favors unbalanced classes, and it is useful when precision and recall are essential.Support: this metric indicates the number of actual occurrences of the class in test data.• AUC, which for this research incorporates confidence intervals.Receiver Operating Characteristic (ROC) is used to analyze the output quality of a classifier. This metric shows the balance between classifier sensitivity and specificity, an ideal curve reaching the top left corner indicates high sensitivity and specificity. For multiclass classification, the metric draws each category. And with this curve is associated a vital metric called The confusion matrix shows the combination of the actual and predicted classes. Each row of the matrix represents the instances in a predicted class, while each column represents the actual class instances. It allows understanding which labels are most easily confused.(CK): it is a standard measure to calculate agreement between the classification of qualitative observations. The output is between 0 and 1, the closer to 1, the better is the classifier. Excellent for evaluating multiple classes in unbalanced datasets. The equation represents the expected number of agreements as Pe and the observed as Po.(LL): this metric measures the percentage of estimation with the classifier’s negative logarithmic probability.(ZOL): it is a way of knowing the number of errors in the classification or the error percentage. When the value is zero, it means that the model is working well.(MCC): it has an extension to evaluate multi-class  models by computing the correlation coefficient between the actual and predicted labels. A coefficient of 1 represents a perfect prediction.: it creates different training and testing sets using several partitions of the complete original data and then trains and evaluates a model using each of these partitions.(RT): this shows the time it took for the algorithm to train and predict.The experimental results and the analysis performed in this study are reported in this section. The results include well-structured tables to compare the models, graphs of the training process, classification report and confusion matrix for the best model, and in addition, the section includes ROC curves with CI. The results for the In the experimentation process, the In terms of deep learning architectures, the CNN outperforms only ANN by accuracy, 96.83%, and 96.39%, respectively. The best model for classifying two-phase flow patterns using the As result of the hyper-parameter adjustment process, Results downsampling data.Results upsampling data.Results balancing classes (over-sampling) with SMOTE.Results balancing classes (over-sampling) with ADASYN.The experiments presented by this work with balanced data are done only with the 12 DB because this is conformed for the This research paper presents the performance of different machine learning models to classify flow patterns, showing the best alternatives for this specific classification problem using two-phase flow regimes (liquid and gas) in pipes.As part of the classification system design, it can be identified five key aspects: dataset partitioning, feature importance and selection, hyper-parameter tuning, deep learning architecture construction, and metric evaluation. First, partitioning the dataset is an essential data preprocessing step. It is essential to find appropriate data partition sizes that allow the model to generalize well in the training process and demonstrate reliability when evaluated. In this case, the training set represents 80% of the total number of samples and the test set the remaining 20%. Since the adjustment of hyper-parameters was performed using the cross-validation technique, the validation set was not necessary. The feature selection process aims to reduce the dimensions of the dataset without affecting the performance of the model or removing the relevant classification features. A correlation analysis of the features of datasets was performed, reducing the Regarding the selection of the model and the adjustment of hyper-parameters, an iterative process allowed to find an optimal configuration for each model in a determined evaluation range. Despite the remarkable ability of machine learning models to process and classify data, tuning their hyper-parameters ensures the best performance on its task, as shown in the results presented in this document. The main contribution of this work is the comparison of machine learning models to find the best alternatives. Furthermore, this work presents a deep learning model. It is necessary to find a balance between computational ability, training time, and performance. As a general design rule, a giant network should perform better, but in practice, larger networks take longer to train and tend to overfitting the training set, reducing test set performance. The architecture presented in this article is the product of a meticulous experimentation process, considering aspects such as the number of layers, number of neurons and activation functions. It is leading to remarkable results in classifying flow patterns.Finally, an extensive set of metrics evaluates the performance of the models. Given the unbalanced nature of the dataset, it is necessary to highlight the F1 score and ROC curves within these metrics to evaluate the classifier. The F1 score is related to two outcomes, precision and recall, and in the case of a multiclass problem, it is necessary to consider the average F1 score of all classes. ROC curves assess the quality of the model and provide a numerical and visual summary of class performance to assess. These curves are useful for analyzing the balance between sensitivity and specificity of the classifier. In this case (multiclass classification problem), the metric draws curves for each category and draws a curve along with its averaged macro and micro-ROC curves. The value to be compared corresponds to the area under the curves, called AUC, with 1 (100%) being the highest value of the ROC curves. The ROC curves with the CI allow the evaluation of the performance of the Extra Trees algorithm. These curves are for The models proposed for the development of this research proved to have distinct levels of effectiveness from one to another. Hyperparameter optimization allows obtaining very accurate models such as Extra Trees with 98.8% with balanced data. However, other models do not achieve good accuracy despite adjusting their hyperparameters, like AB and GNB which did not exceed 67.0% and 69.5%, respectively.The deep learning models achieve accuracy percentages over 96%, making them slightly less effective than the ET model. The DL architectures need more time to train and predict . The ML models do not waste excessive time; GB is the slowest with 56.6 s. The Extra Trees model mentioned above stands out from all others because it differs from both accuracy and runtime, with an accuracy of 97% and a runtime of only 4.3 s (The runtime includes training time and predicting labels for the test set), using One possible limitation of the current work comes from the nature of the data. There could be some bias in the original data because each researcher conducting experiments had to make visual observations and judgments to classify each experiment into a flow pattern regime. As a result, human error is possible in some samples of the data.A novel data analytics approach using machine learning is proposed to quantify the prediction of gas-liquid two-phase patterns in pipes. This is carried out by comparing the prediction of available methods against a comprehensive data base collected from the literature, which includes 9,029 data points.Using an optimized Extra Trees for the classification of flow patterns carries this out. It was shown that the Extra Trees algorithm could learn surprisingly well, as using our hyperparameters tuning allowed us to achieve high precision predicting different combinations of classes.The 12 databases consolidated in the experiments of this paper are among the largest in the literature for the classification of gas-liquid flow patterns, from previous studies for the classification of flow patterns it can be observed that the results presented in this paper provide novel techniques that can be used in the design of powerful artificial intelligence models as an excellent tool to predict the flow patterns in different industrial processes.As future work, the methodology designed and presented in this paper can be used in a more significant amount of experimental data, representing even better the conditions of the means of application.


Acutely ill patients presenting with conditions such as sepsis, trauma, and congestive heart failure require judicious resuscitation in order to achieve and maintain optimal circulating blood volume. Increasingly, emergency and critical care physicians are using portable ultrasound to approximate the temporal changes of the anterior–posterior (AP)-diameter of the inferior vena cava (IVC) in order to guide fluid administration or removal. This paper proposes semi-automatic active ellipse and rectangle algorithms capable of improved and quantified measurement of the AP-diameter. The proposed algorithms are compared to manual measurement and a previously published active circle model. Results demonstrate that the rectangle model outperforms both active circle and ellipse irrespective of IVC shape and closely approximates tedious expert assessment. Acutely ill patients presenting with conditions such as sepsis, trauma, and congestive heart failure require judicious resuscitation in order to achieve and maintain optimal circulating blood volume while avoiding increased morbidity and mortality ,2,3,4. ISpeckle noise present in ultrasound imagery is theoretically considered to be a Rayleigh distributed multiplicative noise  as the eActive contours (ACs) are widely used for segmentation of ultrasound images ,18,19,20Given that the cross-section of the IVC is largely convex, the IVC contour can be represented in polar coordinates and consequently, polar active contours appear as a promising solution for IVC segmentation . In 25]25], a poAccurate segmentation algorithms such as the one proposed in this manuscript have the potential to expand understanding into non-invasive volume status monitoring. Clinical research has demonstrated that the true angle of collapse is actually 25 degrees off vertical rather than the simple AP-diameter .In , the offThe remainder of this paper is organized as follows: u and v are the mean of the intensities for the pixels inside and outside the contour, respectively, and I is the intensity of the pixels on the contour. This functional is used to evolve the parameters of the circle, i.e., R as the circle radius and K points with polar angles kth sampled point are denoted asIn , authorskth contour point. It is shown that the average evolution functional approaches zero when the contour points are on the IVC boundary. In active circle algorithm, the center of the circle is evolved askth contour points obtained from These forces move the contour points to their new positions asR and Equation  indicateIn this section, we develop active ellipse and rectangle models based on the evolution functional proposed in .In , authorskth contour point is evolved asIn general case, during the evolution, the x and y are the coordinates of the points on the conic, a, b, c, d, and e are the conic parameters. Note that with an elliptical model, the values of a and b must be positive. With K points with coordinates The next step is to evolve the ellipse parameters using the evolved contour points. Unlike the circular model, the evolved ellipse parameters are not linearly related to evolved contour points. This non-linearity may result in convergence to local minima. To avoid this, at each iteration, we fit a new ellipse using the following conic equation. Note that since the algorithm still operates iteratively, the ellipse parameters are gradually evolving and hence, we call this algorithm active ellipse algorithm.X is a matrix with kth row, T is the transpose operator. After setting the gradient A is obtained asEquation  can be rA is less than A estimated at nth iteration. After convergence, the algorithm proceeds with the next frame.w = 3 pixels and the only parameters that have to be modified are the center and the height of the rectangle. With the forces defined as Equation . were collected with the IVC imaged in the transverse plane using a portable ultrasound  and a phased-array probe (1–5 MHz). Each video has a frame rate of 30 fps, scan depth of 19 cm, and a duration of 15 s (450 frames/clip). As it was described earlier in the Results section, for all eight investigated clips, the active rectangle algorithm performs closer to the manual measurement than the other two methods. This is due to the fact the AP-diameter is clinically defined as the largest vertical diameter inside the IVC. The active circle algorithm finds the largest circle inside the IVC and assumes that the diameter of this circle can efficiently model the AP-diameter. This is technically correct if the IVC is horizontally aligned, but based on the angle of the ultrasound prob, the IVC in the ultrasound clip can be rotated along the horizontal axis and this makes the results somewhat different from the clinically measured AP-diameter.In five out of the eight cases, the active circle algorithm performed better than the active ellipse algorithm. This is mainly due to the fact that a circle has less degree of freedom than an ellipse and therefore, circle evolution can be performed more accurate than ellipse evolution. Furthermore, from K as the number of contour points, and In this section, we analyze the computational complexity of the proposed algorithms in terms of the number of floating points operations (flops) required to estimate the AP-diameter for each frame. Assume quations , 13)–(1–(1Ni as quations , one canIn this paper, two novel algorithms based on elliptical and rectangular models were proposed for semi-automatic estimation of AP-diameter of the IVC in ultrasound videos. The proposed algorithms were compared with the active circle algorithm and was shown that although IVC usually has an elliptical CSA, both circular and rectangular models provide a more accurate AP-diameter measurement, while the rectangular model outperforms the other two models. This is due that fact that the AP-diameter is clinically measured as the maximum vertical diameter of the IVC which can be modeled better as the vertical side of a rectangle than the diameter of a circle or even the largest vertical diameter of an ellipse.