The traditional feature-extraction method of oriented FAST and rotated BRIEF (ORB) detects image features based on a fixed threshold; however, ORB descriptors do not distinguish features well in capsule endoscopy images. Therefore, a new feature detector that uses a new method for setting thresholds, called the adaptive threshold FAST and FREAK in capsule endoscopy images (AFFCEI), is proposed. This method, first constructs an image pyramid and then calculates the thresholds of pixels based on the gray value contrast of all pixels in the local neighborhood of the image, to achieve adaptive image feature extraction in each layer of the pyramid. Subsequently, the features are expressed by the FREAK descriptor, which can enhance the discrimination of the features extracted from the stomach image. Finally, a refined matching is obtained by applying the grid-based motion statistics algorithm to the result of Hamming distance, whereby mismatches are rejected using the RANSAC algorithm. Compared with the ASIFT method, which previously had the best performance, the average running time of AFFCEI was 4/5 that of ASIFT, and the average matching score improved by 5% when tracking features in a moving capsule endoscope. Regular classical feature-extraction methods, such as scale-invariant feature transform (SIFT) , speed-uAlthough these popular methods have achieved good results in indoor and outdoor scenes –7 and caThe aforementioned methods entail the use of additional equipment and more significantly, may adversely affect or even harm the patient. Moreover, physicians are generally reluctant to introduce additional procedures for lesion diagnosis. Therefore, these techniques may not be feasible in specific clinical settings. To address these issues, this study proposes an adaptive feature-extraction method for capsule endoscopy images, called adaptive threshold FAST and FREAK in capsule endoscopy image (AFFCEI), which draws inspiration from the ORB method. This technique aims to enhance feature extraction by adapting to the specific imaging conditions of the capsule endoscope, without introducing additional equipment or potentially harmful procedures. The method first constructs an adaptive threshold based on FAST to achieve the localization of features and then combines the variability of visual sensitivity to feature distance with proximity to realize feature description based on the FREAK  descriptAn adaptive threshold feature-extraction method is proposed to solve the problem of insufficient FAST feature extraction.The FREAK descriptor is combined with the feature-extraction method proposed in this study to enhance feature distinction in capsule endoscopy images.The innovative contributions of this study are summarized as follows: Feature extraction methods are widely used in many fields, such as target detection and image tracking.The SIFT algorithm which was proposed in 2004, displays strong robustness to image scale changes and has been a milestone for the feature extraction task. Many improved methods , 17 are To address this issue, for faster feature extraction, Rosten proposed the FAST corner detection algorithm , which cSpyrou and Iakovidis  comparedWith the development of deep learning (DL), DL concepts have also been applied in feature extraction methods –29. SupeThis study focuses on the threshold variability strategy, and inspired by ORB, proposes an adaptive feature extraction method, AFFCEI, for capsule endoscopy. The core idea of the method is to dynamically determine the threshold value based on a pixel point and remaining grayscale values in a specific neighborhood using the formula proposed in this study, thereby achieving a self-adaptive threshold value, which combined with the FREAK descriptor , can betThe methodological approach of this study is illustrated in Fig. To ensure the scale invariance of the extracted features, a Gaussian pyramid is first constructed. The original image is used as the first level, and a Gaussian filter is applied to the original image to simulate scale changes and reduce resolution. The image is then downsampled by a scale factor of 1.5 to obtain the second-level image. The same process is repeated for the new image to obtain eight images. Feature extraction is performed on each of the resulting images, recording the layer number where each feature is located.t must be set to determine whether a pixel is a feature. In traditional methods, this threshold value is usually set manually to a fixed value based on experimental results and is applied to all images at different scales. In capsule endoscopy, the camera light source affects the image, with higher brightness in the near field and darker brightness in the far field. As the capsule endoscope moves, the same scene can be affected by different lighting conditions, and applying a fixed global threshold to all pixels is difficult. Therefore, an adaptive threshold is required to increase the accuracy of feature extraction and obtain a sufficient number of features. In this study, the threshold value is associated with the local contrast of the image: 16 pixel points on a circle with the pixel point as the center and three neighboring pixel points with the neighborhood determined by the selected radius [t was calculated as shown in Eq. . Assuming that the sum of the pixels is N, the calculation complexity of the first-level image is O(N), and subsequent images are created by 1.5 times downsampling. Therefore, the total complexity of pyramid construction is O(N). Subsequently, feature extraction is performed for each image. By traversing all the pixels in the image, the proposed threshold calculation method requires iterating over each pixel in the neighborhood to compute the sum while finding the maximum and minimum values. Assuming k is the sum of the pixel neighborhood, this process has a time complexity of O(k). Further assuming that the number of pixels is M, the time complexity reaches In terms of time complexity, the Gaussian pyramid constructed using this method has eight levels, and the time complexity of the Gaussian smoothing operations is approximately O(L), where L represents the length of the feature description. Finally, the time complexity of the GMS algorithm is O(A), with A representing the number of features to match.In feature optimization, the time complexity of a match based on the Hamming distance is First, the original feature-extraction method was compared with the adaptive feature-extraction method in two scenes. The original method extracted features using the fixed-threshold FAST method and described them using FREAK. The results demonstrate the improvements obtained by the proposed method. The results of the rotational scenes are listed in Tables According to the results, the proposed method for feature extraction is significantly more powerful than the original method in extracting highly reproducible features for the same descriptor. As expected, the proposed method computes the corresponding threshold for all pixels in the feature-extraction range, demonstrating the adaptive nature of the approach.The features extracted by the proposed method were compared against those obtained by different descriptors, such as SIFT, Brief, and SURF. The experiments were conducted under rotational and simulated motion scenes. The results of the rotational scenes are presented in Tables In this experiment, several popular feature-extraction methods were compared with the method proposed herein to demonstrate the advantages of the proposed method.First, ORB was considered as a comparison method before improving it using the rBrief descriptor. Subsequently, SURF and ASIFT, which are improved SIFT methods, were considered. Recently, with the development of DL, numerous feature-extraction methods based on learning methods have been proposed. In the experiment, SuperPoint  was consThe performance of SuperPoint, as a self-supervised DL model, depends on the quality and quantity of the training datasets. Capsule endoscopy datasets are rare because of the privacy protection of patients and hospitals, which affects the effectiveness of the model. Moreover, images of the human stomach may differ due to disease and individual body composition. The lack of a large dataset to support the portability of the model is another problem.Additionally, the model structure of SuperPoint is complex and requires considerable computational resources and time for training and testing, which may not be suitable for application scenarios with limited resources.The proposed method is used in medical application scenarios of capsule endoscopy, which require methods with reliable interpretability. Compared with the method used in this study, SuperPoint relies on a pre-trained feature extractor and does not have good interpretability. Although SuperPoint has the best performance in matching scores in Fig. Moreover, compared with ASIFT, the average matching score of this method is improved by 5%, and the running time is 4/5 that of the ASIFT method.To verify that the method in this study is also applicable to other images, feature extraction was performed on images other than capsule endoscopes. This section describes the selection of images from Homography patches. The images used in the universality verification included rainy days, daytime, and nighttime, as shown in Fig. This study proposes an improved feature extraction algorithm called AFFCEI for capsule endoscopic stomach images characterized by weak texture and uneven illumination. The algorithm uses an improved FAST feature-extraction method in the localization stage to extract a sufficient number of features. The threshold value of each pixel point is determined by calculating the gray-level contrast in the local neighborhood of the image, which is regionally adaptive and helps extract features more reliably even when the illumination changes. In the description stage, the features of the stomach image were accurately differentiated using the FREAK descriptor. The proposed method can be applied to extract features from images with poor texture. Overall, this approach significantly enhances feature extraction from capsule endoscopic stomach images and those of similar scenes.


The remote sensing image airplane object detection tasks remain a challenge such as missed detection and misdetection, and that is due to the low resolution occupied by airplane objects and large background noise. To address the problems above, we propose an AE-YOLO (Accurate and Efficient Yolov4-tiny) algorithm and thus obtain higher detection precision for airplane detection in remote sensing images. A multi-dimensional channel and spatial attention module is designed to filter out background noise information, and we also adopt a local cross-channel interaction strategy without dimensionality reduction so as to reduce the loss of local information caused by the scaling of the fully connected layer. The weighted two-way feature pyramid operation is used to fuse features and the correlation between different channels is learned to improve the utilization of features. A lightweight convolution module is exploited to reconstruct the network, which effectively reduce the parameters and computations while improving the accuracy of the detection model. Extensive experiments validate that the proposed algorithm is more lightweight and efficient for airplane detection. Moreover, experimental results on the airplane dataset show that the proposed algorithm meets real-time requirements, and its detection accuracy is 7.76% higher than the original algorithm. With the rapid development of remote sensing technology and sensor technology, the quantity and quality of remote sensing images have been greatly improved in the last few decades, which can be used to describe various objects on the Earth’s surface, such as airports and buildings . Airplanetc. These methods generate object candidate frames first and then perform classification and regression on the images in the candidate frames. Two-stage object recognition methods have high positioning accuracy. But the detection speed of those works is slow and they still perform poorly for the real-time requirements of airplane detection. The other is regression-based single-stage object recognition, such as Single Shot Detector (SSD)  , You Onlor (SSD) , 2018, eUsing a lightweight multi-dimensional attention mechanism in the backbone network to optimize the feature expression ability of the backbone network and enhance the network’s learning ability for small samples.Using the weighted two-way feature pyramid operation fuses features to obtain richer semantic information and location information.The recognition network was reconstructed by exploiting a lightweight module, which greatly improves detection accuracy and speed of airplane objects.We experiment on the airplane dataset, and the results show that the algorithm in this article has a better detection performance than Yolov4-tiny in the detection of airplane images with ground looks similar to background in color, densely distributed, and overexposed.From the perspective of detection accuracy and speed, this article proposes an accurate and efficient airplane detection algorithm (AE-YOLO). The main contributions of this work are summarized as follows:The rest of this article is organized as follows: an airplane object detection algorithm named AE-YOLO is proposed in “AE-YOLO”. “Experimental process and result analysis” presents the experimental environment, datasets, evaluation metrics used in the experiments and the performance of each improved module. Finally, a concise conclusion of the whole work is given in “Conclusion”.This section will introduce the proposed network model AE-YOLO in detail. Its structure is shown in (a) Feature extraction. First, the features are extracted and down-sampled through two ghost  Feature fusion. The three-layer feature maps with resolutions of (c) Object prediction. The input image is divided into lgorithm  is used In the convolutional neural network, the airplane object is susceptible to interference and is not conducive to the detection of small objects in the complex background because the airplane object in the feature map has the same importance as other complex backgrounds. In order to increase the recognition ability of feature maps for specific areas and specific channels, AE-YOLO introduces a multi-dimensional attention module to increase the importance of the object area and improve the detection ability.We propose a plug-and-play lightweight multi-dimensional attention module LCBAM (Light-CBAM) based on the attention mechanism of CBAM  and ECA F, one-dimensional channel attention F′ and input it into the spatial attention module to generate space attention module F′ to generate the final feature map F′′. The expressions can be defined as follows:Given an intermediate feature map F to increase the receptive field of the feature map and to aggregate the global spatial information of the feature map. Two different feature descriptors To generate the channel attention module, follows these steps. First, using the global average pooling and maxpooling operations on the feature map  sigmoid  activatiC represents the number of channels of the input feature map, and F′ can be performed in the spatial dimensions respectively. Since the initial backbone network consists of three standard convolutions and three ResBlocks, each ResBlock module only performs four convolution operations and the entire backbone network uses only 15 convolutions to extract the input image, the network structure is too simple. When the airplane object is very small or the background color is similar, the extracted airplane object information is less and the network’s ability to distinguish useful features is weak, which is easy to cause missed detection. In order to solve this problem, AE-YOLO is based on the idea that shallow feature structure is simple and universal and the deep features are more complex and abstract , an imprThe initial Yolov4-tiny uses FPN (Feature Pyramid Networks)  as the fry small . Local fThe fact that the high resolution of shallow features have richer detail information, the low resolution of deep features have richer semantic information. However, FPN simply performs a feature fusion with P4 and upsampling of the P5 feature layer, and the utilization of the feature information extracted from the backbone network is too low. In order to fully integrate the shallow detail information with the deep semantic information, the AE-YOLO network architecture adds a top-down channel fusion on the basis of FPN. In the meantime, by reference to the bidirectional feature fusion (BiFPN)  idea, thIn order to balance the detection accuracy and speed of the model, we took the perspective of lightweight, the entire network is reconstructed by the lightweight ghost module, which reduces a lot of redundancy in the intermediate feature maps of mainstream convolutional neural networks, and greatly reduces the computation and parameters while maintaining the accuracy.Given an input feature map w′×c×k×k .Y′, a simple linear However, the ghost module uses the cheap linear operations to complete the generation of feature redundancy, which reduce a large number of convolution operations, specifically as follows: First, the standard convolution is used to generate the inherent features Y′. In order to improve the processing speed, The computer hardware and software configuration used in this experiment are as follows. CPU is Intel(R) 3.3Core(TM) i7-10750H and GPU is Nvidia GeForce GTX 1660Ti. The experiments are implemented on the PyTorch deep learning framework with 16 GB RAM in Windows computer. Pycharm compilation software and python programming language is used.etc. For the flying aircraft in the remote sensing image, because the distance between the shooting position and the aircraft is much larger than the distance between the aircraft and the ground, it can not be seen that the aircraft is flying, so the flying aircraft can be approximately regarded as taking the ground as the carrier.At present, there is a lack of public datasets suitable for deep learning training in the research of remote sensing image airplane object detection. The airplane datasets used in this article are derived from the remote sensing image datasets RSOD  and UCASP, precision), the recall rate , and the average precision  to evaluate the detection efficiency of the algorithm. Detecting each image required time  Performance of different attention mechanismsIn order to verify the effectiveness of the proposed attention mechanism, AE-YOLO uses SE attention mechanism, ECA attention mechanism, CBAM attention mechanism and improved LCBAM in the Yolov4-tiny initial backbone network for comparative experiments, and the comparison results are shown in (b) Detection effect of different feature fusion networksIn order to verify the effect of the bidirectional feature fusion network used in this article, the multi-scale feature fusion networks such as two-layer FPN, three-layer FPN, PANet and BiFPN are used for comparison in the case of using the same backbone feature extraction network  and parameter settings, and the results are shown in (c) Performance of lightweight moduleIn order to balance the detection speed and accuracy of the algorithm, the lightweight ghost module is used to reconstruct the object recognition network. (d) Comprehensive experiment In order to verify the effectiveness of the three improved methods proposed in this article, ablation experiments were added, as shown in In order to further evaluate the performance of the algorithm, this article compares the performance of AE-YOLO algorithm with the state-of-the-art neural network algorithms on the airplane dataset, including large convolutional neural networks Yolov3, Yolov4, Vgg-SSD and lightweight convolutional neural networks Mobilenet-SSD, Yolov4-tiny. The comparison results are listed in Small airplane objects are easily affected by natural environment such as illumination and ground background, there is high missed detection rate and low detection accuracy of small airplane objects in remote sensing images. This article proposes an accurate and efficient algorithm AE-YOLO to the above problems. A lightweight multi-dimensional attention mechanism LCBAM was firstly integrated into the backbone network to enhance the feature extraction capability of the backbone network. Then, a weighted bidirectional feature fusion network was used to fuse the features by making full use of the details of the shallow features and the deep features, and to learn the importance of each branch through self-learning weights to enhance the utilization of features. To balance detection rate and detection accuracy, a lightweight ghost module was used to reconstruct the network in the end. The experimental results show that the proposed AE-YOLO algorithm improves the detection accuracy by 7.76% compared with Yolov4-tiny, and reduces the parameters and computation to 50.4% and 51.15% of the original, respectively. It also has a good detection effect in airplane images with densely distributed in the natural environment, overexposure, and ground looks similar to background in color. The experiments also found that the proposed AE-YOLO algorithm is insufficient for the detection of some objects with serious occlusion. For future work, we will continue to enhance the proposed algorithm and improve its detection performance on the data enhancement of small objects and the detection of objects with serious occlusion and blurred background.


In the current industrial revolution, advanced technologies and methods can be effectively utilized for the detection and verification of defects in high-speed steel filament production. This paper introduces an innovative methodology for the precise detection and verification of micro surface defects found in steel filaments through the application of the Eddy current principle. Permanent magnets are employed to generate a magnetic field with a high frequency surrounding a coil of sensors positioned at the filament’s output end. The sensor’s capacity to detect defects is validated through a meticulous rewinding process, followed by a thorough analysis involving scanning electron microscopy (SEM) and energy-dispersive spectroscopy (EDS). Artificial defects were intentionally introduced into a sample, and their amplitudes were monitored to establish a threshold value. The amplitude signal of these created defect was identified at approximately 10% FSH, which corresponds to a crack depth of about 20 µm. In the experimental production of 182 samples covering 38 km, the defect ratio was notably high, standing at 26.37%. These defects appeared randomly along the length of the samples. The verification results underscore the exceptional precision achieved in the detection of micro surface defects within steel filaments. These defects were primarily characterized by longitudinal scratches and inclusions containing physical tungsten carbide. Non-destructive testing (NDT) is by far the most important component of modern inspection technology, and it contributes an important role in ensuring that objects have reliable performance in its shelf-life. Especially, the application of NDT in manufacturing industries, such as for steel wire, is an integral element of the state-of-the-art inspection. Steel wire is used as a reinforcement material in the construction of tire belts, beads, and sidewalls, which are responsible for providing strength and stability to the tire ,2. It isThe production of steel wire begins with a pickling process in which the coils undergo cleaning using hydrochloric acid and water to remove rust after initial inspection. The wire rods are then drawn into thinner diameters through multi-dies while in a dry condition using a special lubricant in powder form. The wire is then coated with a layer of brass, comprising copper and zinc. These brassed wires are then transferred to the wet-drawing process in which they are subjected to multi-pass drawing using a wet lubricant. Finally, the wires are pulled into spools at the end of the wet drawing process. The individual steel wires are then stranded together to form a steel cord, as shown in In the steel wire manufacturing industry, one of the major challenges for steel wire suppliers is to identify failure modes in micro/nano scale due to their influence. With the occurrence of micro surface defects, the risk of steel wire breakage steadily increases, which can lead to serious accidents. Identifying and addressing these defects is crucial to ensure the safety and reliability of steel wire products. Many solutions have been suggested to minimize the likelihood of wire breakage in manufacturing. However, various technical or quality issues remain to be resolved in production workshops ,5. ThereOver the decades, the demand for visual inspection during the manufacturing process has significantly increased. In particular, non-destructive testing (NDT) has been widely used as a quality control gate to ensure that defective products are not delivered to end-users. Due to the high cost of manual inspection, automated observation system for quality control is preferred to replace human labor, and to improve overall equipment effectiveness (OEE) and productivity for manufacturers . The utiAs a part of the NDT industry, the eddy current principle finds extensive use in various applications, including automotive, aerospace and steel inspections. In terms of micro surface defect inspection, conventional monitoring methodologies have demonstrated limited responsiveness, making it applicable only in small scales, with high costs and very poor productivity . A non-destructive testing approach was established by combining mechanical and magneto techniques, enabling the identification of signals emanating from surface defects on the wire . To accuThe fatigue characteristic of the wire was significantly impacted by the presence of micro-defects when wires used. In a test conducted using wire rod specimens with surface defects, the behavior of steel wire was analyzed in terms of fatigue, where the presence of micro surface defects caused early wire breakage . Other rVisual inspection is essential in the cutting-edge manufacturing environment, particularly in aerospace, automotive, and three-dimensional printer engineering. A computer program was installed around the hot-rolling process using a multi-camera system to detect surface defects on wire rods. Another approach called WR-YOLO was also presented to detect surface damage in steel wire ropes. The equipment included a camera and a computer to monitor the broken wire situation on its surface during the stranding process, However, the system was too complicated to install and was not feasible for mass production with various wire diameters. ,21The grooves in steel wire were analyzed to determine the impact of groove width and depth on wire quality. The results of the experiments conducted during production using the multi-die drawing process were compared to those simulated using finite element analysis . In non-Another textile inspection method called computer vision was used to demonstrate the appearance of various typical defects by a fabric monitoring system ,26. SpecIn addition to the detection, surface defect analysis must also be considered. Various methodologies are based on signal processing and are not able to give a detail in defect analysis because they are not able to collect the specimen containing the surface defects. In the state-of-the-art industry, steel makers have the responsibility to reduce the risk of delivering faulty goods to the market. To achieve this goal, several inspection approaches using cutting-edge technologies have been developed to enhance product quality, overall equipment effectiveness, cycle time, etc. Among the various inspection technologies, vision-based monitoring is a common approach used to detect defects in different steel wire ,32,33. HThis paper introduces an innovative methodology for the early identification of micro defects on 0.38 mm steel filaments. It intends to provide the approach to inspect, verify and analyze micro surface defects in steel filaments that can accurately optimize high-speed manufacturing processes and improve the quality of products. The proposed methodology utilizes the eddy current principle to address technical challenges related to enhancing quality and to tackle the problem of micro surface defects in the steel-cord-manufacturing industry. The remaining sections of this paper are structured as follows: The research methodology is provided in The Eddy current is established based on the principle of electromagnetism as the basic for carrying out experiments . An enciZ0, which is a complex number, as described in Equation (1):One of the key parameters of the sensor coil is impedance Since steel wire goes through the test coil, eddy currents appear on the steel wire, leading to a secondary field that is opposite with the primary field. Hence, another impedance is created, which is defined in Equation (2) :(2)Zc=RcFrequency is a part of the testing sensor coil, and when the applied frequency increases, the inductive reactance of test coil will increase accordingly, as indicated in Equation (3) :(3)X0=j2The distribution of eddy current is non-uniform across the complete volume of test specimens. It is more concentrated near the surface and gradually decreases exponentially when the distance from the surface increases. The current flux pattern is described in Equation (4) :(4)J→=J(tion (5) :(5)J(z)=ace is δ :(6)J consists of a coil housing with supporting tube on each side and the centrally positioned coil housing stores the coil unit, as illustrated in A set of encircling sensors was placed on the top of wet-drawing machine, as shown in The sensor coil was situated between two tension rollers at the machine’s output end, which was precisely aligned with the wire path. It was linked to a pre-amplifier, and the resulting signal was subsequently directed to the instrument, as depicted in The specimen containing the created cracks was stretched and placed through the sensor in order to monitor the amplitude signal. The frequency was controlled at maximum level of 1000 kHz to identify the surface defects. Consequently, the amplitude signals of the introduced defects amounted to around 10% and 41% of the full-screen-height (FSH) amplitude for depths of 20 and 80 µm, respectively. Additionally, noise was also observed along with the defects, with a level of 3% attributed to the rusting of the wire, and the dotted line shows the reference threshold at 30% FSH, as illustrated in If the threshold value is set at 10% FSH to identify the defects with a depth of 20 µm, the sensor will detect any defects containing cracks deeper than 20 µm during the experiment. Based on the testing results, the threshold of amplitude signal was set of 10% FSH to stop the machine when desired defect appears.Based on the established setting conditions from previous steps, the sensor was installed at the output stage of the drawing process. Non-defective specimens should have a length of 38 km without any defect or with minor defects characterized by an amplitude signal below 10%. When an amplitude signal exceeds the threshold value, the drawing process will be stopped. To verify the detected defects on filaments, 13 defective samples were selected and pulled back through the sensor coil to verify the signals of micro surface defects and compared them to amplitude signals measured during the experiment.When the sensor detects desired defects, the machine keeps running for an additional 26 m before stopping. Then, the output filament was manually pulled back through the sensor to capture the defective signal and compared with those obtained from the system. The defects were also identified under an optical microscope before cutting them into small pieces containing the surface defects, which were then analyzed by SEM and EDS.In this study, 182 specimens were tested on the steel-wire-manufacturing process, the sensor system identified 48 flawed samples, as outlined in After detecting micro surface scratches, the machine ran for an additional 26 m before coming to a complete stop. By identifying the excess length and rewinding it backward using the sensor coil, the same signals were captured and compared with those recorded in the system for the previous specimen. Eventually, surface defects in steel filament were confirmed and clarified under a microscope, as indicated in Under an optical microscope, the components containing the defect were sectioned into smaller fragments and affixed in parallel with tape prior being placed into a mold, as depicted in In order to compare the different materials inside and outside the inclusion defects, EDS analysis was performed on both regions, and it was repeated for the second defect on the same sample. As illustrated in Considering the fact that tungsten carbide is utilized to produce dies for the drawing process, it is likely for the broken pieces of dies to become stuck inside the steel wire during the drawing process due to the high hardness of tungsten carbide compared to that of steel. Additionally, the drawing process takes place under high pressure and in a single direction, making it easier for the broken die pieces to insert into the surface of the steel filament. Therefore, the primary reason for the inclusion defect containing tungsten carbide is likely due to the broken dies during the drawing processes.For the scratch defects, there is no room for doubt that the micro surface defects are the scratches including the start and end points. It is important to narrow down which process caused these scratches. Furthermore, since the steel wire was coated with a brass layer before going to the wet drawing process, an investigation was conducted to determine whether the scratched defects were created during wet-drawing or by the previous processes. The brass-coating layer was also analyzed using EDS.The analysis was conducted two times at three places  inside and outside  the scratch defect, as shown in In this paper, a comprehensive methodology was introduced for detecting, verifying and analyzing micro surface defects in steel filament, using Eddy current principle and EDS technique. The sensor coil was utilized with created defects to establish the working conditions and to separate unacceptable products with surface defects. The defect rate stood at 26.37% out of a total of 182 produced samples. The detected surface defects appeared randomly along the length of the product and were also verified manually by pulling back the wire and comparing these results to those obtained from the sensor system during the experiment, which confirmed the accuracy of this methodology to detect the surface defects. Two common surface defects have been identified: the first is a longitudinal scratch defect characterized by both a start and an end point, while the second defect exbibits a diamond shape and contains an external material referred to as an inclusion. Furthermore, EDS analysis confirmed that the root cause of the inclusion defect was tungsten carbide, which is the unique material used for drawing dies. The second defect was also confirmed to have been caused during the wet-drawing process owing to the difference in the brass-coating layer between inside and outside of the defects.This innovative approach demonstrates the capability to precisely identify surface defects on 0.5 mm steel filaments at an exceptional speed of 400 m per minute. By accurately pinpointing the location of these defects, the methodology facilitates in-depth analysis essential for quality control and root-cause investigations. This pioneering contribution not only advances the efficiency of defect identification and analysis but also sets a new standard for quality management in high-speed steel-filament-production processes, signifying a valuable step towards improved manufacturing efficiency and product quality assurance.


The main focus of this work is to introduce a single free-breathing and ungated imaging protocol to jointly estimate cardiac function and myocardial  Magnetic resonance imaging (MRI) is commonly used for the diagnosis and prognosis of cardiac disorders. The improved image contrast, clarity, and non-iodizing nature of image acquisition make cardiac MRI the gold standard for cardiac function and tissue characterization. Cardiac function is usually measured using gated breath-held 2D acquisitions ,2. The cnitoring ,4,5. Com (MOLLI) , saturat (MOLLI) , and sat (MOLLI) . The carCompressed sensing and low-rank methods have been extensively used to reduce the breath-hold duration in cardiac cine ,10 and T mapping ,12. MR focardium ,14. These, refs. ,15 use te, refs. ,15. Recee, refs.  relies ou et al.  introducmilar to ,15, whilThe main focus of this work is to introduce a free-breathing and ungated imaging protocol for the joint acquisition of cardiac function and ior work , we modeWe introduce an efficient two-step reconstruction algorithm, which is significantly more computationally efficient than our previous joint estimation strategy ,19. We uWe note that generative modeling is emerging as a powerful tool in medical imaging ,21. We hThe main objective in dynamic MRI is to acquire high-resolution images from possibly undersampled measurements. The image frames The MR images are acquired by multichannel measurement operators, which are often different for each image:i.Here, CNN-based generative models were recently introduced for free-breathing and ungated cardiac MRI ,19 withoThe input The first term in the cost function is a measure of data consistency. The second term penalizes the norm of the Jacobian of the deep network, which serves as a regularization term; it controls the smoothness of the generated image manifold . The lasThe proposed sequence for the data acquisition in this work is depicted in In our previous work ,19, the To minimize the computational complexity and to adapt the method to the The VAE consists of an encoder t, given the 2D latent vectors The decoder generates the approximate k-space navigator signals The standard VAE cost function is specified by n.The first term is the data term that compares the output of the decoder to the k-space navigator signals. The second term is the Kullback–Leibler (KL) divergence of the distribution In our experience, the use of the latent vectors estimated by the above VAE model with G-SToRM can provide good image recovery. However, the latent vectors are often not disentangled and hence may not be interpretable. In particular, each latent signal may capture a mix of cardiac and respiratory motion components. In this work, we are interested in disentangling the impact of cardiac and respiratory components, which will facilitate the generation of images with specific cardiac or respiratory phases. Several disentanglement priors were introduced in machine learning, including beta-VAE , to imprMotivated by self-gating methods that use band-pass filtering with preset bands ,27 to seWe note that the estimation of the latent vectors is challenging in the We use gradient descent to minimize the above cost function.Once the VAE optimization is completed, the mean vectors zt in  are indiWe model the 2D image frames in the time series as the output of a CNN generator:The cost function for the scheme can be described as follows:Here, Once the generator parameters me using , as showWe propose two approaches to estimate the es as in . Since wp with the desired step-size as shown in Once the learning is complete, we fix the latent vectors Once the learning is complete, the generator can be fed with the latent vectors and inversion signal at any arbitrary time point to obtain the corresponding image. In the retrospective binning strategy, we identify the time-instants when the latent vectors are in a specified cardiac and respiratory phase as illustrated in All the data used in this work are acquired on a 3T MR750W scanner .We note that the proposed scheme in this work is totally unsupervised and hence no fully sampled training data are needed. The parameters in the dense auto-encoder as well as the parameters of the CNN are learned from the undersampled k-t space data of the specific subject being worked on, resulting in a subject-specific representation. In other words, we solve the networks based on each different subject.The encoder We determine the parameter We use a CNN image generator with three inputs (2D latent vectors and a 1D conditioning vector involving the inversion time). The CNN generates a two-channel output image, which corresponds to the real and imaginary parts in the MR images. Eight layers are used to implement the generator, and the total number of trainable parameters in the generator is about 15 times the image size of one image frame. For the convolutional layers, leaky ReLU activation is used for the generator except for the last layer, where tanh is used as the activation function. Random initialization is used to initialize all the networks. Note that we progressively increase the dimensions of the features, starting with small dimensions.The parameter 2 samples, 14 MnCl2 samples, 14 proton density samples, and 1 CuSO4 fiducial sample. In this study, we image the 14 NiCl2 samples with different known We use a commercially available  Essential System Phantom containing NIST traceable human tissue mimic solutions measured with landmark accuracy and precision to determine the accuracy of the We use the proposed inversion recovery sequence with three different settings to determine the dependence of the )3] data  are alsoAn in vivo study is performed using the short-axis orientation without contrast. Considering that flip angle We use an algorithm developed in-house to pre-select the coils that provide the best signal-to-noise ratio in the region of interest. We then estimate the coil sensitivity maps using ESPIRiT . A totalIn In The proposed scheme learns a generative model from the measured k-t space data. Hence, one can excite the generator with arbitrary combinations of latent vectors and inversion signals to generate synthetic images on demand. We now illustrate two important modes.We fix the inversion signal and respiratory phase, and only vary the cardiac phase to generate synthetic breath-held cine images. The inversion times may also be varied to generate cine data with different contrasts as shown in Similarly, the latent phase signals can be fixed to desired phases , and the inversion signal can be varied to generate images with arbitrary inversion times. We illustrate this in 2 regions for different settings and compared them to the reference 2 samples.Results of the phantom study are shown in 2 samples  wall analysis results with the 2D conventional gated Cartesian cine images acquired using the 2D conventional gated Cartesian balanced SSFP cine sequence in In this study, we proposed a manifold-based recovery scheme for the joint recovery of inversion recovery prepared free-breathing and ungated cardiac MRI. We represent each image in the time-series as a non-linear function of three variables: the cardiac and respiratory phases and inversion time. The non-linear function is realized using a CNN generator, while the CNN parameters and the phase information are estimated from the measured k-t space data. The data were acquired using a spiral GRE sequence with intermittent inversion pulses. We use a two-step strategy to realize a computationally efficient algorithm. In particular, we use a dense conditional auto-encoder to estimate the cardiac and respiratory phases from the central multi-channel k-space samples acquired at each frame. The latent vectors of the auto-encoder are constrained to be bandlimited functions with appropriate frequency bands, which enables the disentanglement of the latent vectors into cardiac and respiratory phases even when the data are acquired with intermittent inversion pulses. Once the phases are estimated, we pose the image recovery as the learning of the parameters of the CNN generator from the measured k-t space data. The learned CNN generator is used to generate synthetic data on demand by feeding it with appropriate latent vectors. The framework enables the generation of cine movies with different inversion contrasts as well as the estimation of the Phantom studies demonstrate that the proposed scheme is able to provide nd MOLLI ,34. We nThe proposed framework also offers the ability to measure cardiac function using a synthetic breath-hold cine. The experimental results show that the synthetic cine is able to provide comparable cardiac function analysis results compared to the results from the 2D conventional gated Cartesian cine. A benefit with the proposed scheme is its ability to generate multiple contrasts, some of which offer improved visualization of myocardial borders and may facilitate improved analysis.0 inhomogeneities and gradient imperfections in this study.The proposed reconstruction algorithm is general enough to be applied to other sequences, including the free-running transient balanced SSFP acquisition with variable flip angles and inversion pulses as in , as well


To make the best use of available energy resources and reduce costs, improving the energy efficiency of buildings has become a critical issue for the construction industry. Today, developing a three-dimensional model of the energy consumption rates in buildings based on thermal infrared images is essential to visualize, identify and increase energy efficiency. The purpose of this study is to suggest a methodology for generating a thermal leakage map of building facades utilizing the fusion of thermal infrared and visible images captured by Unmanned Aerial Vehicles (UAVs). In general, the proposed method involves three basic steps: the generation of thermal infrared and visible dense point clouds from the building’s facade using Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms; the fusion of visible and thermal infrared dense point clouds using the Iterative Closest Point (ICP) algorithm to overcome thermal infrared point cloud constraints; the use of edge extraction and region-based segmentation methods to determine the location of the thermal leakage of building facade’s. To that end, two datasets obtained for separate building facades are used to assess the proposed strategy. The results of the data analyses for the extraction of the desired components and determination of thermal leakage locations on the building facets provided a Precision and Recall score of 87 and 90% for the first dataset and 87 and 88 for the second dataset. Examining the outcomes of calculating thermal leakage zones indicates improving Precision and Recall. With buIn the building sector, energy inspections are conducted to discover faults or cracks, air leakage sources, moisture, cracks, and the suitability of windows to insulate and to evaluate the air conditioning system performance . ThermalBased on the results shown in Thermography is a cost-effective, non-destructive convenient method widely used not only for interpretive and numerical measurements of building thermal leakage ,11 but aAs two-dimensional thermographic images are insufficient for analyzing the exact location of thermal problems, it seems necessary to use a three-dimensional thermal model of a building’s energy status  in orderIn recent years, computer-vision and photogrammetry activists have taken a keen interest in the usage of UAVs as a result of advancements in the technology of sensors used in UAVs and their ability to provide information with high resolution and accuracy. Moreover, using UAVs increases the flexibility of data collecting and enables the combination of several sensors for data acquisition . In thisMoreover, due to UAVs capability to be equipped with various sensors, including inexpensive and lightweight thermal cameras, many activists in this field have considered UAVs as cutting-edge technology for quantitative and qualitative analysis of thermal properties and air inspection of buildings. Additionally, it is also possible to get high-quality thermal infrared images of a structure from various angles utilizing UAVs. This not only generates a thermal 3D infrared visualization and identifies thermal bridges but also assists in quantifying the thermal transmittance  of wall and ceiling surfaces through the use of thermal infrared orthophoto mosaic.In this paper, by employing the RGB point cloud as the three-dimensional geometric reference, the thermal point cloud drawbacks, such as low density and poor texture, can be overcome by fusing visible and thermal images at the point cloud level ,22. The 1.1Considering the low spatial resolution and contrast of thermal infrared images and the use of a narrow field of view (FOV) in infrared cameras , obtainiLagüela et al.  mapped tThe usual methods of homography transformation and space resection are used to map a thermal texture onto three-dimensional data. Nevertheless, the approach to using space resection has at least three drawbacks: The first is the time-consuming processing because when thermal video data is used, each image’s space resection must be performed independently. The second issue is the difficulty to determine the minimum control points in weakly textured thermal images, which can impair the final orientation’s accuracy. Thirdly, the 3D model’s mosaic of thermal images is prone to exhibit discontinuities in overlapping areas. The proposed solution is the fusion of thermal and visible images.Dadras Javan et al.  generateHoegner et al.  identifiRakha et al.  used UAVZhong et al.  developeThe method proposed in the present study generates an actual three-dimensional thermal model of the building. Additionally, by utilizing UAVs, the challenges and drawbacks associated with traditional thermographic methods, such as a lack of access to inaccessible and out-of-sight regions for evaluation and data gathering, are resolved. Moreover, by utilizing data from UAVs and aerial imagery, a broader area of buildings may be evaluated in less time. All this provides the possibility to use the ICP method along with a region-based segmentation algorithm to fusion thermal infrared and visible images and to detect and determine thermal leakage locations on the building facade.The possible outcomes of the study can be summarized as follows: comprehensive visualization of the thermal condition of building facades, improvement and removal of limitations associated with thermal infrared point cloud (low density and noise), orthophoto mosaic produced by fusing thermal infrared and visible point clouds, automatic extraction of building facade components, and finally identification and determination of areas of thermal leakage in the building facade.The paper is organized into five sections. Section 2The proposed method for thermal leakage determination is composed of three main steps . The firBased on the flowchart in To overcome the constraints of thermal infrared point cloud, such as low density and relative noise, visible and thermal point clouds were fused using the ICP algorithm. As a result, the fused point cloud contained both visible and thermal information with high spatial resolution and density. Additionally, the fused point cloud was employed to generate a three-dimensional model and an orthophoto mosaic of the building facade.Finally, a thermal leakage map was generated by extracting the building facade components using the canny edge extraction operator and segmenting the thermal leakage area through the use of the region-growing segmentation technique.2.1Thermal infrared and visible data on the building facades were collected at the Pedar Salar mansion ecotourism residence in Aliabad village, Aradan-Garmsar city, Semnan province, Iran, with longitude coordinates of 52.509185 and latitude coordinates of 35.266896. In this paper, two datasets of visible images  and thermal infrared video  were collected from the two facades of the studied building. 2.2In this study, to collect the required data, a vertical flight UAV equipped with thermal infrared and visible cameras was utilized. Additionally, in order to control and guide the UAV using telecommunications equipment, a computer, with Mission Planner software was utilized. The flight path was designed to collect data perpendicular to the building facades at a distance of 11 m and a flight height of 1.70 m from the building facades. In this paper, the imaging scale for visible data is 1:314 and for thermal infrared data, it is 1:440, based on the focal length and distance from the building facade. In addition, the UAV flew at a slow speed, and images of the building facades were collected with an 80% overlap. The MC1-640s thermal infrared camera of KeiiElectro Optics Technology at 30 frames per second and the Sony ILCE-7RM4 visible camera was selected for data collection. 2.32.3.1In this study, the self-calibration  approachc stands for the focal length, xa, ya are the image coordinates, xp, yp represent the principal point coordinates, X0, Y0, Z0 are the projection center coordinates, XA, YA, ZA stands for the object space coordinates, and r designates the rotation matrix element. Additionally, the lens distortion parameters are iteratively estimated using Brown equations [As thermal infrared cameras have low spatial resolution and low contrast, circular targets appear as ellipsoids in the image. Therefore, to solve the issue, the Hough Transform algorithm (HT)  was utilquations   video frames to improve geometric accuracy and reduce the number of computations required for the 3D reconstruction process. This step involved determining the standard baseline between successive frames by evaluating the overlap between the features extracted from the sequential frames and the Geometric Robust Information Criteria (GRIC)  to achieIn this stage, to determine the standard baseline between the frames for keyframes extraction, the first frame from the set of frames was chosen as the keyframe. Additionally, features were extracted using the SIFT algorithm . The proRC indicates the proportionality coefficient of the corresponding key points, TC is the number of corresponding key points between the two frames, and Tf stands for the total number of key points extracted from the first keyframe. Following an examination of the standard baseline, the frames would be studied to analyze degeneracy conditions.In Eq. The fundamental matrix is used to investigate the camera’s overall structure in various locations and the relationship between the corresponding features in the two frames. However, it is impossible to determine the camera’s location in degeneracy conditions. In degeneracy conditions, two important factors are motion and structural degeneracy .In degeneracy states, the image pair is matched using a homography matrix. As a result, comparisons between the homography and fundamental matrixes are made to assess degradation conditions using the GRIC criterion as the ideal selection criterion . Eq. (6)n denotes the number of corresponding inlier properties extracted in the fundamental matrix and homography solution, ei indicates the residual vector. The numerical value of k equals the motion model parameters for the homography and fundamental matrices, and d stands for the dimensions of the model structure.In Eq. Finally, using the criteria stated in Eq. 2.4Various methods in photogrammetry and computer vision, such as object bundle adjustment and SfM, have been presented to generate a point cloud and a three-dimensional model based on a set of input images. Image-based reconstruction methods such as SfM and MVS are used to generate photogrammetry outputs with a high spatial resolution  from inexpensive non-metric camera images mounted on UAVs with overlap forward and sidelap (75–80%).Processing the obtained images begins with automatic keyframes extraction ,55. ThenThe reconstructed model is converted to an object coordinate system (absolute orientation) via control points or by estimating the camera position using the UAV based on GPS (using the PPK approach). This information is then entered into the model, and the second bundle adjustment is carried out .The MVS algorithm can be used to increase the density of the point cloud, which generates a depth map for image pixels based on their image stability in the oriented bundle adjustment output block ,59,60. F2.5Various challenges are associated with producing a three-dimensional point cloud from thermal infrared images acquired by UAVs using the SfM method. For instance, the low spatial resolution and narrow FOV of thermal infrared sensors, along with blurred edges and a lack of texture in many sections of the image, can result in the failure of the SfM algorithm or may produce suboptimal results . In thisTo overcome the limitations of the thermal infrared point cloud, including the low-density point cloud, which is resulted from the low spatial resolution of thermal infrared images, it is necessary to fuse the visible and thermal infrared point clouds. In the study, radiometric information was mapped from the nearest three-dimensional point in the thermal infrared point cloud to its equivalent point in the visible point cloud using the ICP algorithm .2.6Following the fusing of visible and thermal infrared point clouds, an orthophoto mosaic was generated utilizing the fusion point cloud. The orthophoto mosaic described earlier was used to extract the components of the building facade in this paper. The first step was to generate the orthophoto mosaic temperature histogram, which was utilized to establish two appropriate thresholds for extracting components and areas associated with building facade thermal leakage. The histogram depicts the grayscale value distribution of the image pixels. Thus, the orthophoto mosaic histogram presents the distribution of grayscale values in orthophoto mosaic pixels, as well as the numerical value of the temperature assigned to each pixel.The first stage involved calculating two thresholds based on the temperature distribution to examine locations prone to thermal leakage. Rather than defining a predefined threshold, two specific thresholds for use in the edge extraction step were chosen for the initial extraction of facade components and areas prone to thermal leakage. The extraction of acceptable thresholds was done by calculating the grayscale value histogram associated with the temperature distribution. It was divided into five distinct groups empirically and through trial and error.The relevant thresholds for extracting facade components and thermal leakage locations based on histogram classification are shown in The horizontal axis in 2.7According to studies on stochastic segmentation algorithms , for detIn computer vision, image segmentation is a technique for dividing a digital image into multiple parcels or regions. The primary goal of image segmentation is to separate the image into several regions with modest grayscale value changes. In other words, there is nothing in common between the regions .The Region-Growing segmentation technique was employed in this paper to determine the locations of thermal leakage. The Region-Growing algorithm is a method for grouping pixels or image portions into more extensive regions depending on predefined growth parameters. This method works by starting with a collection of “seed” points in the image areas and nearby pixels with preset seed-like features . This algorithm’s performance is based on the fact that pixels nearby have similar grayscale value intensities.Then an initial set of small regions are merged repeatedly based on similarity criteria. When a part stops growing, another seed pixel is selected that does not yet belong to any area, and the algorithm steps start again from the beginning. The algorithm will stop if any pixel no longer meets the inclusion criteria in that part .The purpose of this paper was to segment the locations associated with thermal leakage. In this case, because the distribution of the desired image pixels in the HSV color space would provide a better separability for segmentation, seed pixels were selected from the HSV color space saturation parameter. In The distribution of orthophoto mosaic pixels in various color spaces is depicted in 2.8To assess the proposed method for determining thermal leakage areas in this paper, two well-known machine learning evaluation criteria, precision, and recall, were applied. The precision criterion is mainly concerned with the proposed algorithm’s proper detection of thermal leakage locations. In contrast, the recall criterion is primarily concerned with data that have been correctly identified as thermal leakage locations by the suggested approach.TP) to all positives plus false positives (FP). Additionally, the recall criterion can be represented by dividing true positive samples by positive samples plus false negative (Fn) pieces. Also, both assessment criteria range from zero to one, and the closer the output value is to one or one hundred percent, the more acceptable the proposed method’s performance for both precision and recall criteria will be. Precision and recall criteria are defined as follows using Eqs. The precision criteria are the ratio of true positives (•TP): The number of pixels predicted correctly by the proposed algorithm.True Positive (•FP): The number of pixels mispredicted by the proposed algorithm.False Positive (•Fn): The number of pixels we expected the algorithm to predict but it did not.False Negative  of the building facade were employed. Through self-calibration, parameters of interior orientations and lens distortions were computed. The correlation matrix and calibration parameters of the visible camera (first visible dataset) in terms of pixel measurement unit are reported in F indicates the focal length, K1 and K2 stand for the radial distortion, and P1 and P2 represent the tangential distortion of the visible camera lens.In Geometric calibration of the thermal infrared sensor was done using the calibration pattern of a rectangular plate with hollow circles. The circle targets with a diameter of 12 mm each were arranged symmetrically in 13 rows and 17 columns. The distances between the centers of the circle targets were set at 24 mm. The six circles would serve as coded targets for determining the orientation and position of the targets in the various images of the calibration pattern. Hollow circles would provide for temperature differences, which could result in contrast on the thermal calibration pattern plate. As a result, this would improve the accuracy and quality of the HT algorithm in detecting the centers of circles in two-dimensional space. In this context, a passive source (heating radiator) applied heat to a calibration pattern to reveal circles and increase contrast in thermal infrared images. Then, using a thermal infrared camera, convergent images from various orientations and angles were acquired for geometric calibration and parameter estimation. After extracting the precise two-dimensional coordinates of the center of the target focal to establish a relationship between the two-dimensional coordinates of the targets in image space and the object space, data was fed into collinearity equations to estimate the geometric calibration parameters of the thermal infrared sensor. Finally, using collinearity equations, the geometric calibration parameters of the thermal infrared camera were calculated. Fx and Fy in the above table represent the focal lengths along the x and y axes, the Cx and Cy parameters stand for the principal point coordinates, the parameters K1, K2, and K3 indicate the lens’s radial distortions, and the parameters P1 and P2 show the thermal infrared camera’s tangential distortions. The results of computing calibration parameters are reflected in In Then, the Mean Reprojection Error per image (MRE) was used to assess the accuracy of the geometric calibration of the thermal infrared camera. The MRE per Image of the geometric calibration of the thermal infrared camera is shown in The horizontal axis in 3.2In this paper, contrast enhancement of thermal infrared images was performed just to overcome the low contrast and poor texture of thermal infrared frames, to avoid frame sequence disruption, and to increase the extractable features for verifying standard baseline and keyframes extraction. To generate a dense thermal infrared point cloud, after keyframes extraction, the raw keyframes were employed before the contrast enhancement stage.The multiscale top-hat transform approach was utilized in this research to enhance the contrast of thermal infrared images. In this method, two structural elements, proportional geometric and flat, were employed, and as a result, this would improve the detail and entropy of the thermal infrared image. The outcome of the multiscale top-hat transform approach for thermal infrared frame contrast enhancement is depicted in Based on Because of the low rate of features extracted from thermal infrared frames during the standard baseline check step, the overlap condition of 0.6–0.8 was not met, and the keyframes extraction sequence was cut. In this study, the contrast enhancement step was done solely to address the difficulty of low contrast and poor texture in thermal infrared pictures. The effect of contrast enhancement on the number of features extracted in the initial 200 frames is displayed in Based on the results shown in In the present study, the density and value of the point cloud reprojection error were used to evaluate the performance of keyframes extraction in generating the thermal infrared point cloud. Keyframes are effective in producing three-dimensional coordinates because of their strong geometry and high radiometric quality.In the present work, the method for keyframe extracting was developed in Python by utilizing the OpenCV package. Therefore, for this purpose, the initial frame of the thermal infrared video file was extracted and entered the contrast enhancement stage in the first step. To continue, by determining the best threshold, blur frames were removed from the video frame dataset using the FFT metric to improve the geometric and radiometric accuracy of the dense thermal infrared point cloud generated. After inspecting the frames for radiometric quality, 853 blur frames were detected and eliminated from the first dataset with 1957 initial frames, and 214 blur frames were removed from the second dataset with 907 initial frames.Rc and extracting keyframes using the GRIC criterion for two keyframes of the first dataset.Then, using Eq. RC key points. Column B shows keyframes extraction using the GRIC criterion to compare the fundamental matrix with the homography matrix and select the fundamental matrix minimum value as the keyframe, and column C shows the extracted keyframe.The outcomes of the keyframes extraction implementation are depicted in 3.3Thermal infrared and visible point clouds were generated using image-based methods such as SfM and MVS. As a result, the stages for producing point clouds were performed independently for thermal infrared and visible data using SfM and MVS algorithms.Thermal infrared and visible data with identical flight characteristics were collected from a distance of 11 m from the facade of the examined building. The numbers of visible images captured by a UAV for the first and second datasets were 46 and 24, respectively. Also, to generate a dense thermal infrared point cloud, raw keyframes were used before the contrast enhancement step. Then to explore the influence of keyframes extraction on generating a dense thermal infrared point cloud from video data, the criteria for evaluating the output point cloud and the value of reprojection error for both datasets were considered. In this regard, increasing the density and minimizing the reprojection error would indicate the optimal effect of keyframes extraction to produce a thermal infrared point cloud.Sections (A–C) of In 3.4After generating thermal infrared and visible point clouds, the fusion of visible and thermal infrared point clouds was performed to overcome the inferior density thermal infrared point cloud caused by the low spatial resolution of thermal infrared frames. The ICP algorithm was adopted in this research to map the radiometric information of the nearest three-dimensional point in the thermal infrared point cloud onto its equivalent point in the visible point cloud. Thus the resulting point cloud had a high density and level of detail with thermal information. The data in 3.5In this stage, to extract the components of the building facade, the orthophoto mosaic generated in the previous stage was employed. The orthophoto mosaic histogram illustrates the distribution of grayscale values corresponding to each pixel. First, the Python programming language was used to extract the building facade components. Next, the frequency distribution of grayscale values was computed to generate the orthophoto mosaic temperature histogram. Afterward, two thresholds depending on the temperature distribution were employed to extract the edge pixels and the components and locations in the building facade that may have had thermal leakage. As a result, the orthophoto mosaic histogram was experimentally separated into five distinct categories with identical temperature distributions to calculate the two thresholds. As shown in 3.6In this paper, thermal infrared orthophoto mosaics and the Region-Growing segmentation algorithm were utilized to extract and determine the locations of thermal leakage in the building facade. In this regard, it seemed necessary to execute a significant pre-processing phase, including color space transformation, to more precisely pinpoint the regions of thermal leakage. Since the saturation parameter distribution in the HSV color space would allow for the extraction of areas with higher temperatures, orthophoto mosaic pixels in the range of thermal infrared grayscale values were converted to the HSV color space using the Python programming language. In •The seed pixel is chosen based on the maximum intensity of grayscale values and is compared to nearby pixels using the criterion of grayscale value intensity similarity.•Each segment increases the size of the original piece by g similar surrounding pixels starting with the seed pixel.•When a segment’s growth is complete, a new seed pixel is chosen that is not yet associated with any component, and the process is repeated from the beginning.•This is repeated until all pixels have been assigned to a segment.In The Region-Growing algorithm’s segmentation results for thermal leakage areas map on orthophoto mosaics are shown in 3.6.1Two criteria for measuring precision and recall were used in this study to evaluate the proposed algorithm for determining the locations of thermal leakage. Precision and recall criteria are expressed as values between 0 and 1. The closer the output results are to 1 or 100%, the more acceptable the recommended method’s performance will be.•Tp.The values with a numerical value equal to one in the logical matrix represent the variable •Fp.The values with a numerical value equal to zero in the logical matrix represent the variable Moreover, two orthophoto mosaics in HSV color space and segmented were to execute and assess the outcomes. It is worth noting that the orthophoto mosaic in HSV color space was chosen as a reference for evaluating and comparing thermal leakage locations. Therefore, to analysis the results, it was necessary to normalize the grayscale values of pixels in the HSV and the segmented orthophoto mosaic. Then using the Python programming language, the two orthophoto mosaics were compared pixel by pixel. Finally, the output matrix was obtained as a logical matrix following the comparison. Then the precision and recall evaluation criteria were calculated as follows:Fn variable, the obtained numerical value of the Tp variable must be deducted from the total number of pixels associated with thermal leakage locations in the HSV orthophoto mosaic.To determine the In 4In this paper, after geometric calibration of the thermal infrared camera with a mean reprojection error of 0.03 pixels, contrast enhancement and keyframes extraction from the thermal infrared video were performed to improve the geometric precision of the point cloud and reduce the volume of aerial triangulation calculations. Based on the findings reported in In this study, to overcome the thermal infrared camera limitations , the dense thermal infrared and visible spars point clouds were fused at the point cloud level using the ICP algorithm. As a result, the fused point cloud had a high spatial resolution and density, as well as thermal information. Fusion of the point clouds was estimated for both datasets using MSR values of 0.0003 and 0.0004, respectively, and the fused point clouds were found to be sixfold and fourfold denser than the initial thermal infrared point cloud, respectively.Based on the precision and recall evaluation criteria for both datasets, the proposed approach improved the accuracy of determining thermal leakage locations, as shown in •Due to the employment of an aerial platform equipped with inexpensive thermal infrared cameras, it is possible to appropriate flight operations according to the geometry of the building, which can also be highly effective for inspecting buildings with greater heights. In this regard, photogrammetry can be used to conduct periodic energy audits of facilities and to create large-scale maps of building thermal leakage.•Generally, photogrammetry eliminate the need to visualize energy loss, determine thermal leakage locations, and estimate the amount of energy loss through the use of images taken by UAVs from different views of the building and products of photogrammetry.•As the research assumes to consider areas with scattered urban or industrial buildings, we can attest that in the proposed method, the human aspect engaged in thermography is omitted in terms of participation and responsibility. In this respect, using UAV photogrammetry products and automated computer vision algorithms to study the energy condition of the structure and to pinpoint areas of thermal leakage can boost speed, and accuracy and decrease costs and time. As a result, speed and accuracy will increase, while the price and time associated with finding the areas of thermal leakage will decrease.The following sections address the proposed method’s points and advantages, as well as its accomplishments and purpose:5In recent years, enhancing building energy efficiency to maximize the use of energy production resources while minimizing costs has become a critical concern in the construction industry. The present research proposed a method to increase the energy efficiency of buildings by extracting building facade components and determining a thermal leakage map. In this work, the fusion of two data sets including thermal infrared and visible images captured by the UAV was used. According to the proposed method, thermal leakage locations on orthophoto mosaic were investigated utilizing precision and recall assessment criteria. The precision and recall evaluation criteria for the first data set were equal to 90% and 87%, and these criteria were calculated as 87% and 88% for the second data set, respectively. On the whole, the results demonstrate that the suggested method can improve the accuracy of locating thermal leakage areas in a building facade.•Utilizing thermal data that has been geotagged with GPS/IMU  helps improve the accuracy of point cloud generation and fusing of visible and thermal infrared point clouds.•To determine and segment the areas of thermal leakage, recent deep learning-based segmentation approaches can be compared to the method suggested in this research.•As a follow-up to this research, due to the requirement to improve energy efficiency, the production of a thermal leakage map of the whole building’s facade, including the walls and ceilings, can be done.•Depending on a research premise on data collection from places with scattered or industrial structures, complicated urban areas could be investigated for future research.The following are some recommendations for future research on locating leakage areas and thermal leakage map generation of the buildings:Farzaneh Dadrass Javan: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.Soroush Motayyeb: Performed the experiments; Analyzed and interpreted the data; Wrote the paper.Farhad Samadzadegan: Conceived and designed the experiments; Contributed reagents, materials, analysis tools or data.Hamidreza Hosseinpour: Performed the experiments; Wrote the paper.This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.The data that has been used is confidential.The authors declare no conflict of interest.


Cardiovascular diseases (CVDs) are the leading cause of death globally. Morbidity and disability related to non-fatal events are increasing exponentially. There are several symptoms that may arise after invasive therapeutic approaches such as coronary artery bypass graft (CABG), including chronic pain in anatomical areas connected to the mediastinum; these pains can be found not only initially after surgery but also years later. We present a case where non-specific neck pain (NNP), in a patient undergoing CABG five years earlier, was resolved with an osteopathic technique, working the pericardial area. To the knowledge of the authors, it is the first article illustrating an osteopathic approach with resolution of NNP, with a manual technique used on the pericardial area. Acute myocardial infarction (AMI) caused by coronary heart disease (CHD) is the result of one or more coronary artery stenoses . AMI is This practice, especially if performed with median sternotomy, ministernotomy or lateral minithoracotomy, stimulates the post-surgical formation of peritoneal adhesions ,6,7. CABA 36-year-old male patient with previous cardiac surgery in median sternotomy, not infarcted, was examined for NNP in the cervical area. Surgery was performed five years earlier, with double CABG . Four days after surgery, the patient underwent thoracentesis due to a left pleural effusion involving the pleural space up to the sixth rib. The drugs currently being taken by the patient are aimed at blood pressure control (beta-blocker and angiotensin-converting enzyme—ACE—inhibitor), and are taken due to the presence of hypercholesterolemia and dyslipidemia (statins). The patient complained of pain and stiffness throughout the posterior cervical area, without irradiation, with movement limitations that increase with rotations and inclinations of the neck to the right.The relief came from the extension, and from the opposite rotation and inclination. The deep pain was intermittent and present only with active and passive (performed by the operator) right movements. Once a day, the patient takes a pain reliever . The patient did not report direct trauma or cervical pathologies or axial alterations with negative results through previous instrumental tests (magnetic resonance and X-rays). Blood tests do not highlight systemic diseases .Air saturation was 100%, and with the complete absence of dyspnea. With the visual analogue scale (VAS), if pain was solicited, the score was six; the neck disability index (NDI) reported a value of 24 (moderate disability). We excluded patent foramen ovale (PFO), which can be a source of neck pain, because it was not present during surgery. We excluded carotid diseases , which can cause pain in the neck, thanks to the color Doppler ultrasound of the carotids previously performed, before the osteopathic visit. We excluded the temporomandibular joint (TMJ) as the origin of the symptoms, due to the absence of popping, clicking or crepitus, as well as the lack of pain and muscle tension on movement and palpation by the osteopath , and no hearing disorders and/or alterations of the dental class. The patient denies bruxism or nocturnal snoring. We excluded Eagle’s syndrome and hyoid bone syndrome due to the lack of related symptoms , and due to the palpation of the hyoid bone. We performed several manual differentiation tests (less specific like the O’Donoghue’s test or more specific like the Sharp–Purser test), but always inconclusive. The cervical compression and distraction tests were also negative.We manually evaluated the vertebrae that affected rotation and inclination the most, such as the first two cervical vertebrae , C3–C4 and C6–C7 (affect inclination by about 11.7–20.3%), respectively . By isolThis test consists of pressing a tissue area (sternum) while resting the other hand on the symptomatic area (the neck). Then, the clinician tries to perceive a change in tone. Finally, the cervical spine area is pressed, and the response is felt from the sternal area . The tisThe result of the instrumental examination highlighted an adherence between the pericardium and the pleura in the left mediastinal area, and an elevation of the left hemidiaphragm. We decided to perform an osteopathic treatment with a technique used in a previous trial, that is, an indirect approach . The goaWith this manual support to the chest, we tried to “enclose” the pericardial area. The clinician does not induce any movement but follows the movement of the tissues underlying the hands; when the perceived tissue displacement is homogeneous in amplitude, traction force, and times of expression, the technique can be said to be finished. The patient remains seated, while the operator is seated next to the patient. The application time can last several minutes. At the end of the osteopathic technique, when reevaluating the patient, the active and passive movements of the neck had no limitations; utilizing the previous assessment scales (VAS and NDI), the patient did not complain of any pain. After a week, the patient (contacted by telephone) described the recurrence of pain but with a much-reduced entity compared to the period preceding the osteopathic treatment and without taking NSAIDs.Movements were unrestricted. We decided to see the patient again (one week after the first treatment), using the same osteopathic approach. After two weeks, the perceived pain was further decreased, but constantly present, and the neck movements improved more. After 3 months of treatment with the same approach, with one monthly session, there was no limitation of movements (rotation and tilt to the right); the pain returned two weeks after the osteopath visit, but the painful entity perceived by the patient had greatly decreased, with a VAS score of 1 and NDI with a value of 10 (minimum disability). We decided, in agreement with the patient, to have one session every six months, as maintenance therapy.There are many symptoms that can derive from adhesions, such as local and/or referred pain, or they can cause symptoms not necessarily related to pain, such as being the cause of constrictive pericarditis . In thisThe present pain could be connected to the mechanical tension generated by visceral adhesion, stimulated whenever the neck moved to the right. The nervous pathways put under tension are the same ones that innervate the pleural, pericardial area (vagus nerve and sympathetic system) . NoxiousMuscle motor fibers traveling with the cervical and mediastinal fascia can be negatively solicited by mechanical tension abnormalities and form nociceptive information: movement with pain . The phrIt is not easy to identify the cause of NNP. Studies link the presence of this chronic disorder with other clinical pictures or behaviors, such as depression, anxiety, smoking, lack of regular physical activity, unrestful sleep, and previous traumas . The etiThe osteopathic manual approach carried out in this case report finds its clinical validity in a previous trial, where we demonstrated that by working on the mediastinal area, the patient recovers their motor performance more quickly and with a significant decrease in pain perception . The reaThis case report presents a patient with chronic non-specific neck pain (NNP) in the presence of pericardial-pleural adherence following thoracentesis and prior CABG. To the knowledge of the authors, this is the first article illustrating an osteopathic approach with resolution of the NNP, with a manual technique used on the pericardial area. The real reasons for the benefit achieved by the patient need to be investigated.