The occurrence of barely visible impact damage (BVID) in aircraft composite components and structures being in operation is a serious problem, which threatens structural safety of an aircraft, and should be timely detected and, if necessary, repaired according to the obligatory regulations of currently applied maintenance methodologies. Due to difficulties with a proper detection of such a type of damage even with non-destructive testing (NDT) methods as well as manual evaluation of the testing results, supporting algorithms for post-processing of these results seem to be of a high interest for aircraft maintenance community. In the following study, the authors proposed new approaches for BVID reconstruction based on results of ultrasonic and X-ray computed tomographic testing using authored advanced image processing algorithms. The studies were performed on real composite structures taking into consideration failure mechanisms occurring during impact damaging. The developed algorithms allow extracting relevant diagnostic information both from ultrasonic B-and C-Scans as well as from tomographic 3D arrays used for the validation of ultrasonic reconstructed damage locations, which allows for a significant improvement of the detectability of BVID in tested structures. The developed approach can be especially useful for NDT operators evaluating the results of structural NDT inspections. Polymeric composite materials, due to their numerous advantages, have found a very wide application in many branches of industry, especially the automotive, aircraft, aerospace, or marine. They have become competitive alternatives to traditional metallic materials mainly due to significant weight savings, greater strength and stiffness, and corrosion resistance.However, due to their complex nature, anisotropy, non-homogeneity, and construction with interfaces, polymeric composites are characterized by low performance at high temperature, poor through-the-thickness properties, and low resistance to impact loadings . ComposiCommonly applied damage tolerance methodology involves allowing composite components with internal damage to be operated if certain conditions are met . If the One of the most commonly applied NDT methods during routine inspections of aircraft composite structures is ultrasonic testing (UT). In comparison to other commonly used NDT methods, UT is the leader concerning the capabilities for the detection, sizing, and depth estimation of damage . The reaBesides the above-presented technical factors, various human-related factors are of high importance, since they often determine the quality of the evaluation and prognosis of structural properties. One of the crucial factors influencing the results is that raw UT results are often processed manually, which is a typical practice in aircraft inspections mainly due to complexity of the obtained data, and thus, the impossibility of a substitution of an inspector by computer algorithms. During industrial ultrasonic inspections, e.g., during testing of aircraft structures, the UT results have to be analyzed and interpreted by a certified expert, having appropriate knowledge about the specifics of UT, mechanics of composite structures, and the form of the tested element . This often leads to oversights and misinterpretations due to the physical impossibility of the detection of structural changes based on UT raw results in some cases. Therefore, in order to increase the efficiency of the UT results analysis, there are some methods described in research studies, which can be found in the literature, that are focused on applying image processing techniques to the ultrasonic scans in order to extract more diagnostic information than is possible using commercial software dedicated to UT.A 3D damage reconstruction based on ultrasonic scans is one of the presently undertaken goals in the area of UT results visualization, which can provide a very significant support to the inspection operator. The necessity of 3D reconstruction of UT results to enhance damage visualization was noticed and communicated more than 30 years ago, when UT systems started to be digitalized due to the development of computer technologies , howeverPrevious studies of the authors of the following paper include numerous processing and reconstruction algorithms developed for the enhancement of damage detectability and improvement of decision-making processes. The first studies on the enhancement of damage detectability by its 3D reconstruction were introduced in , where tThe aim of this study is the development of universal and automatic algorithms for processing and reconstruction of ultrasonic data obtained using B-and C-Scan testing techniques. In order to verify the quality of the obtained reconstructed data from UT, the validation was performed based on comparison with X-ray CT reconstructions. Since X-ray CT is the most accurate NDT method, which additionally provides raw 3D data for a tested structure, it can be assumed as the reference data for such a comparison. In this study, all the processing and reconstruction algorithms for both UT and X-ray CT data are based on newly developed authored methods with an improved impact damage detectability accuracy and automatization. One of the purposes of the reconstruction of ultrasonic scans is the simplification of the evaluation process of structural damage performed by NDT operators. However, the accurately reconstructed damage sites give much more possibilities of their application. According to the currently developed trends in structural condition evaluation and maintenance, e.g., the condition-based maintenance (CBM) program, besides the structural diagnostics, which includes three main steps , it provides also the fourth step—prognostics. It should be highlighted that prognostics deals with fault prediction before it occurs, i.e., estimation of the residual strength of a tested element, which is not possible when applying the diagnostic methods only. The reconstructed damage from UT inspections, including the exact shape of damage, can be used for structural integrity evaluation and prognosing the condition of composite structures via inverse finite element (FE) models, which is the main goal of the currently implemented research project. The presented results are a part of this ongoing research.Despite the excellent mechanical properties of composite materials and structures made of them, they reveal high vulnerability to impacts, especially CFRP structures, which can be classified due to the impact velocity to low-, medium-, and high-velocity impacts. Although medium- and high-velocity impacts cause visible impact damage (VID), which is well detectable by observing a damaged region with a naked eye , the low-velocity impacts (LVI) result in structural damage, which can be detected using special NDT equipment, since the signs of this damage are invisible or barely visible. This type of damage is called barely visible impact damage (BVID). Due to the lack of clear indications of damage, its presence is very dangerous for structures being in operation and subjected to any kind of loading, since their presence can be overseen. Therefore, as it was mentioned in the previous section, currently implemented maintenance methodologies of composite structures, especially in the aircraft industry, require periodic inspections of such structures in order to detect such a type of damage timely and monitor it appropriately.In aircraft structures, BVID may occur at various operation circumstances, including hailstorms and bird strikes during in-flight conditions, stone lofting during takeoff or landing, and tool-dropping during ground maintenance ,30. The The failure mechanism of composite structures subjected to BVID is highly complex due to numerous interactions between components of a composite as well as the appearance of different types of stresses and their redistribution. Such a complex mechanism is a result of energy dissipation during LVI, in contrast to high-velocity impacts, where the impact energy is concentrated on a small area. During these dissipation processes, most of the dissipated energy causes deformations within the elastic range, while a part of this energy results in fracture of the resin and reinforcement.As it was mentioned before, the indications on the surface of a composite structure after LVI might be barely visible or even invisible, while inside this structure, extended damage usually appears. Interestingly, the geometric structure of BVID has a form of a truncated cone, and is called the pine tree distribution damage ,33 see . This chThe true distribution of various types of damage occurring during LVI is difficult to predict due to numerous factors influencing on the failure process, including a shape of impactor, materials composition, its type and properties , boundary conditions, a thickness of an impacted structure, and many others. However, one can distinguish several main fracture mechanisms which appear in the majority of such cases. The indentation or crushing (depending on the type of a composite) is a type of damage with negligible influence on the structural stiffness and integrity, and usually it is the only visible sign on an impacted surface. The matrix cracks appearing in a composite structure due to LVI are generally of two types: skew cracks occurring at shear stresses and transverse cracks occurring under the influence of bending stresses see . The appIn practice of aircraft ground maintenance, regardless a low influence of BVID on the overall structural stiffness and integrity in general, this type of damage remains one of the most dangerous, since it cannot be detected without a specialist equipment used for advanced NDT. This creates a possible situation where BVID can be left undetected, and may grow under operational loading, which, in consequence may lead to failure of the entire structure. Therefore, the development of appropriate methods of the estimation of BVID extent and the investigation on its influence on structural performance and behavior under operation loading is of high importance for the safety of these structures.The evaluation of BVID and further development of algorithms for damage reconstruction was performed on a CFRP structure subjected to controlled impact loading. In order to obtain valid results, the reconstruction was performed based on data acquired from three approaches to NDT applied to the tested specimen.The CFRP specimen used in this study was purchased from the Dexcraft s.c. . It was manufactured in the VARTM resin infusion technology using 2 × 2 twill weave carbon fabric as the reinforcement and epoxy resin. The nominal dimensions of the specimen were as follows: 100 × 100 × 2.5 mm (width × length × thickness).The impact damage was introduced using the in-house drop weight testing machine see , which sA visual observation of the impacted specimen see  allowed The UT was performed in the NDT lab of Air Force Institute of Technology in Warsaw (Poland). The UT parameters were selected based on the previous studies of the authors presented in . The tes® scanner with a 128-element phased array transducer and a frequency of 5 MHz. The obtained scanning resolution was of 0.508 mm per pixel. A representative set of the acquired B-Scans is presented in For obtaining B-Scans the phased-array UT method was used. In this case, the tests were performed using the FlawInspecta® water-coupling system. The test was performed using a single straight-beam probe with a delay line with a frequency of 5 MHz, and scanning resolution of 0.254 mm per pixel. The obtained ultrasonic C-Scan, also in the amplitude and ToF modes, is presented in The C-Scan used for the reconstruction was obtained using the pulse-echo UT method with the use of the MAUSThe results obtained using both UT techniques indicate the presence of multiple delamination between the composite layers at several specimen depths. As mentioned before, a disadvantage of the presentation of the results in the C-Scan mode is that damage areas located near the top surface of the tested object cover damage fragments being below them. Thus, this is typical for impact damage that there is a partial loss of the diagnostic information visible in the C-Scans. Moreover, the black areas visible in the ToF C-Scan in 2 per pixel. The images (X-ray CT slices) used in this study were generated every 0.01 mm in front section  using the Nikon/Metris XT H 225 ST Computed Tomography System. In order to obtain possibly the highest resolution of the X-ray CT results, the specimen was cut to a smaller section containing the impact damage with surrounding areas, which resulted in reducing the specimen’s dimensions to 40 × 70 × 2.5 mm with the scanning area of 40 × 40 × 2.5 mm, and obtaining the image resolution of ca. 0.022 × 0.022 mm® and are described below together with the results obtained for the considered specimen case.For all the obtained types of NDT results, the new processing methods were developed in order to automatically detect the damaged areas and visualize them in 3D properly with respect to the real specimen’s dimensions. These algorithms were implemented using MatlabA general scheme of the reconstruction algorithm developed for processing of UT B-Scans is presented in Firstly, the B-Scans, which were acquired in a form of video data, are converted to images that correspond to the ultrasound reflections in adjacent transverse cross sections of the specimen see a. The daThe next step relies on an automatic removal of the B-Scans’ sections that represent the signal reflections from the specimen’s boundary surfaces. For this purpose, for each row of processed B-Scans the extreme positions of surfaces are found by finding two minima values that are nearest  from two found biggest maxima values . This method consists in the extraction of image histograms for all the B-Scan layers (images) and setting them together in sequence see a, which Subsequently, the background  is removed and damage is visualized see a after aA general scheme of the reconstruction algorithm based on UT C-Scans is presented in An original C-Scan image see a consistThe second stage relates to 3D matrix building. For this purpose, calculations are made to assign the real values of depths of particular damaged composite layers. Firstly, the obtained values of classes (segments) in the segmented image are substituted by mode (most frequent) values that appear in the segments’ locations in the original C-Scan. This allows for the elimination of noise from the original C-Scan and obtaining planar images of all individual delamination layers. Then, their depth locations (in mm) are calculated based on known relation between the ToF C-Scan values and the real dimensions. Additionally, a mathematical relation is computed in order to estimate the depth location of the delamination identified in the dead zone  are converted from RGB  within a particular image, which refers to the weave of the composite fibers. In order to detect damage fully automatically, a nonparametric texture-based image segmentation method was developed. The principle of operation of this algorithm was based on the assumption that any disorders of the image texture are considered to be damage. Due to the variable size of fibers visible in the CT slices over the composite thickness cf. a–c, the For each image row/column, i.e., data vector with the intensity values  is extracted from the original minimum values after corrections made using several steps. In order to detect all potential texture anomalies, including the intensities just slightly lower than the model texture, the minimum values vector is filtered using a one-dimensional median filter of a fourth order , i.e., slightly smoothed. Then, in order to correct the vector fragments not matching to its overall trend, several calculations are made. The original vectors with the statistical values are significantly smoothed see b by filtThe binarization using thresholds calculated based on image columns  are extracted, which gives the final binary result of the detected damage layer. The final binary results for all the exemplary cases presented in Due to the very large data size, depending on used computing capabilities, it might be necessary to reduce the size of each resulting binary image before displaying the 3D damage visualization. In the last step of the algorithm, real dimensions of the resulting images are calculated based on known image resolution, step of slicing in the thickness direction, and resizing ratio (if the binary slices were resized), and the image axes with labels are appropriately adjusted. The 3D damage visualization is presented in The obtained reconstructions show the damage distribution typical to BVID see , which iSince all the damage reconstructions were performed using different types of data originated from the results of NDT methods exploiting different physical phenomena, it is difficult to compare them directly. However, the authors performed a qualitative, i.e., visual, as well as an attempt of a quantitative analysis of the obtained results.When analyzed the reconstruction results visually, it was noticed that the results of UT, both in B-Scan and C-Scan modes, are a reliable source of the diagnostic information about BVID presence in the tested object since the multi-level character of such a type of damage is well detectable. The 3D damage visualization gives much enriched information than the raw UT results themselves, since the damage distribution over the composite thickness can be easily observed and analyzed. An online display of such a visualization during inspections could be useful for UT operators and helpful in fast identification of the existence of BVID or another type of damage.When comparing the reconstruction results obtained using the phased-array (B-Scans) and pulse-echo (C-Scans) UT techniques, it was noticed that the latter allowed obtaining more details about the shape of damage, including a number of damage layers and their extent cf. . HoweverAnother conclusion about the real shape of the introduced BVID can be made when observed the reference CT reconstruction results, namely the damage layers (see the side views in 3 as well as the area of planar projection (see 2 with other geometrical properties were calculated. The results obtained based on different NDT data are summarized in Based on known resulting image resolutions and calculated thicknesses of the layers, the volume of the detected damage regions in mmtion see  in mm2 wThe analysis of the quantitative data stored in The presented results show that it is difficult to select the one appropriate NDT data type and reconstruction method that could dimensionally map the damage. However, summing up the results of qualitative and quantitative analyses, it can be concluded that the C-Scan reconstruction without gaps filling brought the best results. The authors noticed that such the reconstruction with only a partial gaps filling could be a better solution, which is planned to be developed in further studies.The results presented in this paper are a part of on-going research studies, and it is planned to perform a quantitative analysis, including results from FE simulations, based on the developed algorithms and a larger representation of experimental data. However, it should be underlined that the results obtained within this study allowed to gain knowledge on sensitivity of particular NDT methods to BVID and create further possibilities of simulation of BVID in order to predict structural residual life basing on computational models only.


Automatic image annotation not only has the efficiency of text-based image retrieval but also achieves the accuracy of content-based image retrieval. Users of annotated images can locate images they want to search by providing keywords. Currently most automatic image annotation algorithms do not consider the relative importance of each region in the image, and some algorithms extract the image features as a whole. This makes it difficult for annotation words to reflect salient versus non-salient areas of the image. Users searching for images are usually only interested in the salient areas. We propose an algorithm that integrates a visual attention mechanism with image annotation. A preprocessing step divides the image into two parts, the salient regions and everything else, and the annotation step places a greater weight on the salient region. When the image is annotated, words relating to the salient region are given first. The support vector machine uses particle swarm optimization to annotate the images automatically. Experimental results show the effectiveness of the proposed algorithm. With the development of digital and Internet technologies, there are massive numbers of digital images on the Internet. It is also becoming harder and harder for users to find these images accurately and quickly. The dramatic increase in the number of images on the Internet combined with the subjectivity, uncertainty and laboriousness of manual annotation has led to a gradual failure to apply text-based image retrieval (TBIR) to large-scale image retrieval . In compSegmenting image is very important for automatic image annotation, greatly influencing the annotation results. Most existing automatic image annotation algorithms do not consider the relative importance of different parts of the image to the user when segmenting the image. Other annotation algorithms do not segment the image at all but extract the whole image as a feature. In this paper, visual attention mechanism is introduced into the annotation process. During pre-processing, the image is divided into salient and non-salient regions. During annotation, salient regions of a image are weighted to give them higher priority. This weighting indirectly enhances the performance of image retrieval  of the image.Since this paper uses a support vector machine (SVM) for automatic image annotation, we divided the existing automatic image annotation algorithms into two categories: automatic image annotation using a support vector machines and automatic annotation based on other methods.Automatic image annotation based on SVM: Cusano et al. proposed a classification system using multi-class support vector machines for automatic image annotation which can be used for large-scale video and image management 11], SIA and FICEWhen giving image annotation words, the algorithm considers the relationship between words, as described in section 4.3. Before marking each image, the probability relationship between words is updated. For example, if an image is labeled sun, sky, river, clouds, the frequencies of these four words is increased by one, changing the probabilities associated with these four words. As the image library is continuously updated, the probability of all words and other words is also continuously updated. The final image annotation results are affected as a result. In this paper, the visual attention mechanism is fused into the image an notation process, so that the salient areas in the image can be labeled out first. This is often ignored in past annotations. As can be seen from the above two tables, the results of annotation using the PVSVDD algorithm show the salient areas of the image very well. The annotated words in front are marked the salient regions of the image. These salient regions are the contents that best reflect the content of the image, and are also needed for people to search. The final result can not only reflect the content of the image more accurately, but also improve the retrieval performance of the image.Automatic image annotation not only has the efficiency of text-based image retrieval but also achieves the accuracy of content-based image retrieval. With it, users can find desired images by using keywords. However, until now, almost all the image annotation algorithms extract image features by extracting them without distinguishing the importance of different regions of an image, using the undifferentiated image as training for the annotation process. However, since the keyword given by the user is intended to represent a particular region of interest within the image, if the region of interest of the image can be extracted before annotation, the annotation results can be effectively improved. This leads to better image retrieval.This paper explores these problems. Image preprocessing extracts salient regions of the image with the addition of the visual attention mechanism. A CPAM algorithm performs feature extraction on salient and non-salient regions, producing the corresponding eigenvectors. After weighting the eigenvectors of the salient regions of the image, the non-salient region eigenvectors are fused to obtain the final eigenvectors of the whole image. The SVDD algorithm based on PSO optimization is used to train and corresponding annotation models are obtained. Finally, the algorithm combines marked words and non-prominent marked words to produce the annotation results.


This article discusses the issue of Automatic Target Recognition (ATR) on Synthetic Aperture Radar (SAR) images. Through learning the hierarchy of features automatically from a massive amount of training data, learning networks such as Convolutional Neural Networks (CNN) has recently achieved state-of-the-art results in many tasks. To extract better features about SAR targets, and to obtain better accuracies, a new framework is proposed: First, three CNN models based on different convolution and pooling kernel sizes are proposed. Second, they are applied simultaneously on the SAR images to generate image features via extracting CNN features from different layers in two scenarios. In the first scenario, the activation vectors obtained from fully connected layers are considered as the final image features; in the second scenario, dense features are extracted from the last convolutional layer and then encoded into global image features through one of the commonly used feature coding approaches, which is Fisher Vectors (FVs). Finally, different combination and fusion approaches between the two sets of experiments are considered to construct the final representation of the SAR images for final classification. Extensive experiments on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset are conducted. Experimental results prove the capability of the proposed method, as compared to several state-of-the-art methods. The classification of Synthetic Aperture Radar (SAR) targets into different classes is one of the most challenging algorithmic aspects of radar structures. Unlike optical remote sensing, which cannot achieve its role in bad weather and at night, SAR can operate in all-weather conditions day-and-night and make very high resolution images, and it has played an important role in military and civil applications, such as target classification, reconnaissance, and surveillance. However, the comprehension of SAR images requires specialists, because unlike natural images, SAR images reflect the backscattering electromagnetic wave intensity of targets and speckle. Moreover, for humans, searching for targets of interest in the massive SAR images will take a lot of time. Furthermore, SAR images are covered with speckle noise, which is an important reason behind the reduction of the images quality. Besides this, they are very sensitive to the variation of target pose, and vary suddenly and quickly with small change in aspect angles. Therefore, SAR Automatic Target Recognition (ATR) is a demanding mission and has become a serious research topic for many applications. In the last decade, several methods were proposed in order to analyze and classify SAR images. A basic architecture of SAR ATR was defined as three phases: Detection, discrimination, and classification . The priOf the numerous methods, each approach that has been proposed to deal with the classification stage is included in one of the three main taxonomies, namely template-based , model-bDeep CNN is currently one of the most promising technologies for classification. It has obtained the state-of-the-art results for object detection and automated land cover classification in high-resolution remote-sensing imagery. Deep CNN has been used to classify a wide variety of remotely sensed imagery, including SAR images , hyperspOne technique consists of applying transfer learning from pre-trained deep CNNs that can be modified to produce accurate classifications for specific applications. The main idea of CNN transfer learning is to use the power of networks that are already trained on a large-scale data set of general images in different application areas. These types of networks can be regarded as general feature extractors or classifiers. This technique can be successfully applied in cases where we do not have a large data set for training a CNN from scratch. There are three famous scenarios for transfer learning of a CNN: A pre-trained net is used as a fixed feature extractor, a pre-trained net is used for fine-tuning and a pre-trained net is used for initialization in training from scratch ,28,29. IAnother interesting point that is worth noting is the use of feature coding approaches in the process of images classification. The Bag-of-Words (BoW)  approachAnother important technique that can be utilized to improve classification accuracy is fusion. Sensor fusion is a common technique in signal processing to combine data from various sensors. Feature fusion is another current method, ranging from simple concatenation to very advanced methods like fuzzy integrals. Finally, information fusion merges independent results from signal processing techniques that otherwise can be used alone as the final signal processing result. For example, Reference  has obtaThe CNN architectures proposed in the literature have proven that the activations from high-level layers of CNNs can generate powerful feature representations with outstanding performance. However, we noticed that the best classification accuracy for individual SAR classes varied among the different CNN architectures. In addition, features extracted from lower-level layers, particularly the convolutional layers, lack sufficient study. Only a few works has been conducted in this area; for example, Reference  used croWe simply calculate the CNN activations for the entire SAR images and consider the Fully-Connected (FC) layer activation vectors as the global feature representations for all images.We first compute dense CNN activations from the last convolutional layer of the input image, and then we convert them into a global representation using the Fisher encoding. Then, the global features of the image are fed to a simple classifier for the classification task.The two reasons mentioned above encourage us to propose a new framework. The framework is drawn from the most contemporary techniques in image processing and deep learning. It is a combination of three different CNNs. The three CNNs have the same architecture; the main difference is the sizes of the convolution and pooling kernel in each one of them: Coarse Grain CNN (CG-CNN), Middle Grain CNN (MG-CNN) and Fine Grain CNN (FG-CNN). We train each CNN from scratch using the chosen sizes of convolutional and pooling kernel. We show that through a combination of recent techniques, we can obtain significant performance improvement on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset classification task. We investigate how to obtain better accuracy classification compared to the state-of-the-art results that use the same dataset without data augmentation by forming better representations from the SAR images using CNNs activations. By deleting the last few layers of a CNN, we handle the remainder of the CNN as a fixed feature extractor. Considering that these CNNs are broad multi-layer architectures, we consider two ways of extracting CNN features with reference to different layers:Extensive experiments prove that powerful features SAR images can be generated. The resulting features and classification outputs from each of these CNNs are then either combined or fused using a variety of methods into a final refined classification. Evaluation of the proposed approach is conducted with the MSTAR benchmark data set. Experimental results validate the superiority and effectiveness of the proposed approach.The paper is formulated as follows. In In this section, a quick overview of the previous studies related to SAR target recognition is given. Cui Z  reviewedOver the last decade, many researchers have focused on the classification of SAR ATR using CNN, which normally refers to supervised classification. Using randomly sampled SAR targets patches, Reference  extractsDeep CNNs are neural networks whose topology encodes a spatial relationship between nodes in successive layers that is analogous to the convolution operation. A deep CNN is an algorithm, which learns a nonlinear classification function from the labeled input data. It is characterized in this way: First, an input layer, where the algorithm receives the SAR images of the MSTAR dataset and transfer them to the next layers for further processing. Next, a number of hidden layers follows the input layer. The role of hidden layers is to perform feature extraction and aggregation from the raw data. In a typical CNN, the hidden layers are alternating sequences of convolutional, batch normalization, ReLu and pooling layers. Finally, the architecture has an output layer that acts as a classification mechanism. Each layer has neurons that act upon the input features and pass the result to the next layer. An effective network learns weights in each layer, which produce accurate classification outputs for their input images. When training a deep neural network, the main challenges are in setting up the correct topology, choosing appropriate parametric initializations, defining an objective function to quantify misclassification error and an appropriate algorithm for training. All these factors affect the performance of the network. A multitude of extensions to the basic architecture and training mechanism have been proposed, the most recent one which have better results is composed of multiple cascaded stages, as is shown in The convolutional layer’s outputs are feature maps; each one is computed by a dot product between the local regions (receptive field). Training CNN is complicated since the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change . This slows down the training by requiring lower learning rates and careful parameters initialization, and makes it notoriously hard to train models with saturating nonlinearities. Therefore, a Batch Normalization (BN) layer is added before the nonlinearity where elements of feature maps, at different locations, are normalized. The BN layer’s role is to reduce the internal covariant shift and the gradients dependence on their initial values or on the scale of the parameters. It also reduces the need for dropout and regularizes the model . PrincipFeature coding is a key component in image classification. It has been extensively studied over the past decade, where numerous coding algorithms have been proposed. The bag-of-features (BoF) , developFVs coding-based methods ,42 estimGenerally, FVs coding process can be divided into front-end stage and back-end stage. In our case, the front-end stage consists of extracting features from the last convolutional layer. The back-end stage employ GMM as the generative model for these local features. The FVs are then derived from the gradients with respect to the GMM parameters. Most existing methods employ a diagonal GMM as a generative model to represent the distribution of local features, for instance SIFT  and HoG x is represented by the codebook B in feature coding. This process generates responses on M codewords, consisting of a coding vector v with M elements. In most coding algorithms, only a part of codewords is chosen to represent a feature, and therefore the coding vector v is usually sparse, i.e., the responses are zeros on the codewords which are not chosen. The GMMs define the probability density distribution of features in IFK. Its parameters nth Gaussian distribution can be generally estimated by the Expectation Maximization (EM) algorithm [Let lgorithm . SupposiThe derivative to eference . Thus, iferences ,45, withThe FVs are obtained via quantizing the set of local feature descriptors with a small codebook and aggregating first and second order residual statistics for features quantized to each centroid. In other words, the fisher encoding captures the average first and second order differences between the image descriptors and the centers of a GMM, which can be thought of as a soft visual vocabulary. The residual statistics of centroids are concatenated together to attain the high-dimensional global descriptor representation. The performance has a proportional relation with the dimensionality of the global descriptor as shown in Reference . FVs can be aggregated on descriptors in the image  which isIn this section, we focus on information fusion of the CNN classifier outputs to obtain improved results for the classification of the MSTAR data sets. The techniques summarized in the following are used in this paper to fuse the different CNN architectures outputs to obtain the final classification result. In our subsequent classifier fusion discussion, we use the following notation:kc, given a target X, is represented by a column vector.m is the component of the output vector k,my represents the estimated posterior probability that target X belongs to the class m, estimated by classifier kc. k,my satisfies the following two requirements:Suppose that we have a set of K classifiers, each of which classifies targets into one of distinct M classes. The output vector of classifier kθ is:The classification decision of classifier M classes; The fusion takes place using the Each network has a different classification output from the other networks. This type of fusion select the max per-element (for each target) classification output across all classifiers. In simple terms, the network that has the greatest confidence in its classification is accepted as the final output classification.Then, the predicted class is computed using This takes the highest confidence class for each . A null classification is generated, if the majority of CNNs do not agree on a class. This is counted as a missed classification during evaluation.Then, the classification is simply jth row of the classification matrix, thenThis takes the classification matrix C and computes a weighted confidence across all classifiers for each class using the classification accuracy . Intuitively, these weights are calculated from each network by its past performance measured from the cross-validation testing.The predicted class, For SAR image classification, several CNN architectures were proposed in the literature and have proven that the activations from high-level layers of CNNs can generate powerful feature representations with outstanding performance. However, two important points were noticed. First, the best classification accuracy for individual SAR classes is varied between the different CNN architecture and second, features extracted from lower-level layers, particularly the convolutional layers, lack sufficient study. These two reasons have encouraged us to propose different scenarios for utilizing CNN features for SAR images classification for the sake of investigating the effectiveness of features from the last convolutional layer and FC layer as well as the effect of combination and fusion different features and classifiers. We use a set of different convolutional neural networks learning at different kernel sizes of convolution and pooling to produce high level invariant features;The high-level features obtained from the FC layer are classified for each CNN model separately.The features obtained from the last convolutional layer are classified after being encoded using the FVs coding.A final feature vector from the FC layer is obtained by different combination of results of steps 2 and 3.A final classification decision using different fusion methods based on results of steps 2 and 3 is obtained.The proposed method of SAR image classification is based on the following steps:In this work, we have elected to apply the network topology presented in The choice of different parameters of each CNN model was mostly based on both previous types of research architecture ,27,28,29Accuracy is calculated from each FC layer of the three CNN models using the SoftMax classifier, as well as from the dense features obtained from the last convolutional layers of each model, using the SVM classifier after being encoded by FVC. The results of each individual CNN model motivate us to use different techniques of combination and fusion. Since the CNN which produced the best classification accuracy for individual classes is varied between the different CNNs, this suggests that differences in the CNN architecture types may allow one network to consistently perform better on a subset of the SAR classes even though it underperforms, on average, across all the classes compared with another network. Consequently, feature combination and fusion of the CNNs information outputs as shown in In our experiments, we implement different CNN models for feature extraction and classification and try different combinations of CNNs to improve the classification accuracy. Moreover, further treatment was applied using the FVs to improve the quality of the obtained features, followed by several combination and fusion methods. In this section, we investigate the representative power of CNNs features and evaluate the proposed models on real SAR image classification problem. The detailed experimental setup and numerous experiments with reasonable analysis are presented. We evaluate the effectiveness of CNN features on the following publicly available SAR dataset.The MSTAR benchmark data set is generally used to compare and test the performance of SAR-ATR algorithms. It was collected by the Sandia National Laboratory SAR platform . HundredBefore introducing the experimental setup, it is worth noting that only the original number of images is used i.e., No data augmentation algorithm is applied. Each target with the same serial number in the training and testing sets differs in the azimuth and depression angle. Images for training are captured at a 17 ° depression angle, and images for testing are acquired at a 15 °C depression angle. We resize all images to For the first set of experiments, all the CNN models introduced in We evaluate the time consumption (measured in terms of seconds) of training and testing all the three CNNs models all image in the MSTAR dataset, shown in For the second set of experiments, we evaluate the global CNN features of the three CNN models. Dense features are extracted from the last convolutional layer and then encoded into global image features through one of the commonly used remote sensing feature coding approaches which is FVs coding. The dense features are L2-normalized prior to applying feature coding. We empirically set the number of Gaussian components in the GMM to 100. We randomly select samples of each class for training the SVM classifier. The classification accuracy is measured by  library  is used  library  for implIn the first set of experiments, we generated the global image representations solely via the CNN models. We can take it a step further by combining the features computed by these three frameworks. We use two techniques of combination: (1) addition and (2) concatenation, to combine among the extracted features. For addition, the size of transformed features is the same. In the concatenation combination, the size of the features is multiplied by 3. Finally, SVM is used for classification. Here, we present some tentative combinations using the three different CNNs, and the results are shown in In the second set of experiments, we generated the global image representations via the FVs after extracting the dense features from the last convolutional layer for each CNN model. We combine the features computed by these three frameworks by using the same techniques (addition and concatenation). Finally, SVM is used for classification. Some tentative combinations using the three different CNNs are shown in It can be easily observed the combined features generated by the last convolutional layers improve performance with approximately 1–2% gains compared to the separate situations. So in both cases, the combination of features improves the accuracy performances even though the same structure was used in the three CNN models, which proves that differences in the CNN architecture allow one network to consistently perform better on a subset of the SAR images. In another words the gain would not be possible if the three CNN models classify the SAR images in the same way. Consequently, feature combination of the CNNs information improve the overall robustness and accuracy of the result compared to a single CNN.After classification of the three CNN models separately by their individual classifiers for the two sets of experiments, decision fusion algorithms mentioned in In this paper, we have investigated the SAR images classification task on the MSTAR dataset using the activations of different CNN models. In order to obtain strong global image representations for classification, various sets of experiments are proposed by directly extracting features from the FC layers and encoding dense features from convolutional layers. We evaluate our methods on the public MSTAR dataset and achieve remarkable overall classification accuracies of 98.60% for the ten-classification task. As expected, our method obtained remarkable results competitive with the state-of-the-art methods. Even though we do not apply any data augmentation strategies in our method in contrast with the methods that also explore CNNs, our method is only 0.4% worse than the best method that elaborately uses transfer learning from SAR images scenes from the CNN, and better than methods which use pre-trained CNN models. Furthermore, combining and fusing scenarios improves performance by about 2–3%. These impressive results demonstrate that not only the features of FC layers but also dense convolutional features of CNN are discriminative and powerful image representations. In future studies, we plan to investigate more sophisticated strategies for encoding the dense CNN features to improve the invariance of representations.


Internet of Thing (IoT) is the most emerging technology in which all the objects in the real world can use the Internet to communicate with each other as parts of a single unified system. This eventually leads to the development of many smart applications such as smart cities, smart homes, smart healthcare, smart transportation, etc. Due to the fact that the IoT devices have limited resources, the cybersecurity approaches that relied on complex and long processing cryptography are not a good fit for these constrained devices. Moreover, the current IoT systems experience critical security vulnerabilities that include identifying which devices were affected, what data or services were accessed or compromised, and which users were impacted. The cybersecurity challenge in IoT systems is to find a solution for handling the identity of the user, things/objects and devices in a secure manner. This paper proposes an effective multifactor authentication (CMA) solution based on robust combiners of the hash functions implemented in the IoT devices. The proposed CMA solution mitigates the authentication vulnerabilities of IoT and defends against several types of attacks. Also, it achieves multi-property robustness and preserves the collision-resistance, the pseudo-randomness, the message authentication code, and the one-wayness. It also ensures the integrity, authenticity and availability of sensed data for the legitimate IoT devices. The simulation results show that CMA outperforms the TOTP in term of the authentication failure rate. Moreover, the evaluation of CMA shows an acceptable QoS measurement in terms of computation time overhead, throughput, and packet loss ratio. The explosive growth of wireless network technology has led to scientific challenges, notably in terms of managing the communications between objects and infrastructures without human intervention. One of the most interesting wireless network technologies is the Internet of Things (IoT) which allows IP connectivity and data gathering for a network of devices without human interference. In general, IoT devices are extremely heterogeneous and differ in term of connectivity interfaces, battery, processing and memory capabilities, as well as in dimensions, costs, and hardware features . The emeThe IoT device based on 3G/4G/5G can receive the requested information from the sink or directly from the sensor network devices. Furthermore, the 5G base station is used to bridge the IoT network to another network or platform using software defined network (SDN) technology. The SDN offers a logical centralized and programmable method of IoT networks that resolve the weaknesses of traditional networks, such as troubleshooting and reconfiguration of connection for all devices in IoT, effective usage of network resources, reducing latency due to a distributed mechanism, etc. . The IoT cyberattacks mean an attempt to damage, disrupt, or gain unauthorized access to any IoT devices . Generally, cyberattacks aimed at accessing, changing, or destroying sensitive information; extorting money from users; or interrupting normal business processes which may outweigh the IoT benefits. Despite the architectural design of IoT, the network model of IoT is vulnerable to several types of cyberattacks at three layers . The IoT cyberattacks might include man-in-the-middle, denial of service (DoS), replay, link spoofing, forced delay, session hijacking, cross-site scripting, and SQL injection attacks ,8. The main challenges of an IoT network are the lack of cybersecurity standards that can countermeasure the aforementioned cyberattacks. Furthermore, the IoT based on a cryptosystem requires imperative computation overhead for resource-constrained devices because of the exponentiation operations to be executed in the encryption and decryption phases. Thus, IoT networks should implement lightweight authentication mechanism to prevent the unauthorized devices and to verify the access to the IoT services. An effective authentication solution determines if upon identification, the person or device is permitted to receive a service ,10. In fSince IoT devices have limited resources, the cybersecurity approaches that relied on complex and long processing cryptography are not a good fit for these constrained devices, especially when those approaches are applied to protect real-time applications . The cybThis paper reports the following contributions. Firstly, it proposes an effective multifactor authentication mechanism that uses robust combiners of fast hash functions. Each hash function is calculated based on preset key and the idea of one-time password (OTP). Also, the hash function is applied to both the key and the message. Secondly, it proposes a time-enhanced-based one-time password (TEOTP) hash function that is implemented in the base station/sink and IoT device to resolve the problem of time synchronization in time-based one-time password (TOTP). The time margin is caused by the clock skews, network latency and user delays. Therefore, the proposed TEOTP uses dynamic time synchronization based on round-trip time (RTT) to resolve the time margin problem. Finally, the proposed multifactor authentication mechanism has the capability to manage the direct access control of the IoT device to communicate with the sensor objects directly without intervention of the sink. It uses the concept of open authentication (OAuth) and secure the token credential session period, which will speed up the transfer of sensed data to the IoT device. Moreover, the proposed multifactor authentication (CMA) guarantees the integrity, authenticity and availability of sensed data for the legitimate IoT devices.The paper is structured as follows: Many researchers have studied the IoT cybersecurity from layer-level perspectives; however, some cyberattacks appear at most layers of the IoT network . The relCirani et al.  proposedThe limitations of previous literature studies ,27,28,29The proposed multifactor authentication algorithm presents three scenarios that cover the main important authentication cases in IoT network. The authentication scenarios that will be described in this section comprise of the communication between an IoT device and the base station (scenario1); the communication between an IoT devices and sensor device through the base station (scenario2); and the direct communication between IoT devices and sensor device (scenario3). Each sensor device has three secure keys—two privates (K1 and K2) and one public (K_DSA: key for direct access control)—which are stored during device programming.Each sensor device static or mobile is aware of its location.Sink is a trusted base station.A sensor device cannot use TOTP because it has limited resources which affect the precision calculation of the absolute time that is required in a synchronous TOTP.I, ID) and one public K_DSA.Each IoT device has two secure keys—one private  and the initial value of the counter number iC are uploaded into the sensor device during programming. A), the calculated authentication code (CAC) should be sent from the sink to sensor device A which will be used to verify the authorization of the sink. In order to validate the CAC at sensor device A, the first factor authentication is calculated based on a first hash function H1 which will use 1K and the last value of iC. In addition, the sensor device A will calculate the second factor authentication based on the second hash function H2 which will use the geolocation of sensor devices A and 2K. After that, the sender authentication code (SAC) is calculated based on combiners of both hash functions, and SALT random string as can be expressed in the following equation:L is the current location of sensor device A which will increase the strength of hashed code. Moreover, the SALT consists of a uniform random string which is used to defend against the password attack. Also, Equation (1) shows that the maximum output length of the combiner is 62 bytes. The SAC is compared with the CAC; if they are matched, then the sink is authorized. Otherwise the sink authentication is failed. Furthermore, the sensor device A will use H1 to hash the sensed data and send both the original message and the output of H1 to the sink. Upon receiving the authenticated message, the sink will recalculate the hash of receiving data. If the received hash code is matched with calculated code, the sink will accept the authenticated message. Otherwise, the sink will discard the received message. Moreover, the proposed micro IoT authentication is a mutual authentication which means if the sensor device A periodically sends the hashed sensed data to the legitimate sink, the sink will not accept the sensed data unless the calculated hash message code is matched with the received hash code.The micro IoT authentication paradigm is developed based on two authentication credentials which are “WHAT YOU KNOW: Private Key” and “WHERE YOU ARE: Geolocation” that verify the genuineness of the sensor nodes and the sink. We assume that each sensor node has two private keys and a counter number synchronized with the legitimate sink. The first private key is The macro IoT authentication paradigm is developed based on the multifactor authentication that used “WHAT YOU KNOW: Passwords” and “WHAT YOU HAVE: Token” authentication credentials to verify the genuineness of the IoT device and the sink. We assume that each legitimate IoT device has a login information  and a counter number synchronized with the legitimate base station. The main advantage of RTT is to regulate the interval durations which will resolve the delay variation due IoT network congestion or forced delay attacks. After that, SAC is calculated based on the output code of TEOTP which is sent to the base station and is compared with the CAC. If SAC and CAC are matched, then the IoT device is authenticated and the requested data is sent; otherwise the IoT device authentication operation fails. Like the micro IoT authentication algorithm, the macro authentication is a mutual authentication and it is applied to both the key and the message. This means if the base station/sink sends certain data to the legitimate IoT device, the IoT device will not accept these data, unless the calculated hash message code is matched with the received hashed message code.Ci in the micro IoT authentication algorithm. If the transmitted H_TIDS and the created H_TIDC by the sensor device are matched, then the IoT device is authenticated and the DSA session is started; otherwise the IoT device authentication operation fails. Like the micro and the macro IoT authentication algorithms, the micro-macro authentication is a mutual authentication and it is applied to both the key and the message. This means if the sensor device sends certain data to the legitimate IoT device, the IoT device will not accept these data, unless the H_TIDS and the received hash message code are matched with the H_TIDC and the calculated hash message code.The micro–macro IoT authentication paradigm is aimed to allow IoT devices to communicate with the sensor device without intervention of the base station or the sink. The open authentication (OAuth) concept has been used to develop the micro–macro IoT authentication algorithm. OAuth permits sensor devices to share sensed data with an IoT device without intervention of the base station. At the initial state, the base station uses the OAuth mechanism to forward the token credentials to both the IoT and sensor devices which will use the hashed token ID to verify each other. The basic cybersecurity problem in the IoT is that the attackers can gain access to the sensitive information that are restricted from obtaining by the third party. This section will explain how the proposed CMA mechanism can reduce the authentication vulnerabilities in IoT and how it defends against several types of attacks such as replay attack, link spoofing attack, man-in-the-middle attack, dictionary attack, brute force attack, sensor capture attack, stolen-verifier attack, and session hijacking attack. The CMA underlying hash functions achieves multi-property robustness (MPR) which means it should simultaneously preserve the collision-resistance (CR), the pseudo-randomness (PRF), the message authentication code (MAC), and one-wayness (OW). For multiple properties PROP = {CR, PRF, MAC, OW} one can either demand that the proposed combiner inherits the properties if one of the candidate hash functions is strong and has all the properties (weakly robust), or that for each property at least one of the three hash functions has the property (strongly robust). We denote by PROP(H) Theorem 1.The proposed combiner in Equations (1) and (2) is a strong MPR combiner for PROP = {CR, PRF, MAC, OW}.Proof. As we have mentioned that a strongly robust multi-property combiner should inherit all properties that are provided by at least one of the underlying hash functions. Thus, we have to prove that each property CR, PRF, MAC and OW is preserved individually. □Lemma 1.The proposed combiner in Equations (1) and (2) is CR-robust.Proof. K1, iC) and in TEOTP  in which the value of counter number iC and Time are distinct for all inputs. This is mainly because the value of iC and Time is continuously incremented after each authentication operation and the distinctness of the combiner of K1iC and TEOTP  is also preserved for all inputs. Hence, the expected time for the CR in the proposed combiner is n) where n is 62 bits in Equation (1) and 31 bits in Equation (2). □In CR property, it should be hard to find two distinct inputs that evaluate to the same hash value. The proof of CR property in CMA can be basically observed in H1 (Lemma 2.The proposed combiner in Equations (1) and (2) is PRF-robust.Proof. In PRF property, the combiner is called pseudorandom if no efficient adversary can distinguish the output of the combiner from a uniform random function with noticeable advantage. Since the proposed combiner xors the SALT (a uniform random function) with the concatenation of  in Equation (1) and with TEOTP in Equation (2), the final output of proposed combiner preserves the PRF property. Hence, the final output is indistinguishable from the uniform random and is PRF robust. □Lemma 3.The proposed combiner in Equations (1) and (2) is MAC-robust.Proof. The CMA used a MAC concept in the three proposed algorithms which allows a sender and receiver, both sharing a secret, to exchange information in an authenticated manner. A MAC is considered secure if it is unforgeable under chosen message attacks, i.e., an adversary after adaptively learning several tags , ,…,  should not be able to compute a forgery for a fresh message M. In CMA, MAC is calculated based on xoring of a secret K1 with the data message (M). Furthermore, the strong condition in MAC calculation is that the SAC should be verified between the sender and the receiver before evaluating the MAC. Therefore, even if the adversary creates a forgery message M, it will not be accepted because the verification of SAC is not known by the adversary. □Lemma 4.The proposed combiner in Equations (1) and (2) is OW-robust.Proof. 31, it is hard to invert the H1 and H2 for longer input length then 231. Moreover, the input length of iC and Time is continuously incremented, and the SALT also increases the input length of the proposed combiner to reach the maximum value with 262. Therefore, the probability for the adversary to determine the preimage of the combiner output is 1/262 which means the OW is preserved in CMA. □In OW-robust, the proposed combiner intuitively requires that it is infeasible to determine the preimage of a hash value. Since H1 and H2 in Equations (1) and (2) use the modulo operation with 2The proposed cybersecurity mechanism ensures three security requirements which are authenticity, integrity, and an availability of sensed data. The authenticity of the legitimate sink and IoT devices is achieved using the multifactor authentication which does not allow the intruders to gain access to the IoT network. Moreover, the SALT which consists of a random string is used to defend against the password attack. In addition, the proposed cybersecurity mechanism provides seamless integrity for the sensed data that is transferred between the IoT devices. This is primarily due to the proposed hash function is applied to both the authentication code and the message which can detect the alteration of sensed data when it is transferring between IoT devices. Also, it guarantees the availability of sensed data to the authorized IoT devices. This is mainly due to the proposed cybersecurity mechanism does not allow the intruders to associate with the IoT network and make flooding/congestion attacks.The multifactor credentials should be inspected between a man-in-the-middle and the sender before accepting any data message.The reply message should be hashed using the secret key which is not included in the original message and only known by the sender and the base station.The replay attacks cannot forward the copy of sending messages, because the proposed CMA uses TEOTP which changes after a set time period (30 s + RTT).A man-in-the-middle attack intercepts the legitimate communication between IoT devices, and it forges a fictitious response to the sender. In an active man-in-the-middle attack, the contents are intercepted and altered before they are sent on to the recipient . MoreoveA dictionary attack is a password attack that creates encrypted versions of common dictionary words and compares them against those in a stolen password file. Moreover, a brute force attack is a password attack in which every possible combination of letters, numbers, and characters is used to create encrypted passwords that are matched against those in a stolen password file . HoweverA spoofing attack is impersonating another IoT device in which an intruder spoofs the network address of the target IoT device so that their malicious actions will be attributed to valid IoT devices. Moreover, a session hijacking is an attack in which an intruder attempts to impersonate the IoT device by using its session token identity. However, the proposed CMA mechanism can defend in advance against these attacks by using the multifactor authentication, which uses the combination of geolocation information, OAuth hashed token ID, and TEOTP credential code. Moreover, the limited time of TEOTP credential code will make the session identity code periodically change. Even if the intruder still gains the session ID, he/she cannot gain access to the IoT network because he/she needs to calculate the multifactor of credential code for every sending or receiving message.Ci and Time. Moreover, the embedded hash functions in the execution file of the source code will prevent the node capture attack to gain access to the IoT network because those functions are uploaded into the sensor device using machine language code. Furthermore, the requirement of geolocation information which depends on the actual physical location of sensor devices will restrict the effect of the captured sensor node. Also, the OAuth mechanism can prevent the stolen-verifier attack because the one-way hash function is implemented at the trust base station and the intruder cannot verifies the hashed token ID.A sensor node capture attack means that an intruder captures a sensor node, steals all the information stored (keys and measured data) and uses the compromised keys to perform various operations on the IoT network. Moreover, a stolen-verifier attack means that the intruder who has stolen the verifier data for the sensor/IoT device can impersonate a legal device from the next authentication session. However, the proposed CMA mechanism defends against those attacks using the combiners of hash functions that used the one-time synchronize random input based on To demonstrate the CMA functionality, several experiments have been conducted using Mininet-IoT emulation software . A small2). The mobility model of IoT devices and intruders is established using random movement on the straight line. Also, the traffic load is measured using the number of packets that are sent per second (pkt/s). The emulation time has been set to 1000 s to give the intruders the enough time to implement dictionary and brute force attacks. The velocity of IoT device was randomly varied between 5 m/s and 15 m/s. Furthermore, the key length size  is 32 bits and the uniform random function that has been used in the sensor device is simulated based on [based on .In this section, the performance of using the three proposed authentication algorithms—micro, macro, and micro–macro paradigm authentication—has been analyzed in terms of latency, throughput, and packet loss ratio over IoT network. Moreover, the comparison between the proposed TEOTP and TOTP algorithms has been investigated. The throughput can be defined as the amount of data transferred successfully to the destination in a given period. The packet loss ratio is defined as the ratio of the total number of lost packets due to using CMA to the total number of sending packets. The latency is the period between the transmission and the reception of the packet when CMA is applied to an IoT network.In this experiment, the effect of using CMA authentication algorithms was evaluated in terms of latency, throughput, and packet loss ratio over IoT network. The Iperf of IPv6 standard tool was used to measure the performance of conducting TCP data traffic tests. In order to create data streams to measure the performance between IoT and sensor devices, the Iperf client function was run in the sensor3 and the Iperf server function was run in BaseST1 for micro authentication experiment. Moreover, the Iperf server function was run in IoTDev5 for macro and micro–macro authentications experiment. DiscussionThe above results show the performance of the three authentication mechanisms which are mainly achieved due to the following reasons. Firstly, macro and micro–macro authentication mechanisms use embedded TOTP with RTT modification in an authentication decision which improves the cybersecurity, but it decreases the throughput of IoT network. Secondly, the long processing delay of macro and micro–macro authentication increases the latency of authentication. This is primarily due to the two types of multifactor authentication mechanisms that are implemented between the sensors and the IoTDev5. The first type of multifactor authentication is implemented between the sensors and the sink and the second type of multifactor authentication is implemented between the sink and the IoTDev5. Finally, the packet loss ratio is higher at the beginning of macro and micro–macro authentication mechanisms because of the dropping packets at the verification of data credentials that involve in the combiners of hash function. Overall, the cost of developing an unbreakable cybersecurity mechanism in CMA is acceptable and it does not affect the QoS of an IoT network in terms of packet loss ratio, latency, and throughput.In this experiment, the ICMP and Iperf tool was used to measure the authentication failure rate and computation time overhead based on UDP data traffic tests. In order to emulate the TOTP in Mininet-IoT, the library source code of PyOTP  was instDiscussionThe above results show that the TEOTP outperforms the TOTP in terms of the authentication failure rate. This achievement is primarily due to the fact that the TOTP authentication mechanism does not handle the network latency between the source and the destination of IoT devices. Moreover, the computation time overhead is higher in TEOTP because of the RTT overhead that is essential to measure the latency between the authenticated parties of IoT devices. More importantly, the RTT involved in TEOTP calculation is used to decrease the authentication failure ratio due to the time synchronization between IoT devices.Although the CMA was carefully designed for IoT, there were some of unavoidable hardware limitations. Firstly, the CMA cannot be applied to the available IoT network devices due to the limitation of existing hardware and software which means a new generation of IoT network devices should be developed. Secondly, IoT networks scalability and energy consumption need more investigation to cope with the emergence of IoT hardware specifications which yet well-defined in the main standard document of IoT technology. Finally, the proposed CMA software needs advance programming to integrate all authentication mechanisms of CMA together in a sensor/IoT devices.This paper presents CMA that uses robust combiners of fast hash functions to achieve MPR and preserve CR, PRF, MAC, and OW. Also, it proposes a TEOTP hash function implemented in the base station/sink, and IoT device to resolve the problem of time synchronization in TOTP. The simulation results show that CMA outperforms the TOTP in term of the authentication failure rate and it ensures the integrity, authenticity and availability of sensed data for the legitimate IoT devices. Moreover, the evaluation of CMA shows an acceptable QoS measurement in terms of computation time overhead, throughput, and packet loss ratio. The future work of this research will focus on developing cryptography mechanism based on encryption and decryption for an IoT network, which will guarantee the confidentiality of the sensed data on IoT networks.


The rational function model (RFM) is widely used in the most advanced Earth observation satellites, replacing the rigorous imaging model. The RFM method achieves the desired calibration performance when image distortion is caused by long-period errors. However, the calibration performance of the RFM method deteriorates when short-period errors—such as attitude jitter error—are present, and the insufficient and uneven ground control points (GCPs) can also lower the calibration precision of the RFM method. Hence, this paper proposes a geometric calibration method using sparse recovery to remove the linear array push-broom sensor bias. The most important issue regarding this method is that the errors related to the imaging process are approximated to the equivalent bias angles. By using the sparse recovery method, the number and distribution of GCPs needed are greatly reduced. Meanwhile, the proposed method effectively removes short-period errors by recognizing periodic wavy patterns in the first step of the process. The image data from Earth Observing 1 (EO-1) and the Advanced Land Observing Satellite (ALOS) are used as experimental data for the verification of the calibration performance of the proposed method. The experimental results indicate that the proposed method is effective for the sensor calibration of both satellites. With the accelerating development of technical aeronautics and space exploration, many countries have launched advanced Earth observation satellites in recent years. The linear array push-broom sensor is the payload which has been carried by the most satellites. This significant quantity of Earth observation has had a great impact on overall production and human life. However, the imaging process of sensors is easily influenced by a variety of errors, including thermal deformation error, optical distortion error, satellite position, attitude measurement errors, and so on ,2. CommoThe imaging process involves projecting a point on the surface of the Earth toward the sensor’s focal plane . The imaIn order to solve the above-mentioned problems, this paper puts forward a geometric calibration method by using sparse recovery to remove linear array push-broom sensor bias. The errors relating to the imaging process are approximated to the equivalent bias angles , and theOver the past decade, the linear array push-broom sensor has been used as a main payload assembled by the most advanced Earth observation satellites. The imaging process of the linear array push-broom sensor projects a point on the surface of Earth, such as a GCP, to the sensor’s focal plane . The basCompared with the true values, the imaging parameters usually solve some undistinguished errors in the imaging process. The long-period errors, such as thermal distortion error, optical distortion error, assembling error, and the short-period errors—such as satellite orbit error and attitude error—of the imaging process are approximated to the equivalent bias angles. The sensor geometric calibration model is presented as ,2,10(2)All on-orbit satellites suffer from an environment that varies cyclically between cooling and heating due to the circular orbit around Earth. The thermal distortion error resulting from the satellite in the Earth’s orbit causing a solar eclipse is the biggest error in the imaging process . The perAs a new theory in signal processing, compressive sensing was proposed in 2004 and rapidly developed in the following years. The core idea of this theory mainly involves two aspects. The first is the sparsity of the signal, which means that the majority of the elements are have a value of zero or are otherwise very small . The othThe geometric calibration method consists of five primary steps: (a) periodic wavy pattern recognition; (b) sparse GCP image shift calculation; (c) dense GCP image shift calculation; (d) equivalent bias angles recovery; and (e) image calibration. The procedure of the proposed method is shown in The first step is to recognize the periodic wavy pattern of the raw image sequence frame-by-frame. Once the raw image exists with periodic wavy patterns, which are caused by short-period error, such as attitude jitter error, the subsequent process of Step 3 is necessary for the frame. There are three processes that need to be completed for periodic wavy pattern recognition. First, a line or edge detection algorithm is used to extract the road, dike, and coastline from the raw images. Then, the smoothness of the extracted lines or edges is extracted. Finally, the existence of periodic wavy patterns in the images is determined. The details of this step can be seen in .ith GCP.There are three main processes in this step. First, a handful of images with good imaging conditions are randomly selected from the raw image sequence, excluding images with periodic wavy patterns, as the measurement scenes. Then, a small number of GCPs with high precision from each measurement scene are extracted and matched. The sampling interval of GCPs is set to larger, and these GCPs are defined as sparse GCPs in this paper. The latitude and longitude ranges of each measurement scene are calculated to determine whether there are GCPs in each measurement scene; the templates of GCPs used in this GCPs extraction process are obtained from the Shuttle Radar Topography Mission data. The details of this process are seen in . FinallyThe raw images with periodic wavy patterns are selected as the measurement scenes, and a number of GCPs with intensive distribution from each measurement scene were extracted and matched with periodic wavy patterns. The sampling interval of the GCPs is set to smaller; these GCPs are defined as dense GCPs. The exact number of GCPs is determined by the sparsity and length of the recovered signal, the recovery algorithm, and so on. Generally speaking, the length of the measurements is far below the length of the recovered signal . The resThe most important thing in this crucial step is the signal fusion. It is known that the compressive sensing method can only recover 1-D column vectors, therefore, 3-D equivalent bias angles signals must be merged into a 1-D time varying signal according to the time order. The problem regarding solving 3-D signal errors is transformed into the problem relating to 1-D signal recovery. After obtaining the image shift of GCPs, the remaining tasks of this step are to select a desired sparse basis and an appropriate recovery algorithm and to construct a suitable measurement matrix to exactly recover the 1-D time varying signal. The 3-D signals are separated from the 1-D recovered signal. The details of this important work are presented in the following sections.There are two processes in this step. One involves calculating the calibrated position, and the other is to do with resampling the raw image. Once the equivalent bias angles are recovered, the calibrated image positions are easily calculated using Equation (2) with the imaging parameters. The image resampling can be completed using the bilinear interpolation method. The gray values of the calibrated image are obtained according to the calibrated image positions and the raw image.If the equivalent bias angles are considered to be errors in an ideal situation, the estimation of equivalent bias angles is a typical adjustment of indirect observations. The function model of the adjustment of these indirect observations is expressed asGenerally, the estimated parameter is expressed asLetAccording to Equations (6) and (7), Equation (8) can be simplified as(a) Measurement variablesAccording to Equation (8), According to Equation (5), (b) Measurement equationAccording to Equation (9), the measurement equation of one GCP is expressed asThe measurement equation of M GCP is expressed as(c) Linearization of measurement equationThe upper limit of the equivalent bias angles The first parameter The image positions of the mth GCP by using the imaging parameters and the equivalent bias angles After obtaining the The other important issue regards obtaining the element of the measurement matrix The equivalent bias angles signals based on Equation (4) are not sparse in the time domain. According to previous studies ,17, the To easily recover the equivalent bias angles signals, the 3-D equivalent bias angle signals need to be merged into a 1-D signal according to time order. The 1-D merged signal and the results of the 1-D DFT are displayed in From It is obvious from Equation (12) that the 2-D measurement equation used to solve the 3-D equivalent bias angles of one GCP is a pathological problem. The traditional methods assume that the equivalent bias angles are unchanged in the frame period and use classical optimal estimation methods—such as least squares estimation, Bayes estimation, or maximum likelihood estimation—to combine a number of GCPs to estimate the equivalent bias angles of each frame. However, this assumption is unreasonable when equivalent bias angles contain short-period errors. Due to The measurement equation of M GCP in Equation 13) is rewritten as is rewriL1=Bx^=BΨmplexity ,19. The Two experiments were used to compare the proposed method with the RFM method in this section. The first method employs the image data from the Hyperion of Earth Observing 1 (EO-1) and shows the performances of two methods designed to estimate the long-period components of equivalent bias angles. The second method, which uses the image data from the Panchromatic Remote-sensing Instrument for Stereo Mapping (PRISM) of the Advanced Land Observing Satellite (ALOS), shows that the proposed method has a superior performance regarding the estimation of the short-period components. The details of the experiments are described as follows.Hyperion, as the main payload of EO-1, is a typical linear array push-broom sensor. A total of 100 Hyperion frame images from 7 June 2002 were selected as the experimental data. In the proposed method, we selected 40 frames with some evenly distributed GCPs from the experimental data as the measurement scenes. The measurement scene used in the proposed method is represented in The calibration results of both methods for one frame showing 30 GCPs are displayed in To make accurate comparisons between the two methods, the calibrated residuals of the RCPs were employed in the statistical analysis. The mean value and root mean square error (RMSE) of the calibrated residuals were used as the measurable indicators. The variation trends of the mean value and the RMSE for the total GCPs across the two methods are shown in PRISM, the primary sensor carried by ALOS, was designed to obtain a high-resolution digital surface model to derive orthoimages on a global scale. PRISM has three panchromatic linear array push-broom sensors  with a 2.5 m spatial resolution and a width of 35 km swaths. The raw image data of PRISM was observed on 20 August 2006 and covered 50 scenes; these were used as the experimental data. To verify the performances of the two methods regarding the estimation of short-period components, one scene from the experimental data was used as the typical scene, as shown in We used the line or edge detection algorithm to extract the road in a typical scene; the detection results are shown in The calibration results of the two methods for 50 RCPs in typical scene are shown in Geometric calibration must be carried out before the application of raw images. This paper proposed a geometric calibration method using sparse recovery to remove linear array push-broom sensor bias. The errors in the imaging process were approximated to the equivalent bias angles in this method. By using the sparse recovery method, the proposed method exactly estimated long-period errors with a small number of GCPs available. Also, the proposed method effectively removed short-period errors by recognizing periodic wavy patterns in advance. The preliminary experimental results indicated the practicality and superior calibration performance of the proposed method when used for image data captured by the EO-1 and ALOS satellites. Compared with the traditional methods, the proposed method did well in situations with insufficient GCPs and short-period error calibration.Future research will focus on the effects of GCP distribution on the proposed method. It is also important to apply the proposed method to other types of sensors.