Grid theory is rather commonly-used through out the research of integer ambiguity. In order to promote the efficiency of computation, it is of great necessity to reduce the correlations of the grid basis through the reduction. The classical reduction algorithm is known as the LLL (Lenstra–Lenstra–Lovász) algorithm. So as to further enhance the reduction effect, the deep-insertion LLL algorithm can be utilized as an alternative to the basis vector exchange algorithm. In practice, the deep-insertion LLL algorithm can achieve a better reduction effect, but it requires more time for reduction. The PotLLL algorithm replaces the basis vector exchange condition of deep-insertion LLL with an improving in the basis quality, and it can run in polynomial time, but with certain limitations. Therefore, this article proposes a global deep-insertion PLLL algorithm (GS-PLLL) to address the issue of integer ambiguity. GS-PLLL adopts a global strategy for deep-insertion processing, and introduces a rotation sorting method for preconditioning the grid basis. Comparative evaluations were conducted using simulation experiments and real-world measurements on the LLL, DeepLLL, PotLLL, and GS-PLLL algorithms. The experimental results indicate that the GS-PLLL algorithm achieves a better reduction effect than the PotLLL algorithm while improving the efficiency of reduction. The high-precision positioning equation of the Global Navigation Satellite System (GNSS) can be concluded as the problem of minimizing the following equation subject to constraints.In the equation, The core of the Eq.  lies in mentclass2pt{minimmentclass2pt{minim1 is the most representative. The LAMBDA algorithm first proposed the concept of decorrelation, which reduces the correlation between vectors and improves the efficiency of ambiguity search by using unitary matrices to process the covariance matrix. Subsequently, many scholars conducted extensive research on the ambiguity and correlation issues and proposed numerous practical solutions. Chang2 improved and applied the sorting-based QR decomposition strategy to the LAMBDA algorithm, proposing the MLAMBDA algorithm, which improves the decorrelation and search process of the LAMBDA algorithm. Xu3 used the Cholesky decomposition to construct an integer Gaussian matrix and proposed an inverse integer Cholesky decorrelation method based on the pre-sorting strategy. The inverse integer Cholesky decorrelation method outperforms the integer Gaussian decorrelation and LLL algorithms, thus indicating that integer Gaussian decorrelation is not the best decorrelation technique and can be improved further. Chen et al.4 used the floating-point transformation matrix to completely decorrelate non-diagonal elements and proposed the Zero-correlation algorithm, experiments have demonstrated that the method can improve the integer resolution efficiency without decreasing the accuracy.With the continuous development of GNSS technology, solving the problem of integer ambiguity [Eq. ] has bec5. Therefore, the problem of integer ambiguity resolution can be approached from the perspective of lattice theory. A Cholesky decomposition is performed on the covariance matrix in the ambiguity solution process in Eq.  in lattice theorys in Eq. :3\documeSince the matrix entclass1pt{minimaIn the equation, Further equivalence of Eq.  is achie6. Korkine and Zolotareff7 extended the Gauss reduction algorithm to any dimensionality and proposed the KZ reduction algorithm to solve the shortest vector problem in lattices. Lenstra et al.8 proposed the LLL reduction algorithm, which can solve the shortest vector problem in polynomial time. Hassibi et al.9 first used the LLL reduction algorithm in the field of GNSS integer ambiguity resolution, it is shown that the integer least squares problem associated with parameter estimation can be solved efficiently in practice. Grafarend et al.10 provided a detailed analysis of the principles of the LLL reduction algorithm for solving integer ambiguities in GNSS. Liu et al.11 proposed an improved algorithm based on matrix-wide rounding to address the rounding error in LLL. Fan12 used a block Gram-Schmidt orthogonalization algorithm to improve the average correlation coefficient and condition number and thus enhance the efficiency of the LLL algorithm. Lu13 used block dimensionality reduction to improve the LLL algorithm and effectively increase its operational efficiency. Xie et al.14 improved the LLL reduction algorithm by using Householder orthogonal transformation based on system rotation to reduce the basis vector length. Through experimental analysis, Householder LLL (HLLL) algorithm can significantly improve the effect of correlation reduction processing. However, the algorithm still has the shortcoming of long statute time, which needs to be further improved. Recently Li et al.15 made further advancements to the HLLL by using column selection to obtain higher reduction efficiency and superior reduction performance. Schnorr proposed the DeepLLL reduction algorithm based on the idea of deep insertion16. Although DeepLLL has better output quality than LLL in practical applications, the complexity of DeepLLL may be super-exponential. Fontein et al.17 proposed a new variant of DeepLLL called PotLLL, which has a provable polynomial runtime by replacing the basis vector exchange condition of DeepLLL with an improvement in the quality of the basis matrix.Equation  represenHowever, both DeepLLL and its variant PotLLL only operate on the sublattice The rest of this paper consists of the following sections. “i.ii.The LLL algorithm is a reduction algorithm with groundbreaking significance in the field of lattice basis reduction proposed jointly by three scholars, Lenstra, Lenstra, and Lovasz. The basic idea of this algorithm is to gradually map the basis vectors to a set of more independent vectors that are closer to the standard orthogonal basis through a series of basis transformations . Perform Schmidt orthogonalization on the upper triangular matrix B in (3):The lattice basis When converting between different bases of the same lattice, it can be done using unimodular matrices. If the orthogonal transformation matrix Lattice basis reduction requires that all elements of the unimodular matrix must be integers and the absolute value of the determinant must be 1. To meet these requirements, rounding must be used to standardize the orthogonalization coefficients. Under the constraint of scaling reduction, complete integer orthogonal transformations can be achieved between vectors, resulting in a good lattice basis reduction effect. The effect of lattice basis reduction depends on the magnitude of the orthogonalization coefficients, which are related to the dot product of the two vectors and the lengths of the orthogonalized vectors. The coefficient for orthogonalizing a subsequent vector is inversely proportional to the length of the orthogonalized vector corresponding to the previous vector. If the longer orthogonalized vector is arranged first, the coefficient for computing the subsequent vector will be smaller, and may even become 0 after rounding. At this point, it is considered that the angle between the two vectors is greater than 60 degrees, and good orthogonalization has been achieved. According to the cosine formula for the angle between two vectors, the cosine of the angle between vectors depends on the lengths of the two vectors. The calculation of the orthogonalization coefficient replaces two vectors with one vector. If the length of this vector is large, the calculated cosine value will be small and cannot reflect the true angle between the two vectors. Therefore, while constraining the orthogonalization coefficients, constraints must also be placed on the lengths of the vectors.Due to the restrictions of unimodular matrices, it is difficult to achieve the goal of LLL-reduced basis. The LLL algorithm analyses the projection vector lengths of two adjacent vectors in the subspace spanned by the earlier basis vectors to ensure that the length of the first basis vector is the shortest, that is, the basis vector exchange condition.To summarize, the process of LLL reduction transformation based on the matrix B and O obtained from the Schmidt orthogonal transformation of the lattice basis is as follows:The orthogonalization coefficient reflects the angle between two vectors. The larger the value, the smaller the angle. Therefore, LLL reduction performs a magnitude reduction through an integer transformation, which makes the orthogonalization coefficient of each vector satisfy the scaling reduction condition. For the basis vector This process aims to ensure that the length of the orthogonalized vectors meets the length reduction condition. It is implemented simultaneously with the magnitude reduction process by changing the order of the vectors According to the exchange condition of basis vectors in the LLL algorithm, only the adjacent basis vectors For a lattice basis The advantage of using the potential of a lattice basis is that it takes into account not only the number of vectors in the lattice basis, but also the influence of the arrangement order of the basis vectors. The contribution of earlier basis vectors to the value of Pot(B) is significantly greater than that of later basis vectors. Since PotLLL involves the order relationship between basis vectors, the following concepts related to basis vector sorting are introduced.Let Among them, 18, or a new evaluation criterion can be used to improve the basis exchange condition and enhance the quality of the reduced basis. The quality of the reduced basis is closely related to the decoding time of the ambiguity, and a higher quality reduced basis means a higher search efficiency. In this paper, a GS-PLLL lattice basis reduction algorithm is proposed based on matrix pre-sorting and global deep-insertion exchange, aiming to improve both reduction efficiency and the quality of the reduced basis.The basis exchange condition in the DeepLLL algorithm can help the lattice basis to meet the ascending order requirement to some extent. However, the deep-insert exchange condition limits the exchange to be only within the current sublattice, which may result in too many exchanges and negatively affect the reduction efficiency and quality of the reduced basis. To solve this problem, a pre-sorting strategy can be adopted to reduce the number of basis exchanges19 based on Gaussian decomposition. Its main idea is to rotate the smallest conditionally variance element through the sorting matrix to the first element position in the matrix during each sorting. The rotation sorting method based on lattice requires improvement from the perspective of lattice basis reduction. Assuming the first rotation sorting matrix is The rotation sorting method was initially proposed by Xu et al.LLL, DeepLLL, and PotLLL all use linearly increasing or decreasing basis vector indices for k in the range of In addition to processing the entire lattice basis during basis vector exchange, a greedy strategy is also used to select an index In summary, the algorithmic flow of GS-PLLL, combined with rotation sorting and greedy deep insertion, is shown in Fig. To evaluate the performance of the GS-PLLL reduction algorithm and compare it with the LLL, DeepLLL, and PotLLL reduction algorithms, both simulated data and actual measurement data were used. The simulated data provides a general representation unaffected by external factors, allowing for a better illustration of the advantages of the three lattice basis reduction algorithms. The simulation experiments verified the theoretical advantages of the GS-PLLL algorithm, while the actual measurement data further validated its effectiveness. All experiments were conducted on a computer platform with Windows 10 operating system, using an Intel® Core™ i7-8750H CPU @ 2.20Hz with 16GB memory.20. In order to measure the performance of lattice basis reduction, the Hadamard ratio is often used as an evaluation index21. The Hadamard ratio is used to describe the degree of orthogonality of a set of vectors, with a range of The goal of lattice basis reduction is to transform the basis vectors of a lattice to make them shorter and more orthogonal, thus simplifying computations and improving efficiencyIn addition, the number of basis vector exchanges directly affects the efficiency of basis reduction algorithms. Therefore, the superiority of the GS-PLLL reduction algorithm is explained by comparing the number of basis vector exchanges, computational efficiency, and the Hadamard ratio.2 was used, with the covariance matrix of the integer ambiguity constructed using a random simulation method. In order to prove the universality of the algorithm, the simulated random matrix had dimensions ranging from 5 to 40, covering the low- to high-dimensional cases of the covariance matrix. Additionally, in order to avoid accidental situations, each experiment was conducted with 100 repetitions to obtain the mean value.In Experiment 1, the random matrix construction method proposed by Chang et al.Figure For a more visual comparison, Table 17, which indicated a higher improvement rate for the PotLLL algorithm when compared to DeepLLL and LLL techniques.The table clearly demonstrates that the GS-PLLL algorithm achieves the highest restoration quality in terms of the Hadamard ratio. Combining the findings from Table Overall, the results presented in Table 22. For a lattice basis reduction algorithm, the fewer the number of base vector exchanges, the higher the reduction efficiency. Therefore, Fig. In lattice basis reduction algorithms, scaling reduction and basis vector exchange directly impact the efficiency of ambiguity resolution. Among them, the impact of scaling reduction on efficiency improvement is limited, whereas basis vector exchange directly affects the resolution efficiencyOverall, the results in Fig. To further highlight the advantages of the GS-PLLL algorithm, a comparison of the computation times of the four algorithms is presented in Fig. Based on the comparison data from Fig. Furthermore, it can be observed that in the case of 25 dimensions, the computation time of all four algorithms significantly increased. The reason for this phenomenon is that the low correlation of the original 25-dimensional matrix led to a substantial increase in the number of basis vector exchanges. Combining the conclusions from Fig. To further validate the performance of the GS-PLLL reduction algorithm in real-world scenarios, after verifying the theoretical advantages of GS-PLLL in experiment1, actual measured data was used for further validation. The experimental procedure involved the use of actual measurement data collected by Huace P5 satellite differential receivers. The base station and mobile station observations were recorded in the RINEX 3.02 format, with an observable elevation angle cutoff of 5° and a sampling rate of 1 Hz. Precise ephemeris information was obtained from the Satellite Navigation and Positioning Research Center of Wuhan University. The L1 frequency signals from GPS satellites are used in this experiment for double-difference modelling to compute the ambiguity floating-point solutions of the double-difference observations and their corresponding covariance matrices. Like simulation experiments, actual measurement experiments also discuss three aspects: Hadamard ratios, the number of base vector exchanges, and computational efficiency.Figure As illustrated in Fig. Figure In this paper, a new reduction algorithm, GS-PLLL, is proposed to solve the integer ambiguity problem in high-dimensional GNSS positioning. The algorithm incorporates global depth insertion and rotation sorting strategies. To validate the feasibility of GS-PLLL and evaluate its performance, we compare GS-PLLL with other algorithms  using simulated and real-world measurement data.The evaluation was based on various metrics, including the Hadamard ratio, the number of basis vector exchanges, and computation times. The experimental findings consistently demonstrated the superiority of the GS-PLLL algorithm. Compared to the other algorithms, it exhibited a shorter reduction time and achieved a higher Hadamard ratio, indicating its low computational complexity and ability to generate better reduction basis. Besides, the statistical analysis of computation times revealed that the GS-PLLL algorithm outperformed the other algorithms in speed and stability. Accordingly, it is concluded that the GS-PLLL algorithm is an effective and practical approach for fast integer ambiguity resolution in high-dimensional GNSS positioning.25 to improve the computational efficiency of lattice-based reduction is also a direction for future research.In addition, how to borrow the idea of heuristics algorithmsSupplementary Information.


With the development of smart agriculture, deep learning is playing an increasingly important role in crop disease recognition. The existing crop disease recognition models are mainly based on convolutional neural networks (CNN). Although traditional CNN models have excellent performance in modeling local relationships, it is difficult to extract global features. This study combines the advantages of CNN in extracting local disease information and vision transformer in obtaining global receptive fields to design a hybrid model called MSCVT. The model incorporates the multiscale self-attention module, which combines multiscale convolution and self-attention mechanisms and enables the fusion of local and global features at both the shallow and deep levels of the model. In addition, the model uses the inverted residual block to replace normal convolution to maintain a low number of parameters. To verify the validity and adaptability of MSCVT in the crop disease dataset, experiments were conducted in the PlantVillage dataset and the Apple Leaf Pathology dataset, and obtained results with recognition accuracies of 99.86% and 97.50%, respectively. In comparison with other CNN models, the proposed model achieved advanced performance in both cases. The experimental results show that MSCVT can obtain high recognition accuracy in crop disease recognition and shows excellent adaptability in multidisease recognition and small-scale disease recognition. Diseases have always been an important disadvantage in agricultural production. Early detection and removal of diseases are effective means to improve the quality of crop growth. In traditional agricultural production, the identification of crop diseases is mainly performed through manual analysis combined with professional knowledge. The manual identification of diseases is inefficient, is error-prone, requires high expertise, and is costly in terms of human resources . TherefoIn recent years, with the development of deep learning , deep leSome researchers have also applied CNN models to agricultural disease recognition with good results. Mohanty et al. adapted the pretrained models AlexNet and GoogleNet, using transfer learning, based on the PlantVillage dataset containing 14 crops and 26 diseases. Both models achieved more than 99% accuracy on the testing set, demonstrating the effectiveness of deep learning models in intelligently diagnosing crop diseases . SuryawaTo solve the problem that heavyweight models are difficult to be applied to agricultural production, some researchers have improved some existing lightweight models or designed their model structures with simple structures and a small number of parameters according to the characteristics of crop disease datasets and obtained good recognition performance. Kamal et al. proposed a lightweight model, Reduced MobileNet, with a deep separable convolutional structure for leaf disease recognition . They trCNN-based deep learning models have achieved excellent results in a crop disease identification work, thanks in large part to their ability to extract local information efficiently. In many cases, however, obtaining larger sensory fields can often only be achieved by using large convolutional kernels or stacking multiple small convolutional kernels in succession . This chPrevious disease identification models are still dominated by CNNs, which have limitations in the expansion of the receptive field and the perception of global information. Our work integrates the features of CNN and Transformer and designs a multiscale self-attention (MSSA) module for the information fusion of local and global features of disease images.For crop disease identification, the self-attention mechanism lacks practical applications to verify its effectiveness. It is still difficult to design lightweight and high-performance hybrid ViT models with real-world application datasets. We propose a hybrid model fusing multiscale convolution and vision transformer (MSCVT) for crop disease identification based on the ResNet five-stage architecture and insert MSSA modules in each stage. In addition, we validated the effectiveness of the proposed model on two disease datasets with different scales.To ensure that the proposed model is lightweight, we introduce the inverted residual block to replace the ordinary convolution of All of these models achieved performance beyond that of traditional CNN models on publicly available large image datasets. However, there is still a lack of relevant studies combining a self-attention mechanism to design models with high recognition accuracy on smaller-scale agricultural disease datasets. The self-attention mechanism brings new possibilities for a deep learning model design in agriculture. Based on the current research status of CNN models in the field of agricultural disease identification and the advantageous performance of ViT models in constructing long-range feature relationships, we summarize the limitations of previous studies and the main contributions of our work from the following points:As shown in Our model uses the average pooling layer instead of the fully connected layer. In earlier networks, such as AlexNet, VGG, etc., the fully connected layer takes up nearly 80% of the total number of parameters in the network, which makes the model itself very bulky. An average pooling layer accomplishes the role of dimensionality reduction by averaging each feature map, and also significantly reduces the number of parameters . FinallyThe inverted residual (IR) block is a lightweight network structure proposed by MobileNetV2, which is based on MobileNetV1’s deep separable convolution (DSC) improvement.The advantage of DSC is the ability to obtain good information aggregation while maintaining a low number of parameters and computational effort . DSC divThe number of parameters of the ordinary convolution is A × S × S × B. It can be obtained that DSC converts some of the multiplication operations in ordinary convolution into addition operations, thus greatly reducing the computational complexity. In addition, the outputs of both DC and PC are input to the batch normalization and ReLu activation functions to improve the convergence speed during training and enhance the nonlinear representation of the model. The structure of DSC is shown in The residual block has been proven in ResNet to help improve the accuracy to build deeper networks, so MobileNetV2 also introduces a similar block. The classical process of the residual block is to downscale by The self-attention mechanism first appeared in Transformer, proposed by Ashish Vaswani et al. It is based on the idea of obtaining long-distance feature relationships by computing the similarity between words in a sentence, and has been widely used in text classification  and sentto words . SimilarAs shown in alue (V) . Then, tIn order to obtain complete and detailed global information, the size of the sliced patch is set to An MSSA module consists of a multiscale feature fusion layer and self-attention module (SA module), and the overall architecture is shown in ormation . The aboThe self-attention mechanism is introduced to work on the extraction of global information. The traditional CNN model obtains a wider receptive field by stacking convolution blocks, and the obtained receptive field is also limited. In addition, in previous crop disease recognition studies, most of them revolve around how to obtain more detailed local information and often ignore the correlation between different locations of the images. Therefore, we input the feature map output from the multiscale convolution layers to the SA module to complement the global receptive field of the model. Due to the ability of the self-attention mechanism to capture long-range dependency, local feature blocks at different locations are correlated. Even the edge information of the image will be identified.As the network deepens gradually, the shape of the feature map becomes progressively smaller due to multiple downsampling. Additionally, the input feature map of each stage is the result of mapping the image several times, and inputting feature maps with different resolutions in different stages will cause the region information represented by the generated patches to change. Therefore, the position information parameter position embedding and the dependencies between patches need to be relearned in each stage.In order to make the model more expressive, we repeatedly stack the MSSA modules at each stage to form a double-loop structure. The feature map output from the MSSA module is re-entered into the MSSA module, and it is worth noting that the positional embedding in the SA module is removed when re-entered. This is because the location information of patch is already embedded in the computed result of the first MSSA module, and the calculation process of the second MSSA module is able to receive the long-range dependence information obtained from the previous calculation. This improves the reuse rate of information and reduces the number of unnecessary parameters and computations. In addition, residual connections are added to the outermost part of the entire MSSA module to prevent the occurrence of model overfitting.Experimental data were obtained from PlantVillage, a public dataset produced by the Pennsylvania State University, and the Apple Leaf Pathology dataset produced by Northwestern Agricultural and Forestry University. The PlantVillage dataset contains 12 healthy leaves and 26 diseased leaves with a total of 54,306 images. It is the most widely used public dataset for the identification of crop diseases. Among all the crop disease datasets, the PlantVillage dataset has been selected by most research works to assess the multiclassification recognition capability of the model because of its large data size and multiple classifications. In the experiments, our proposed model was compared with other research works on the PlantVillage dataset. In addition, the ability to generalize on small datasets is an important basis for the robustness of the model. The Apple Leaf Pathology dataset with an original size of 1591 images was expanded more than 10 times after data augmentation, which is prone to model overfitting; i.e., the model achieves high accuracy on the training set but low accuracy on the testing set. Therefore, we choose the Apple Leaf Pathology dataset to demonstrate that the proposed model can also exhibit the high-accuracy generalization performance of small disease datasets.The models are trained on one NVIDIA RTX 3090 GPU with 24 GB RAM based on a 64-bit Ubuntu 20.04 operating system and PyTorch framework, with Python version 3.8.0, PyTorch version 1.10.0, and CUDA API version 11.3. The experimental parameters are set as follows: stochastic gradient descent (SGD) is used as the model optimizer, the initial learning rate is set to 0.01, the momentum is 0.9, the batch size of the model is set to 64, the number of iterations (epochs) is set to 200, and the loss function is CrossEntropy.Several evaluation metrics are used to evaluate the model, including accuracy, recall, precision, sensitivity, specificity, and F1-score. Accuracy is the proportion of correctly classified samples to the total number of samples, precision is the proportion of samples with positive predictions that are actually positive, and recall is the proportion of samples with positive predictions that are actually positive. Sensitivity (Sens) is the ratio of all positive cases to all positive cases identified, and specificity (Spec) is the ratio of negative cases to all negative cases identified. F1-score takes recall and precision into account and is a weighted average of the two. The calculation formula for each metric is as Equations (5)–(10):We compared the performance of our method with the ResNet family of the ResNet18, ResNet34, ResNet50, and ResNet101 networks on the PlantVillage dataset. As shown in Next, MSCVT was compared with the traditional heavyweight models VGG16 and VGG19 and the lightweight models MobileNetV1 and MobileNetV2. As shown in We also performed a detailed comparison of MSCVT with models used by other researchers. Among the models used for comparison are classical CNN models, such as VGG and GoogleNet, the lightweight MobileNetV1 and MobileNetV2 models, and some improved versions. For fairness, models used for comparison need to be trained and tested on PlantVillage’s dataset. As shown in In the same way, on the smaller Apple Leaf Pathology dataset, we still compared the ResNet family of networks with the proposed MSCVT experimentally. As shown in As shown in In addition, we plotted the confusion matrix based on the whole testing dataset. Class activation mapping (CAM) is a technique for visualizing the image features extracted by a model, which can show how much attention the model pays to each region of the image . A darkeTo highlight the ability of the SA module in capturing global information, we use CAM to visualize the effects produced by the SA module. After removing the SA module from MSCVT, the differences of the feature maps are analyzed. Previous CAM visualization experiments have demonstrated the effectiveness of the self-attention mechanism in the process of crop disease identification by the model. To verify the effect of the SA module on the model performance, we compared the recognition accuracy for this module. The experiments were conducted on the PlantVillage dataset and the Apple Leaf Pathology dataset, respectively. The performance of both the original model and the model with the SA module removed on the two different datasets is shown in In this paper, by combining the benefits of CNN and ViT, we propose a new network MSCVT based on the ResNet architecture, which has a low number of parameters and advanced recognition accuracy for effective crop disease identification. The network is based on the five-layer stage structure of the ResNet network with SGD as the optimizer, a learning rate of 0.01, and a batch size of 64. After 200 epochs of training, the network achieves high recognition accuracy with fewer parameters on two crop datasets. On the work of model compression, the lightweight module of MobileNetV2, an inverted residual block, has been incorporated into our model, which can significantly reduce the cost of parameters required in convolutional operations. For recognition accuracy improvement, we propose the MSSA module, which uses multiscale convolution layers to identify localized disease features and the SA module to enhance the ability of the model to capture disease information at edge locations. Experimental results show that MSCVT exhibits significant recognition effects on the PlantVillage dataset and can achieve accurate recognition of multiple crop diseases. It achieves state-of-the-art performance compared with different types of CNN models. In addition, the proposed model achieved advanced recognition accuracy on the small-scale Apple Leaf Pathology dataset. The proposed model meets the demand for high-precision agricultural disease identification, can accurately identify a variety of crop diseases, makes up for the shortcomings of modern deep learning models for plant disease analysis and processing, effectively reduces human identification errors, and saves the time cost of technicians and experts arriving for identification because agricultural areas are far from cities. Additionally, its lightweight feature also meets the demand for mobile intelligent recognition in actual agricultural production, can quickly identify leaf disease images, and significantly improves the efficiency of pest and disease identification. Future research will attempt to deploy the model on mobile devices for real-time crop disease identification. Additionally, we hope to apply the model to other crop identification tasks.


Quantitative phase imaging and measurement of surface topography and fluid dynamics for objects, especially for moving objects, is critical in various fields. Although effective, existing synchronous phase-shifting methods may introduce additional phase changes in the light field due to differences in optical paths or need specific optics to implement synchronous phase-shifting, such as the beamsplitter with additional anti-reflective coating and a micro-polarizer array. Therefore, we propose a synchronous phase-shifting method based on the Mach–Zehnder interferometer to tackle these issues in existing methods. The proposed method uses common optics to simultaneously acquire four phase-shifted digital holograms with equal optical paths for object and reference waves. Therefore, it can be used to reconstruct the phase distribution of static and dynamic objects with high precision and high resolution. In the experiment, the theoretical resolution of the proposed system was 1.064 µm while the actual resolution could achieve 1.381 µm, which was confirmed by measuring a phase-only resolution chart. Besides, the dynamic phase imaging of a moving standard object was completed to verify the proposed system’s effectiveness. The experimental results show that our proposed method is suitable and promising in dynamic phase imaging and measurement of moving objects using phase-shifting digital holography. Phase-shifting digital holography (PSDH) as a non-contacting, full-field and highly accurate phase measurement technology, has evolved as an important tool in the world for surface topography measurement ,2, biomeAccording to the recording method of holographic images, PSDH can be classified mainly as two types of implementations: asynchronous phase-shifting and synchronous phase-shifting. In the past, driving piezoelectric ceramics  and chanIn order to improve the acquisition rate of recording multi-frame phase-shifted holograms, synchronous phase-shifting techniques based on multi-CCDs recording ,15,16, pIn summary, the current synchronous phase-shift methods face challenges in ensuring an equal optical path for both object and reference waves or may require specialized optics to mitigate issues such as half-wave loss and achieve synchronous phase-shift. To address the aforementioned issues, we present a novel synchronous phase-shifting method based on common optics that is very simple and easy to implement. The common optics include beamsplitters, polarizers and wave plates, which are easily obtainable and do not require special processing. Additionally, no additional anti-reflective coating or reflection-enhancing coating is required for each optical element of the system to avoid half-wave loss. Furthermore, both object and reference waves pass through the same optical elements exactly, so that the additional phase changes in the light field can be avoided. This system can be used not only to accurately reconstruct the phase distribution of static objects, but also to efficiently retrieve the phase distribution of dynamic objects with four CCDs recording phase-shifted holograms simultaneously.In PSDH, the reference wave is imported phase delays to realize phase shifting for each hologram. In this work, we utilized commonly used polarizers and wave plates to introduce additional phases of According to these phase-shifted holograms, Our system based on the Mach–Zehnder interferometer is illustrated in The object wave travels through the observed sample and is introduced in a phase delay of the sample. Then it continues to travel through a linear polarizer, and a quarter-wave plate sequentially to produce polarized light. Because its axis is parallel with that of the polarizer, the quarter-wave plate on the object path does not change the polarization state of the object wave. It should be noted that we used the quarter-wave plate to keep the influence of these optical elements on object and reference waves consistent. The object and reference waves will recombine at the 50/50 polarizing beamsplitter cube. Afterward, their quadrature components are separated by reflection and transmission and will travel in vertical directions. The remixable waves outputting from the polarizing beamsplitter cube travel through a quarter-wave plate oriented at 45 degrees, and their polarization changes from the linear polarization state to the circular polarization state. Then the quadrature components of the mixed circularly polarized light are separated by a 50/50 polarizing beamsplitter cube. Finally, each quadrature component is recorded as a phase-shifted hologram by a CCD.In order to further explain the principle of the synchronous phase-shift method realized in our system, we provided a concrete mathematical description. The system assumed that the coherent beam from the laser is an ideal planar light field after being filtered and expanded. Besides, the thermal noise of the laser, aberration of the optics, shot noise of these CCDs and energy loss of the light are ignored in the following mathematical derivation. The planar coherent beam travels through the polarizer to produce the linearly polarized light. The electric field of a light wave in an arbitrary polarization state can be expressed using a vector notation as:The mixed orthogonal components pass through a quarter-wave plate oriented at 45 degrees relative to the If the refractive index of the sample is equally distributed and known, the height distribution of its surface can be recovered further :(15)Δh=λIn this section, we described the performance of the system for measuring quantitative phase information of the undyed transparent samples. To thoroughly evaluate the effectiveness of our proposed method, we chose the phase-only resolution chart as the static phase object and polymethyl methacrylate (PMMA) pellets as moving objects to reconstruct their phases.In this section, we used a phase-only USAF 1951 resolution chart  as the observed sample to introduce the process of reconstructing the phase. The phase-only USAF 1951 resolution chart was used for confirming the resolution of our system. The phase elements of the resolution chart were magnified by a microscope objective  and the substrate (dashed green rectangle), the height difference between the elements of the resolution chart and the substrate can be calculated. We first measured the phases of the 200 nm to 350 nm elements on the resolution chart. The reconstructed phases of 200 nm, 300 nm, and 350 nm using our system were presented in To verify the ability to reconstruct dynamic objects of our system, four imaging cameras were used to record a standard object at approximately 30 frames per second. As a standard object, its attribute parameters are strictly consistent with the manufacturer’s specifications, with an error of no more than 5%, and it has a high degree of recognition among peers. So, we chose the PMMA pellets (refractive index of 1.49) with a diameter of 10 µm as the standard object. We added PMMA pellets to mirror oil (refractive index of 1.51) and used the aforementioned microscopic objective and lens to image. Because PMMA pellets were placed vertically, they will move downward under the influence of gravity. The photography and the recording schematic were displayed in In this study, we established a quantitative phase imaging system that can be implemented using simple optical components without the need for special processes. Besides, the system could ensure that both the object and reference waves traverse through the same optical elements, thereby avoiding any additional alterations in the optical field. When capturing the phase-shifted hologram, the proposed system is capable of recording four phase-shifted holograms simultaneously. By reconstructing the phase of the resolution chart, we confirmed the high resolution of the system and successfully reconstructed the phase distribution of thin samples. Subsequently, we successfully recorded and reconstructed the dynamic phase distribution of PMMA pellets, demonstrating the ability of the system to measure moving objects. The above two experiments have demonstrated that our system is capable of accurately measuring the phases of both static and dynamic objects without sacrificing resolution. It is expected that this proposed system will contribute to the field of surface topography measurement and fluid dynamics. As a fast, non-destructive, and high-resolution measurement method, synchronous phase-shifting digital holography has the ability to study cellular dynamics and morphological changes ,30. The Note that the proposed system still has some limitations. Although our system can be easily implemented with common optics, it may result in a bulky optical system and higher costs due to the complexity of the components required. In order to accurately reconstruct the phase image of the object, our proposed method relies on an image registration algorithm that aligns the holographic images captured by the four CCDs. This alignment process is a crucial step that ensures precise alignment of the images before proceeding with the phase reconstruction, ultimately leading to reliable and accurate results. Therefore, our main focus will be on simplifying the structure of the synchronous phase-shift system and optimizing the image registration algorithm.


Melanoma is a malignant cancer type which develops when DNA damage occurs . Often, melanoma results in intense and aggressive cell growth that, if not caught in time, can bring one toward death. Thus, early identification at the initial stage is fundamental to stopping the spread of cancer. In this paper, a ViT-based architecture able to classify melanoma versus non-cancerous lesions is presented. The proposed predictive model is trained and tested on public skin cancer data from the ISIC challenge, and the obtained results are highly promising. Different classifier configurations are considered and analyzed in order to find the most discriminating one. The best one reached an accuracy of 0.948, sensitivity of 0.928, specificity of 0.967, and AUROC of 0.948. According to the World Health Organization (WHO), skin cancer is considered one of the most common types of cancer worldwide . Over onMelanoma is a skin cancer that develops from the pigment-producing cells known as melanocytes . The incMelanoma skin cancer requires early diagnosis and treatment, aspects vital to improving outcomes in affected individuals and preventing it from rapidly spreading to other areas of the body. To this aim, detecting and classifying it could represent a useful tool supporting decision making in the diagnosis phase and the whole treatment process ,8.Computer-aided diagnosis (CAD) technologies are utilized to help with the detection of melanoma skin cancer. These systems integrate artificial intelligence (AI), with machine learning (ML) and deep learning (DL) models used to analyze skin images, detect abnormal tissue patterns, and classify them into cancerous or non-cancerous areas ,10. AmonIndeed, CNNs investigate images by applying filters that are able to capture specific features, such as edges or shapes. However, such filters must often deal with the downside of having a limited receptive field. This condition prevents the model from effectively connecting information associated with features located in distant areas of the image with what has been extracted locally. Although specific methods  have been developed to deal with this issue, CNNs still exhibit a lack of comprehension of the global picture context. To further complicate matters, the idea of modifying CNN filters to accommodate for wider receptive fields would lead to a number of parameters that would be unbearably large from a computational cost perspective. Aside from that, tackling the problem of grasping long-range spatial relationships among different parts of an image by leveraging recurrent neural networks (RNNs) has been explored in the literature.Despite their aforementioned challenges, several studies have explored the use of RNNs for image classification. The authors of  proposedLong short-term memory (LSTM) models were born to process sequences and pay attention to their sequential inputs, even at long distances. However, these models are sequential and process one input at a time. Therefore, although they have memory cells designed to keep track of the sequential information, updating the weights progressively considers the contribution of long-range dependencies less and less. By contrast, transformer architectures use the self-attention mechanism to propagate information over time, and each region of the sequence pays attention to all other regions. In this way, all the dependencies—even those at a very long range—are modeled without loss of information. Furthermore, transformers are parallelizable, as they can process multiple inputs simultaneously. Given these characteristics, the advantages of using transformers led us to choose this architecture instead of LSTM or the RNN.The proposed method—based on Vision Transformers (ViTs) —addresseVarious approaches have been implemented using ViTs to classify images. The authors of  presenteThe proposed model is trained and tested on public skin cancer data from the ISIC challenge, making a fair comparison with competitors with similar approaches to dealing with the same clinical issue possible;The performance of the presented ViT-based architecture for skin lesion image classification exceeds that of the state of the art by modeling long-range spatial relationships;Extensive tuning studies are performed to investigate the effect of individual elements of the ViT model.Specifically, the main contributions of the proposed work are as follows:The remainder of this paper is organized as follows. https://challenge.isic-archive.com/data/#2017, accessed on 1 June 2023). Specifically, the dataset served as the basis for the ISIC 2017 Challenge  [The dataset considered for this study is based on a collection of dermoscopic images of skin lesions obtained from the ISIC Archive (ISIC Archive website tection) , where rThe training dataset contains 2000 color dermoscopic skin images in JPEG format, comprising 374 images from melanoma cases, 254 from seborrheic keratosis cases, and the remainder being benign nevi (1372). The validation and test sets were evaluated in new databases with 150 images  and 600 images , respectively . The imaThe proposed methodology is based on ViTs, used to encode images which are able to model long-range spatial relationships among areas in the dermatoscopic image, with respect to traditional deep-learning architectures. Before proceeding with the model’s set-up, the images were properly processed .Finally, the model exploiting the ViT and the MLP head was trained and validated. The end-to-end workflow is represented in Input image: The input image was fed to the model at its original size. A blue-colored block represented the workflow’s starting point.Resizing: The images were resized to 384 × 384 in order to be properly processed by the model.Random resized cropping: The training images were randomly resized and cropped as a part of the data augmentation procedure. This block was marked with yellow to indicate that it only operated on the training images.RandAugment: The RandAugment library [ library  was applPatching: The original image was divided into equally sized, non-overlapping 32 × 32 square patches.Flattening: Each patch was vectorized (flattening) in order to have an embedding for the corresponding token in the input sequence.Embedding + positional encoding: The patches were encoded to account for their positional information.Concatenation (input sequence): The positional encoded vector was added to the corresponding token.Layer normalization: The resulting vector was layer normalized and fed to the transformer encoder.Multi-head attention: The multi-head attention mechanism came into play, reproducing the behavior of multiple “experts” analyzing different areas of the picture, exchanging information, and interacting to come up with a coherent interpretation of the image.Layer normalization: The output of the multi-head attention blocks was added to the embedded input by means of a skip connection, and it was later normalized once more.Multilayer perceptron: The resulting output was fed to an MLP consisting of two fully connected layers with a GELU activation function [function .Classification head: The MLP output was added to the multi-head attention output through another skip connection. The classification head is a fully connected layer that will produce the model output.Classification output: The final outcome was a value that would be used to determine which class the input image belonged to. This block, highlighted in green, marks the end of the workflow.The dataset splits were unmodified with respect to the original database, as described in the previous sections. The images went through the following processing procedure:multi-head attention . In a sense, this process resembles a team of specialists working on the same picture, with each one of them focusing on a specific subsection of the image, drawing conclusions, and sharing knowledge with the others.To achieve the goal introduced above, each input was associated with three main quantities, namely the In order to determine the optimal configuration, the experimental phase was subdivided into two main steps: (1) a search for the best learning rate and (2) an ablation study, with the dual purpose of investigating the effect of the network depth on performance and determining the proper trade-off between model complexity and efficiency. Specifically, to determine the best learning rate, we conducted a systematic search by training the model with different learning rate values. We started with a relatively large learning rate –(3), respectively. Moreover, the area under the receiver operating characteristic curve (AUROC) was calculated :(1)AccurThe experiments were conducted in the Google Colab environment while deploying the models available on Hugging Face. The best selected model was ViT-Large (307 million parameters), with an input resolution of 384 × 384 and a patch size of 32 × 32.The proposed model, which was pretrained on ImageNet , was finOccam’s razor, a value of 24 was chosen for the number of hidden layers , minimizing the architecture’s complexity and optimizing the classification performance.A large difference in performance, on the other hand, was found as the number of layers varied. Regarding the best configuration, the model reached a final test accuracy of 0.948. In the literature, various methods have been proposed to improve the decision-making process in the diagnosis of skin lesions, particularly in cases of melanoma. The authors of  proposedMelanoma detection is a critical task in improving cancer diagnoses and prognoses. To this aim, this work proposed a ViT-based architecture for efficient MM recognition compared with the NCN and SK classes.The proposed model (training and testing procedures) was built up using publicly available skin cancer data from the ISIC challenge. This choice enabled a fair comparison with other competitors who have employed similar methods to address the same clinical issue by using the same dataset. The performance in classifying skin lesion images of the introduced architecture based on ViTs outperformed the current state-of-the-art models. This improvement can be attributed to the model’s ability to capture and model long-range spatial relationships within the images effectively.An in-depth analysis was given in order to assess the influence of the hyperparameters and to consequently select the best model configuration in terms of batch size, learning rate, and number of layers. First of all, the influence of the learning rate during the training phase was evaluated. After selecting the best learning rate value, further investigations were conducted to explore model performance and the number of layers within the architecture. This process was motivated by the necessity to balance efficacy and complexity. Indeed, despite the apparent direct proportionality between performance and network depth, the availability of a tool that can guarantee comparable results when deployed on devices with limited computational resources  is paramount. Furthermore, extensive ablation studies were conducted to explore the impact of individual components of the ViT model, which highlighted their respective contributions to the overall performance.falses (positive as well as negative). This is due to the self-attention mechanism which, by taking into account the correspondences between patches, can better understand the image’s content. The transformer generally works in overfitting. Our ablation analysis also confirmed this, which shows that not all layers are needed. Considering that the last layers on the transformer represent the most abstract relations in the image, it can be deduced that the keys of the better classification require only the low-level features of the image. However, we cannot forget the high computational cost of the transformer, which implies that the proposed classification is offline.The proposed architectural solution represents a trade-off between computational cost and classification performance. The study performed in the tuning phase of the hyperparameters allowed obtaining a predictive model with very high sensitivity and specificity, corresponding to very low amounts of Despite the better results of the proposed approach at convergence, the dynamics of the training see  is an isMelanoma detection is crucial to improving cancer diagnoses and prognoses. This work presented a ViT-based model to efficiently recognize malignant melanoma, nevocellular nevus, and seborrheic keratosis cases. The model was evaluated on a known dataset from the literature called the ISIC challenge to ensure comparison with the state of the art. The results show how the proposed architecture outperformed the state-of-the-art models mainly based on convolutional networks. Ablation studies conducted on the presented model with its hyperparameters highlighted the robustness of the model for classification purposes. Future plans could include the use of attention maps to provide explainability to the model and understand which regions of the images are relevant for classification purposes.


For this study, we investigated efficient strategies for the recovery of individual links in power grids governed by the direct current (DC) power flow model, under random link failures. Our primary objective was to explore the efficacy of recovering failed links based solely on topological network metrics. In total, we considered 13 recovery strategies, which encompassed 2 strategies based on link centrality values (link betweenness and link flow betweenness), 8 strategies based on the products of node centrality values at link endpoints , and 2 heuristic strategies (greedy recovery and two-step greedy recovery), in addition to the random recovery strategy. To evaluate the performance of these proposed strategies, we conducted simulations on three distinct power systems: the IEEE 30, IEEE 39, and IEEE 118 systems. Our findings revealed several key insights: Firstly, there were notable variations in the performance of the recovery strategies based on topological network metrics across different power systems. Secondly, all such strategies exhibited inferior performance when compared to the heuristic recovery strategies. Thirdly, the two-step greedy recovery strategy consistently outperformed the others, with the greedy recovery strategy ranking second. Based on our results, we conclude that relying solely on a single metric for the development of a recovery strategy is insufficient when restoring power grids following link failures. By comparison, recovery strategies employing greedy algorithms prove to be more effective choices. The power grid system is vulnerable to disruptions caused by manufacturing defects, natural disasters, and human actions, such as terrorist attacks ,2,3. EveThe power grid can be represented as a complex network, where nodes represent generators and loads, and where links represent transmission lines and transformers. Analyzing the power grid’s robustness from a network perspective has attracted significant attention. Network robustness is typically evaluated by assessing changes in network performance due to fluctuations such as node removals or link removals . IdentifNumerous studies have focused on strategies for power grid recovery , from blWe examined recovery strategies based on various network metrics, including degree, betweenness, flow betweenness, eigenvector centrality, weighted eigenvector centrality, closeness, electrical closeness, electrical weighted closeness, zeta vector centrality, and weighted zeta vector centrality. Additionally, we compared these strategies to the random recovery, greedy, and two-step greedy strategies.To assess the effectiveness of recovery methods, we utilized the general recoverability framework proposed by He et al. , to measOur study did not consider cascading failures after the recovery or removal of a single transmission line. Instead, we used the direct current (DC) power flow model to maximize power flow satisfaction for loads after a link removal or addition.Despite extensive research on power grid recovery, there is still a lack of investigation into the effectiveness of recovery strategies that rely solely on different network metrics following transmission line failures. From this study, our main contributions are as follows:The paper is structured as follows: R was proposed to measure the performance of a network at a certain time, which is related to the function of the network, i.e., the type of service the network is supposed to support, such as road transport, neuron transport or the spreading of news on a social network. The R-value is normalized in the range In a framework for computing network robustness , networkK. An elementary change in a network is defined as an event that changes the topology of a network, such as a link addition, a link removal or a change in the link weight. An elementary change in the network may result in a change of the R-value. For this paper, we considered elementary changes as the random removal of a link during the attack phase and the recovery of a link previously removed in the attack phase. Additionally, we denoted the number of challenges in the attack process as We assume that perturbations are imposed on a network through a number of elementary changes ,23,24. WR-value, which possibly changes when an elementary change occurs. The R-value at challenge k is denoted as R-value line during the processes can reflect the effectiveness of the attack strategy or the recovery strategy [The performance of a power grid at any time can be captured by a specific strategy , which iThe recoverability energy ratio N nodes is denoted as L links is denoted as A has elements i and j are connected by a link A power grid can be represented by a graph , a˜ij=0 .N, the number of links L, and the average degree For this study, we chose three power grids : IEEE 30R-value in a power grid: the ratio of the total satisfied demand to the total initial demand, called the yields. At challenge k, we denote the amount of satisfied demand at bus i by i by i satisfies i is R-value at a challenge k—can be calculated byAs proposed by Cetinay et al. , we emplk obeys the optimization situation:Laplacian matrix is i; Compared to cascading failure models in power grids, we propose a power flow redistribution mechanism to achieve a steady state by optimizing the total satisfied demand value. The mechanism adjusts the supply and demand of buses, using a linear programming method to ensure that line flows are below the line capacity. Specifically, the objective of the proposed model is to optimize the total satisfied demand while satisfying several constraints, including: (1) the power grid satisfies Kirchhoff’s Law and Ohm’s Law; (2) the total supply matches the total demand at each challenge; (3) the supply and demand of a bus are not beyond its initial values; (4) the absolute value of a line flow is not beyond the line capacity. The DC power flow model  at challi, respectively. The initial supply and demand values of all buses are stored in the vectors L elements, each representing the power flow i and j. The capacity vector L elements and each element The supply and demand vectors For the power grid, the line capacity ce level .R value of the power grid falls below a predetermined threshold. The flowchart of the attack process is shown in In the attack process, we select links uniformly at random, to be removed iteratively until the In the recovery process, links previously removed from the power grid are gradually added back individually, with the order of additions determined by a chosen recovery strategy. The flowchart of the recovery process is presented in A link is randomly selected, to add to the power grid one by one from the set of links that were removed during the attack process.R-value of the power grid in each step and adds the link to the grid. The greedy recovery strategy flow chart is depicted in The greedy recovery strategy is to optimize the performance of the power grid at each stage of the recovery process. The strategy selects the link that yields the largest R-value in one step, an improved approach involves selecting n links that collectively maximize the summation of the R-value in n steps, which can be achieved by exhaustively enumerating all permutations of n links from the set of potential links to be added. By considering multiple steps, more information is incorporated, resulting in potentially better solutions for the n-step greedy recovery strategy. However, if the value of n becomes excessively large, the computational cost becomes prohibitive. Therefore, in this study, we chose to investigate the performance of the two-step greedy recovery strategy. Specifically, the two-step greedy strategy aims to determine two links to be added sequentially, optimizing the sum of the R-values achieved in the two steps. We present the flowchart of the two-step greedy strategy in In contrast to the greedy recovery strategy, which focuses on identifying a single link that significantly improves the i, which can be used to measure the node’s importance. By using the adjacency matrix A, the calculation of the degree is The degree link lij , denotedThe degree recovery strategy involves a sequence of link additions, based on the descending order of the product of the degrees of the end points of each removed link. Consequently, the first link to be added possesses the largest product of the degrees of its end points in the removed link set.s to node t as s to node t through the link The betweenness of a link is widely used to measure the importance of a link  network . If we ds and t if one unit of power is injected to node s and one unit of power is extracted from node t. The flow betweenness s and extract one unit of active power from node t.The shortest paths are the most efficient ways for a flow in a network to travel from one node to another node if the traveling cost of every link is the same and there are no other limitations in the network, like link capacity. However, in a power grid, the distribution of the power flow is determined by the Kirchhoff and Ohm laws, not by the shortest paths. Therefore, Newman  proposedThe sequence of added links in the betweenness or flow betweenness recovery strategies assumes a descending order of betweenness or flow betweenness of links.i is the i-th element of the eigenvector with the largest eigenvalue of the adjacency matrix A. Compared to the eigenvector centrality, the weighted eigenvector centrality i is the i-th element of the eigenvector with the largest eigenvalue of the weighted adjacency matrix The eigenvector centrality measures the importance of a node by not only taking into account the number of its connected nodes but also considering the importance of its connected nodes ,33. The The sequences of links are added in the descending order of the products i and node j is denoted by i is defined asThe closeness centrality of a node relates to the node’s distance from all other nodes . If the We calculate the product i and node j is denoted as Q of the adjacency matrix A: i is given byAs the flow in a power grid follows the Kirchhoff law—not only the shortest path—using so-called effective resistance is a more appropriate way to quantify the distance between a pair of nodes in a power grid . The eff−2(Q†)ij . By analWe use the product i is calculated byCompared to the electrical closeness of a link, the difference in calculating the electrical weighted closeness of a link is that we use the Laplacian matrix The product In the closeness, electrical closeness, and electrical weighted closeness recovery strategies, we prioritize the recovery of links based on the descending order of the products Q. Thus, i.The zeta vector spreader . Specifii.When employing the weighted adjacency matrix To assess the significance of a link based on the zeta vector values of its end points, we introduce the zeta vector metric To restore the removed links in the zeta vector and weighted zeta vector recovery strategies, we adopt a ranking scheme based on the descending order of the links’ zeta vector metric values or weighted zeta vector metric values.R-value of the system fell below a specified threshold. We then used various strategies to restore the system until all the removed links were added. Finally, we computed the recoverability energy ratio of each recovery strategy, given the same random removal process. To explore the impact of thresholds and tolerance levels, we selected two thresholds (0.8 and 0.5) and four tolerance levels  and performed 1000 realizations for a power grid with each setting. In R-value varied in a realization with different recovery strategies for the IEEE 39 bus system and the abbreviations of the strategies used in the following figures and tables in To investigate the effectiveness of recovery strategies in power grids, we conducted simulations on three different power grids: the IEEE 30 bus system, the IEEE 39 bus system, and the IEEE 118 bus system. In each realization, we randomly removed links until the We present the recoverability energy ratios of various recovery strategies concerning two thresholds and a tolerance level equal to 1 for the considered power grids, as shown in For the recovery strategies utilizing link metrics or the product of node centralities of link end points, we observed noteworthy variations in performance across different power grids. Specifically, the recovery method based on link betweenness exhibited diverse rankings of mean recoverability energy ratio when applied to the three power grids under consideration. In the IEEE 30 bus system, the mean recoverability energy ratio rank of the betweenness recovery method fell within the bottom two. Conversely, within the IEEE 39 bus system, the betweenness recovery method achieved an intermediate rank among all the methods based on the mean recoverability energy ratio. Remarkably, the recovery method based on betweenness centrality yielded the highest mean recoverability energy ratio among all the recovery strategies based on ink metrics or the product of the node centrality values of link end points in the IEEE 118 bus system.In addition to the betweenness recovery strategy, the zeta vector recovery strategy exhibited a similar performance pattern. The strategy demonstrated effectiveness in the IEEE 30 bus system, with a threshold value of 0.5, as well as in the IEEE 39 bus system. However, the efficiency diminished when applied to the IEEE 118 bus system. The results indicate that the efficacy of recovery methods based on link metrics or the product of node centralities of link end points is contingent upon the underlying network topology and dynamics of the networks.The observation can be attributed to the distinct allocation of generators in the IEEE 39 bus system, as illustrated in Compared to the IEEE 30 bus system and the IEEE 39 bus system, the recovery methods based on link metrics and the product of node centrality values of link end points in the IEEE 118 system demonstrated substantially better performance. Specifically, most recovery strategies based on link metrics and the product of node centrality values of link end points exhibited higher average recoverability energy ratios than the random recovery strategy for both thresholds.Link capacity is a crucial factor in causing blackouts in power grids. Although our method did not employ the cascading failure model, we investigated the impact of link capacity on recovery strategies. To this end, we conducted simulations with different tolerance levels (Based on our study, we observed significant variations in the performances of different recovery strategies. Firstly, the recovery strategies based on link metrics and the product of the node centrality values of link endpoints did not achieve the highest performance. Secondly, the performance of those recovery strategies varied notably across different power systems, emphasizing the significance of network topology and generator placements in power grids. Thirdly, the two-step greedy recovery strategy consistently outperformed all other network metric-based recovery methods across all power systems, thresholds, and tolerance levels. The greedy recovery strategy, which was slightly less effective than the two-step greedy recovery strategy, also showed promising results. The findings confirm that implementing an effective recovery strategy can significantly improve the recoverability of power grids and that the two-step greedy strategy is particularly efficient, while relying solely on one network metric for recovery is less effective.For future research directions, several avenues are worth exploring. Firstly, we can develop hierarchical recovery strategies, by classifying links into two groups—those connected to generators and those not connected to generators—and then prioritizing restoring links connected to generators. Secondly, we can investigate whether allowing cascading failures after link recovery or removal impacts the conclusions of this study. Thirdly, we can introduce additional constraints—such as generator costs—and incorporate cost minimization as an objective within the DC power flow model. Finally, given that distributed energy systems enhance power system resilience, it is promising to research how to establish and integrate such systems after a blackout, using the discussed strategies.


This paper proposes, analyzes, and evaluates a deep learning architecture based on transformers for generating sign language motion from sign phonemes (represented using HamNoSys: a notation system developed at the University of Hamburg). The sign phonemes provide information about sign characteristics like hand configuration, localization, or movements. The use of sign phonemes is crucial for generating sign motion with a high level of details (including finger extensions and flexions). The transformer-based approach also includes a stop detection module for predicting the end of the generation process. Both aspects, motion generation and stop detection, are evaluated in detail. For motion generation, the dynamic time warping distance is used to compute the similarity between two landmarks sequences (ground truth and generated). The stop detection module is evaluated considering detection accuracy and ROC (receiver operating characteristic) curves. The paper proposes and evaluates several strategies to obtain the system configuration with the best performance. These strategies include different padding strategies, interpolation approaches, and data augmentation techniques. The best configuration of a fully automatic system obtains an average DTW distance per frame of 0.1057 and an area under the ROC curve (AUC) higher than 0.94. Motion generation is an innovative research field which consists of producing movements, gestures, or animations by computer systems. This process involves the use of mathematical algorithms to create dynamic, natural, and fluid motions that simulate human movements, enhancing human–computer interactions. Motion generation has many different applications, such as movement of virtual characters in video games or animated films, robotics, and sign language representation in communication systems for deaf people. In sign language communication systems, motion generation allows for an increase in the naturality of interactive avatars or virtual assistants that respond with a sign language output. Increasing naturality allows for the development of friendly communication systems between deaf and hearing people. These systems empower and enhance the communication capabilities of deaf individuals, encouraging inclusivity and facilitating their integration into various social and professional scenarios.Most state-of-the-art sign language motion generation systems are based on expert rules or prerecorded movements. This work proposes to train a module to automatically generate sign language motion from sign phonemes, represented using HamNoSys . The proProposal and evaluation of a deep learning architecture based on transformers for generating the sequence of landmarks to represent a sign.This proposed approach also includes a stop module for deciding the end of the generation process. This stop module is also evaluated in different scenarios.Additional analyses for improving the system accuracy, considering different padding strategies, interpolation approaches, and data augmentation techniques.To the best of the authors’ knowledge, the proposed system is the first motion generation system for sign language based on transformers. The main contributions of this paper are the following:This paper is organized as follows. Motion generation is a research field aimed at developing realistic 2D or 3D animations based on different inputs such as image sequences, speech recordings, or text descriptions. In the literature, we found several datasets for motion generation and different evaluation metrics to assess the quality of generated motion. This section reviews some previous works about motion generation, including information about evaluation metrics and specific works related to sign language.Within human motion modeling ,4,5,6, mRegarding the assessment of the generated motion, there are several possible ways to proceed. Some metrics, like smoothness, refer to how natural the generated motion is, and others, such as diversity or multimodality , refer tEven though all these metrics have been used in the literature, some previous works have highlighted the efficacy of DTW compared to others, especially for aligning landmarks sequences. For example, a previous work , focusinIn this work, we use a DTW-based metric to evaluate the proposed landmark generation model. The metric consists of using DTW to align the sequences of ground truth and generated landmarks, and computing the average error between them.Some previous works in the literature have directly focused on sign language motion generation, exploring the interplay between linguistic expression through videos or textual descriptions and physical movements that represent that language.A recent work  proposedSome works focused on generating motion from natural language. For example, in a previous work , the autOther works generated motion from speech. For instance, a previous work  focused Finally, a previous paper  providedThe state-of-the-art technology for sequence generation is based on transformers. This technology has been used to generate different types of sequences. Li et al.  used a tThis previous system  has a stThe dataset used in this work includes sign descriptions (phonemes specified in HamNoSys), sign videos, and landmarks for each frame of the videos . The datThis dataset includes sign phonemes represented using HamNoSys . This noThis dataset includes landmarks (the x and y coordinates) of 57 pose and hand relevant points extracted with MediaPipe  for eachThese 57 landmarks specify 15 locations of the body including the face, shoulders, and elbows, and 21 locations for each hand including the wrist and four points along each of the five fingers. For this representation, it was necessary to combine MediaPipe Pose and MediaPipe Hands models to extract information from the whole body, and from both hands without ambiguity.This section describes the methods used in this work: a transformer-based architecture to generate motion, which includes a stop detector. This description includes also the different variants considered in the experiments: several padding strategies, interpolation approaches, and data augmentation techniques.In our proposal, we used a transformer-based architecture, similar to the architecture used in a previous work  for speeTo develop this architecture, we used a transformer code implemented with keras_nlp . In our The encoder module codes the sign phonemes in order using a token and position embedding layer, which prepares the data for further processing by the decoder. This layer sums a token and position embedding to generate an embedding vector for each sign phoneme. The decoder takes a sequence of landmark vectors as inputs, which do not require tokenization, so only a positional embedding layer is needed. This layer learns a position embedding for decoder inputs sequences.The decoder aims to predict the next set of landmarks based on the encoder information and the previous landmarks. In summary, the system generates two outputs: the predicted landmarks and a stop output, which indicates the end of the generated sequence. The system architecture incorporated two distinct loss functions to optimize its performance. The mean absolute error (MAE) was employed for the generation of landmarks, providing a measure of the absolute difference between the predicted and ground truth landmark coordinates. Simultaneously, the binary cross-entropy loss function was strategically integrated into the architecture for the binary classification of the stop detector. Various techniques for representing each sign through landmarks and designing the stop output will be analyzed in the following subsections.To implement the whole system including the landmark generation and the stop detection layer, we have included two different output layers in the decoder . These lThe primary focus of the proposed deep learning architecture based on transformers is generating sign language motion sequences based on landmarks, using sign phonemes as input.This section outlines the processes undertaken for generating the landmarks using the transformer architecture. To generate the predicted landmarks for the transformer, we used a subset of the database described in For the encoder’s inputs, we used the SiGML representations of the signs. We only used the sign phonemes corresponding to the manual aspects . These characteristics are compiled in a single line, as shown in Secondly, for the decoder’s input we used the landmarks described in In our system, only the encoder inputs need to be tokenized. To accomplish this, we considered the WordPiece tokenization algorithm, used by BERT and other models, that splits the text, in our case the hand configurations, into smaller units called tokens . [PAD]: Padding tokens which are appended to the input sequence length when the input sequence length is shorter than the maximum sequence length.[UNK]: Unknown tokens.We trained the transformer considering two reserved tokens:Here is an example of tokenizing and detokenizing an entry of the encoder, where next indicates the end of the sequence. Hand Configurations: hamflathand hamthumboutmod hamextfingeru hampalmu hamshoulders hamlrat hamparbegin hammovei hamarcu hamreplace hamextfingerui hampalmd hamparend hamrepeatfromstart next nextTokens: tf.Tensor , dtype = int32)Recovered text after detokenizing: hamflathand hamthumboutmod hamextfingeru hampalmu hamshoulders hamlrat hamparbegin hammovei hamarcu hamreplace hamextfingerui hampalmd hamparend hamrepeatfromstart next nextDuring training, the tokenized sign characteristics are passed as inputs to the encoder. The decoder inputs are the landmarks of the first frames [0 to end −1 frames] and the output are the last frames [1 to end frames]. This way, the model is autoregressive (the output of the current step depends on previous steps), being trained in teacher forcing mode (using previous ground truth landmarks for predicting the next landmark).During testing, the tokenized sign characteristics are also passed as inputs to the encoder. The main difference is in the decoder. The inputs of the decoder start with the landmarks of a resting position and, iteratively, the predicted landmarks generated by the transformer (next frame) are fed back as inputs to the decoder, allowing the transformer to predict the next set of landmarks based on the previous ones (autoregressive loop).Having defined the entire data processing pipeline, the last step is to construct the transformer model. The inputs and output of the transformer differ during training and inference:The stop information (indicating with 1 when to stop) is incorporated into every frame, as an additional column: 0 when a sign is taking place, and 1 to represent the end of a sign .In this work, we implemented and evaluated several padding strategies for making the sign language sequences have the same length.In the first strategy, zero padding, all the signs of the landmark-based sequences are padded with zeros at the end until they reach a fixed maximum length (80 frames in our case). This approach extends the shorter sequences to match the length of the longest sequence in the dataset. The second strategy, sign padding, consists of repeating the sign until the sequence reaches a fixed maximum length. Finally, the third strategy, hybrid padding, combines both ideas, padding with zeros and repetition of the sign: an intercalation of several zero frames is included before repeating the sign. In this work, three zero frames have been included between sign sequences.These padding strategies only affect the landmarks. The stop information is fixed to 0 for the first occurrence of the sign and 1 for the rest of the frames until the maximum length is reached.In the experiment section, the impact of the padding strategy will be evaluated in both transformer outputs: landmark generation and stop detection.The proposed approach based on a transformer predicts the landmarks of a frame given in the previous frames, generating the whole sequence in an iterative process. This prediction is more difficult when there are big differences in consecutive frames. One interesting idea to facilitate the learning process of the transformer is to reduce these differences by interpolating intermediate frames. We considered and evaluated a linear interpolation of one, two, or three intermediate frames between two original frames.Data augmentation is a technique commonly used in machine learning to artificially increase the size and diversity of the training subset by applying different transformations to the existing data. This helps to improve the generalization and robustness of machine learning models. In the context of landmark representations, we applied two data augmentation techniques: shifting and zooming.For shifting, we added a random number between −0.2 and 0.2 to all the landmark coordinates in a sign, displacing the signing subject.For zooming, we compressed or expanded the images by 10% from the center of the scene. First, the center was subtracted from the landmark points, and then we multiplied each coordinate of the landmark points by a random number between 0.9 and 1.1. And at the end, we summed the center to the landmarks. This process uniformly compresses or expands the size of the landmark representation from the center of the scene.These techniques increased the diversity of data, which could enable the model to better handle different positions, sizes, and scales of landmarks, improving its ability to accurately identify and localize landmarks in unseen and diverse data. We evaluated the impact of introducing these additional data during the training process.The evaluation methodology in the experiments was a 5-fold cross-validation, where the 2262 examples were divided into five groups or folds to train, validate, and test the system with different data. Specifically, 80% of the data was used for training, 10% for validation, and 10% for testing. The data splitting and the system evaluation were repeated five times, with different subsets. The final results were the average of the outcomes obtained from all iterations.Optimizer: Adam optimizerNumber of epochs: 250Batch size: 32Number of attention heads: 6Learning rate: 0.001Dropout rate: 0.3Intermediate dimension: 1024Embedding dimension: 128The validation subset is used to optimize the hyperparameters of the transformer-based architecture: In the next sections, we evaluate in detail the process of generating the landmarks and the detection of the stop frame. These two aspects were independently examined to analyze which is the optimal setup for both tasks (landmark generation and stop detection) in terms of padding strategy, interpolation, and the use of data augmentation.To assess the quality of motion generation, we used the dynamic time warping (DTW) distance that compares ground truth and generated motion sequences. The goal of this metric is to find the best mapping between the predicted sequence of landmarks and the ground truth sequence. This best mapping corresponds to the minimum accumulated distance, computed by the DTW using dynamic programming. This distance is divided by the length of the sign (in frames). The required distance between two landmark frames is defined by Equation (1), where the subscripts “O” and “P” indicate the original and predicted frames, respectively, and the subscript “i” denotes the landmark number. In this work, we used the DTW functionality obtained from the dtaidistance.dtw_ndim.distance Python function .(1)DTWOIt is important to highlight that, in this subsection, we compute the DTW distance per frame between the predicted landmark-based motion sequence and the ground truth, assuming a perfect detection of the stop frame. As needed by the analysis, we will include the effect of automatically detecting the stop frame.In the next subsection, we compare the different padding strategies, interpolation approaches, and data augmentation techniques.Additionally, We hypothesize that sign padding works better than the other two strategies because we provide the transformers with more examples and opportunities to learn how to generate the landmarks of the next frame.Having concluded that the best padding strategy for landmark generation is sign padding, this section will examine the effect of interpolation. As commented previously, the purpose of introducing intermediate frames is to smooth the landmark sequences and facilitate the learning process of the transformer-based architecture.From these results, we can conclude that the interpolation helps the training process obtain a significant reduction in the DTW distance: the average distance is around 0.05, and all examples are lower than 0.2. As shown, when increasing the number of interpolated frames, we observed a saturation behavior of the system with a small difference between two and three interpolations. We think that using only two interpolated frames can be a good compromise between performance and time processing.Finally, we evaluated the system incorporating synthetic data in the training of the transformer. In the next figure , we showAs shown, we did not obtain any improvement when introducing the augmented data during the training of the transformer. Analyzing the results, we observed a good match between the training and testing subsets , and introducing modified data did not improve the results. In future work, we will consider more different subsets for training and testing.To evaluate the stop detection module, since this detection is a binary classification task, standard evaluation metrics for two classes were used for evaluating the performance: accuracy, receiver operating characteristic (ROC) curves, and area under the curve (AUC). Accuracy is defined as the ratio between the number of correctly classified samples and the number of total samples. ROC is a curve representing the true positive rate (TPR) versus the false positive rate (FPR) obtained at several thresholds settings. AUC refers to the ratio of the area under the ROC curve. This measure represents the separability between two classes. The higher the AUC, the better the model distinguishes between the two classes of the binary classification problem.N samples when the confidence level is 95%.For accuracy values, we also considered confidence intervals  to analyp-values.To analyze the statistical significance of the AUC differences in the two experiments, we provided two-tailed As in the previous section, we analyzed the impact of the different padding strategies, interpolation approaches, and data augmentation techniques for the stop detection. p-values. p-value between the sign padding and hybrid padding strategies, as they are the best strategies. To determine the best padding strategy, we analyzed the AUC values with the corresponding p-value analysis confirmed the significance of the difference between hybrid padding and the rest of strategies.As observed, all padding strategies showed very good results with AUCs over 0.93. The hybrid padding strategy provided the highest AUC value. Furthermore, the p-value analysis for the three interpolation approaches using a hybrid padding strategy, and p-value between the AUC of “No Inter” and “IntPol-1” experiments, and the p-value between the AUC of “IntPol-2” and “IntPol-3” were higher than 0.001, which implies that the differences between these experiments were not statistically significant. However, the p-value between the AUC of “IntPol-1” and “IntPol-2” was lower than 0.001, which determines a significant improvement: including two interpolated frames provided significant better results than using no interpolation.After selecting the hybrid padding strategy, we proceeded to analyze the effect of interpolation on stop detection. Similarly to the previous section, we analyzed AUC values and ROC. As observed, introducing interpolated frames obtained improvements compared to no interpolation. These results are consistent with the improvement already obtained in landmark generation. When comparing the number of interpolated frames, we observed a better AUC with two interpolated frames, but the differences were not significant. We can conclude that interpolating with two frames is a good compromise between performance and processing time, and it is also consistent with the best scenario regarding interpolation obtained in landmark generation. Lastly, similar to the evaluation of motion generation, we assessed the effect of applying data augmentation by employing the hybrid padding strategy and adding two rows of interpolation. This analysis will help us to understand the impact of data augmentation on the overall performance.p-value between the hybrid padding + interpolation of two frames strategy and the hybrid padding + interpolation of two frames + data augmentation strategy.p-value confirmed statistical independence between the results. This significant difference supports the conclusion that the interpolation and data augmentation strategies improved the performance of the stop detector. As observed, applying data augmentation resulted in a better AUC value for the stop detector due to the increase in the number of examples in the training process. Additionally, the analysis of the Despite obtaining better results with data augmentation in the stop detector, subsequent experiments continued without this strategy. This decision was based on the observation that data augmentation did not provide any improvement in landmark generation. Thus, to maintain consistency and focus on the most effective strategies, data augmentation was not applied to subsequent experiments for landmark generation.In order to extend the analysis, we showed the detection accuracy depending on the decision threshold. We also showed the average length  of the ground truth and predicted sequences, and the length difference between them. We carried out this analysis with the architecture obtained in the previous subsections (hybrid padding with two interpolated frames). The table also includes the difference between the number of predicted frames by the transformer and the number of frames of the ground truth. This analysis also allowed us to examine the effect of the threshold on this aspect.When increasing the threshold, the average number of predicted frames increases. Considering that the automatic system can generate signs with a different pace compared to the ground truth (faster or slower sign representations), we decided to select the threshold based on the detection accuracy. In this case, we can conclude that the threshold reporting the best validation accuracy was 0.5.In In this section, we present the results with the test subset combining motion generation and stop detection (with a 0.5 threshold) in the same system. For this purpose, we considered a scenario where both systems performed well, and we chose the sign padding strategy, as it was the best scenario for landmark generation, along with two interpolated frames. As observed, when adding the stop detector, the DTW worsened compared to the best scenario analyzed in the previous sections. However, even with the addition of the stop detector, we achieved a DTW value lower than the one obtained in the scenario where only the sign padding strategy was applied without the stop detector (using perfect sign limits). The degradation when using the stop detector was not constant in all experiments; for example, when considering the stop detector without interpolation, we observed a higher performance degradation (average DTW of 0.1908 ± 0.0992) than in the case with interpolation (average DTW of 0.1057 ± 0.0659). This result is because both systems (landmark generation and stop detector) share the same transformer (a relevant part of the deep learning architecture), and the quality of the transformer has a strong influence on the performance of both tasks.Considering previous research focusing on generating pose animation from natural language, a previous work  providedThis work proposed a sign language motion generation system using sign characteristics or phonemes as inputs. The sign characteristics are represented using HamNoSys, a notation system for any sign language developed at the University of Hamburg. The sign phonemes provide information about sign characteristics like hand configuration, localization, or movements. This information is necessary to generate sign motion with a high level of detail. The proposed architecture based on the transformer also includes a stop detection module for predicting the end of the generation process. Both aspects, motion generation and stop detection, were evaluated in detail considering specific performance metrics. In order to improve the proposed architecture, different padding strategies, interpolation approaches, and data augmentation techniques were considered and evaluated. Regarding the padding strategies, the repetition of the sign (instead of including zeros) empowered the learning process of the transformer reporting significant performance improvements. The introduction of interpolated frames smoothed the frame sequence, facilitating the learning process of the transformer and reporting major improvements. This was the most significant contribution. The introduction of data augmentation in the training did not provide significant improvements due to the fact that the data used for training and testing are very similar. More difficult scenarios must be addressed to demonstrate its utility.Regarding the limitations of the work, the motion generation is limited to sign-level representation, instead of complete sentences in sign languages. In future works, we will explore the idea of generating motion from complete sentences instead of from individual signs. Additionally, we would like to reproduce the experiments with bigger datasets. Finally, we could incorporate an additional motion evaluation metric based on a deep learning classifier. The idea is to develop a sign recognition system and evaluate how well this system can recognize the generated signs.