This mixed-methods study investigated the impacts of the self-regulated strategy development (SRSD) model on the self-efficacy of students for text revision in English-as-a-foreign-language (EFL) writing at the tertiary level. An SRSD treatment group and a comparison group were involved in this quasi-experimental design research. Both groups completed a self-efficacy scale before and after the instruction, and six SRSD-trained students participated in pre- and post-test interviews. The quantitative analyses did not detect any significant differences between groups, suggesting that the SRSD instruction did not influence the self-efficacy of participants for text revision. The qualitative findings provided insights into the quantitative results. The interview data indicated that the interviewees might have overestimated their revision abilities before instruction and, with the relatively more accurate estimation of their abilities resulting from receiving the SRSD instruction, the over-time comparison of their responses to the self-efficacy scale did not reveal any statistically significant changes. Our findings suggest that students might have recorded evidence of closer calibration between judgments of their revision abilities and their actual performance after SRSD instruction. The implications of the findings were discussed and directions for further studies were provided. Revision is a critical part of the writing process as well as writing instruction not only in one's first language (L1) but also in a foreign language (FL). It is of great instructional significance as inexperienced L1 and FL writers need to learn how to revise effectively to become proficient writers. It also provides an opportunity for instructors to teach students about the characteristics of effective writing in a way that will carry over to future writing . Independent samples t-tests were performed on the baseline data of the two groups, and no statistically significant differences were found between the groups. The two groups were comparable in terms of their age, length of formal English learning, and gender proportion , a 17-item scale developed to evaluate the perceived capabilities of students to perform activities involved in text revision in EFL writing  of it were satisfactory . The researchers used a 10-point Likert scale with gradations from 0 (I cannot do it) to 10 (Highly certain that I can do) instead of a 0–100 response format that was advocated in many self-efficacy studies  and their perceived capability to revise at the low level . Factor analyses were performed in their study to assess the psychometric properties of the scale, and their results showed that the reliability  and validity  and showed them how to utilize positive self-statements . Then teachers introduced the strategies to be learned, along with the instructional goals and the significance of revision. Teachers also asked questions to elicit the reflection of students on their revision process to explore further their current revision ability. This was carried out in a positive, collaborative manner with an emphasis on the fact that they have not mastered the target strategies. The importance of students' efforts was emphasized to facilitate the development of motivation and positive attributions. In the modeling stage, teachers modeled how and when to use the REVISE strategy and revision tactics  , self-evaluation , and self-reinforcement . Students participated in engaging activities designed to help them recite the revision tactics and the steps and the meaning of the REVISE strategy. They were encouraged to monitor their use of the revision strategies and graph their revision performance by counting the number of revisions.The instruction began with teachers developing students' background knowledge of argumentative writing and writing evaluation criteria. During this stage, teachers also helped students to identify whether their writing and revising performance were hindered by negative self-statements .The fidelity of treatment implementation was evaluated using a checklist of lesson components and ratings of the key elements of the SRSD model. The key components were rated on a 3-point scale . One of the researchers observed and completed the checklist for all SRSD lessons and an EFL teacher evaluated a random 20 percent of the audio recordings of the lessons for each SRSD instructor. Overall, the SRSD instruction was implemented as in lesson plans with high fidelity in the treatment group. Across the two instructors and all lesson components, the average rating was 1.82 , the 8 week SRSD writing instruction began (once a week for 90 min). The comparison group received regular writing instruction required by the University curriculum and syllabus. All groups practiced using the same writing tasks in their writing textbook and received the same length of writing instruction. To ensure participants in the comparison group were not disadvantaged, they were offered the SRSD instruction after the research and were provided with all the resources used in the SRSD groups.p > 0.05) along with P-P plots, and the results indicated that the data were approximately normally distributed. The assumption of homogeneity of variances and covariances, as measured by Levene's test (p > 0.05) and Box'es M test (p > 0.05), was also met  and instructional conditions (treatment and comparison) as the independent variables and self-efficacy scores as the dependent variable. Prior to the test, the assumption of normality was assessed by the Shapiro–Wilk test of normality . In cases where the two coders find the concept conveyed by text chunks could not be neatly categorized, the third researcher coded these parts and disagreements were resolved Data coding started with us familiarizing ourselves with the transcripts to reflect on their overall meaning and the tone of the ideas  = 0.32, p = 0.57, partial η2 = 0.01], suggesting the two groups developed similarly in their ratings for self-efficacy for high-level text revision over time. There was also no significant interaction in mean ratings for self-efficacy for low-level text revision , indicating minimal variance in the ratings for this subscale at different time points.Descriptive information on the ratings for the subscales of L2TRSS is presented in A considerable number of comments from the pretest interviews suggested that they lacked the motivation to revise while being overconfident in their revision abilities. As can be seen from A cluster of responses is related to the unwillingness of participants to initiate or sustain efforts in revision in specific situations, thus forming the theme of indication of a lack of motivation to revise. Four codes are related to this theme: first, no attempts to revise on one's own initiative if teachers did not instruct them to revise; second, no attempt to revise on one's own initiative in absence of readers; third, discontinuing revision when desirable writing scores were obtained; and lastly, discontinuing revision when confronted with difficulties. The comments clustering around the first code, no attempts to revise if not being required by teachers, indicated that they were inclined not to engage in any forms of revision on their own initiative. As shown in the example in this code in Students mentioned another external source for their motivation to revise, the presence of readers, forming the second code that related to students' reluctance to revise when readers were absent. As can be seen from the excerpt under this code in Comments in the third code, discontinuing revision after desirable scores were obtained, indicated that student's motivation to revise was driven by better performance in writing tests. A typical comment, as presented in The final code in this theme related to these students' discontinuing their engagement with revision the instant that they met difficulties in the revising process. As can be seen from the example under the final code in Despite the indication of the unwillingness of interviewees to revise the content, evidence in support of their confidence in revising the surface features of texts was detected, which was encapsulated in the theme of indication of over-confidence in revision. Comments in the code of insisting on the use of words illustrated that these students might stick to their way of using words even if others pointed it out that they were used incorrectly. A typical response is included in The interviewees indicated a greater willingness to revise the higher-level features of writing with the use of revision strategies and the procedures to carry them out. They also indicated the need for further practice in using the revision strategies to hone their revision skills, which contrasted with their overconfidence prior to the treatment. The first salient code in the theme of indication of willingness to revise higher-level features of texts is the willingness to revise across different levels of texts, reflecting students' emergent willingness to attend to global issues in texts rather than solely focusing on local issues. Students articulated their desire to improve the organization of ideas in texts. As can be seen from the example in Another cluster of comments pointed to the attention of students to argument elements in texts; students expressed a motivation to engage in extra efforts to revise argument elements in their texts in comments that were categorized into the code of willingness to revise argument elements. An example was provided under this code in The second theme, an indication of efforts exerted and sustained in learning to revise, is subdivided into two areas: initiating efforts in practicing revision strategies and sustaining efforts in learning the use of revision strategies. The comments related to the first code expressed a view that the teacher's modeling of the revising process, particularly of how to select and apply the revision strategies, prompted these students' motivation to explore the use of these revision strategies. As can be seen from a typical response in this code in These students also spoke of continuing engagement with the practice of revision strategies, forming the code of sustaining efforts in learning the use of revision strategies. As one student explained, although he knew that he had not fully mastered the use of the revision strategies, as evidenced by his comment that he was “not an expert in using them” see , he beliTo sum up, compared with the pretest interviews, the posttest interviews showed that these students reportedly had become more efficacious in applying the revision strategies and expressed increased intrinsic interest in revision. Specifically, the students interviewed expressed an increased willingness to perform high-level revisions after the treatment, actions they used to find burdensome and thus avoided doing. They were also less likely to make negative statements, suggesting that they lacked motivation for revising.The quantitative results obtained in this study showed a minimal variance in ratings of students for the self-efficacy scale after the SRSD instruction, indicating a limited instructional impact on the self-efficacy of students. These findings were at odds with previous studies, where positive effects of SRSD instruction on the writing self-efficacy of students were found (e.g., Teng and Zhang, Nevertheless, previous studies in L1 writing have also reported no obvious relationship between SRL strategies and levels of self-efficacy (e.g., Graham et al., Participants in the current study might also have overestimated their revision abilities, especially their capabilities to revise the high-level features of texts, at the outset. Their pretest ratings for the subscales were both a little bit over the median see , althougThis study attempted to explore the effects of the SRSD model on self-efficacy of college students in the domain of text revision in an EFL context. Although conclusive evidence supporting students' enhanced self-efficacy for text revision after the SRSD revision instruction was not obtained, our results suggest that SRSD strategy employment might have helped EFL students to form more accurate judgments about their capabilities to revise the following instruction. An implication for educational practice in teaching revision is that EFL instructors are advised to be aware of learners' overconfidence, particularly about difficult tasks such as text revision, when feedback suggesting poor performance has been provided. Moreover, it is recommended to monitor self-efficacy judgments of students along with their performance in case either unrealistically high levels of self-efficacy or undesirably low levels of efficacy negatively influence the learning behavior of students. Researchers (e.g., Schunk and DiBenedetto, Several limitations of this study should be borne in mind when interpreting the findings. First, convenience sampling was used to recruit participants in the current study for easy accessibility, and hence, the two groups were of different majors, which might have functioned as a variable influencing their self-efficacy level. Although the two groups reported similar levels of self-efficacy at the outset, they developed differently in self-efficacy levels because of their various majors. Another limitation, as discussed above, is that the interviewees might not be adequately representative of the population. For ethical considerations, to ensure that every student had an equal opportunity to participate, the first six students who volunteered were selected, regardless of their writing proficiency or self-efficacy level. These volunteers might be relatively motivated to write. Nevertheless, in line with the primary objective of the study, namely, the possible changes in the indicators of self-efficacy, the analysis and interpretation of interview data focused on the variance in the self-efficacy. Also, previous studies have shown that it is the students of relatively low writing proficiency and weak motivation that benefitted the most from the SRSD instruction (De La Paz and Sherman, Besides addressing the limitations of the current study, further research is required to explore several issues. First, further studies should extend the duration of the SRSD instruction to allow the self-efficacy of students to fully develop. Second, to understand whether a lack of task familiarity accounts for overestimates, future studies could explore whether improvements in students' task knowledge are related to changes in their self-efficacy. A better understanding of the reasons explaining such overestimation could shed lights on how to counter the mismatch between students' actual performance and perceived capabilities. Also, other motivational factors need to be examined in future SRSD studies to better understand the instructional effects of the SRSD model as well as to understand the complex operation of self-efficacy. Prior studies have explored other motivational factors such as mastery motivation, and positive results were found in both self-efficacy and mastery motivation (MacArthur et al., The original contributions presented in the study are included in the article/The studies involving human participants were reviewed and approved by The University of Auckland Human Participants Ethics Committee. The patients/participants provided their written informed consent to participate in this study.JC and LZ conceived and designed the study. JC collected the data and drafted the manuscript. LZ finalized the manuscript for submissions. All the authors revised the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The computer vision community has paid much attention to the development of visible image super-resolution (SR) using deep neural networks (DNNs) and has achieved impressive results. The advancement of non-visible light sensors, such as acoustic imaging sensors, has attracted much attention, as they allow people to visualize the intensity of sound waves beyond the visible spectrum. However, because of the limitations imposed on acquiring acoustic data, new methods for improving the resolution of the acoustic images are necessary. At this time, there is no acoustic imaging dataset designed for the SR problem. This work proposed a novel backprojection model architecture for the acoustic image super-resolution problem, together with Acoustic Map Imaging VUB-ULB Dataset (AMIVU). The dataset provides large simulated and real captured images at different resolutions. The proposed XCycles BackProjection model (XCBP), in contrast to the feedforward model approach, fully uses the iterative correction procedure in each cycle to reconstruct the residual error correction for the encoded features in both low- and high-resolution space. The proposed approach was evaluated on the dataset and showed high outperformance compared to the classical interpolation operators and to the recent feedforward state-of-the-art models. It also contributed to a drastically reduced sub-sampling error produced during the data acquisition. Single-image upscaling, known as Single-Image Super-Resolution (SISR), is a classical computer vision problem, used to increase the spatial resolution of digital images. The process aims to recover fine detail High-Resolution (HR) image from a single coarse Low-Resolution (LR) image. It is an inherently ill-posed inverse problem, as multiple downsampled HR images could correspond to a single LR image. Reciprocally, upsampling an LR image with a 2× scale factor requires mapping one input value into four values in the high-resolution image, which is usually intractable. It has a wide range of applications, such as digital display technologies, remote sensing, medical imaging care, and security and data content processing. Classical upscaling methods, based on interpolation operators, have been used for this problem for decades, and it remains an active topic of research in image processing. Despite their achieved progress, the upscaled images are still lacking fine details in texture-rich areas. Recently, the development of deep learning methods has witnessed remarkable progress achieving performance on various benchmarks ,2 both qThe computer vision community has paid more attention to the development of visible sensor images, and over the last decade, non-visible light sensors such as infrared, acoustic, and depth imaging sensors were only used in very specific applications. The images produced by those sensors lack the consideration of the potential benefits of the non-visible spectrum due to their low spatial resolution, the high cost incurred as their price increases dramatically with the increase of their resolution , and theIn contrast to visible or infrared cameras, there is not a single sensor for acoustic imaging, but rather an array of sensors. As a result, the image resolution in acoustic cameras is directly related to their computational demand, requiring hardware accelerators such as Graphical Processor Units (GPUs)  or Fieldond rate ,8. Acousond rate , which mond rate . Due to Learning-based SISR methods rely on high- and low-resolution image pairs generated artificially. The conventional approach is generally by downsampling the images using a bicubic interpolation operator and adding both noise and blur to generate corresponding low-resolution images. However, such image pairs do not exist in real-world problems, and the artificially generated images have a considerable number of artifacts from smoothing, removing sensor noise, and other natural image characteristics. This poses a problem as models trained on these images cannot be generalized to another unknown source of image degradation or natural image characteristics. There are a few contributions where the image pairs were obtained from different cameras in the visible imaging, but none in the acoustic imaging.A novel backprojection model architecture was proposed to improve the resolution of the acoustic images. The proposed XCycles BackProjection model (XCBP), in contrast to the feedforward model approach, fully uses the iterative correction procedure. It takes low- and high-resolution encoded features together to extract the necessary residual error correction for the encoded features in each cycle.https://doi.org/10.5281/zenodo.4543786), provided simulated and real captured images with multiple scales factor . Although these images shared similar technical characteristics, they lacked artificial artifacts caused by the conventional downsampling strategy. Instead, they had more natural artifacts simulating the real-world problem. The dataset consisted of low- and high-resolution images with double-precision fractional delays and sub-sampling phase delay error. To the best of the authors’ knowledge, this is the first work to provide such a large dataset with its specific characteristics for the SISR problem and the sub-sampling phase delay error problem;The acoustic map imaging dataset  features (coarse image) and high-frequency (HF) features . The results of the classical interpolation operators and the previous deep learning-based models have high peak signal-to-noise ratios (PSNRs), but they lack HF features . As the In contrast to this simple feedforward approach, Irani et al.  proposedThe iterative error correction mechanism in the feature space is very important for the HF features’ reconstruction in the SR problem. If the model pays more attention to this correction procedure in the upsampled HR and the LR feature space, it might be possible to obtain improvements in the residual HF detail in the super-resolved image. This paper proposed a fully iterative backprojection error mechanism network to reconstruct the residual error correction for the encoded features in both low- and high-resolution spaces.Acoustic cameras acquire their input signal from arrays of microphones placed in well-defined patterns for high spatial resolution. Microphone arrays come in different shapes and sizes, offering different possibilities for locating a neighboring sound source. The process of locating a sound source with a microphone array is referred to as the beamforming method. Beamforming methods comprise several families of algorithms, including the Delay-and-Sum (DaS) beamformers ,23,24, tThe well-known DaS is the most popular beamformer and was the one selected to generate the acoustic images in this work. The beamformer steers the microphone array in a particular direction by delaying samples from the different microphones. The acoustic signals originating from the steering direction are amplified, while acoustic signals coming from other directions are suppressed. The principle of DaS beamforming can be seen in M microphones and m delayed by a time m in the array, and the unitary steering vector c) in air.Here, The principle described is valid in the continuous time domain. Therefore, the principle of DaS is extended with the sampling frequency This For a sufficiently high sampling frequencies i yields:Based on these equations, the output values Equation  can be tL samples for each of the steering vectors with:From Equation , one canBy computing the SRP values for each of the steering vectors, a matrix of SRP values can be generated. When all steering vectors have the same elevation, the matrix will be one dimensional, and a polar plot can be used for visualization and finding the origin of the sound source. On the other hand, when the steering vectors have a changing elevation, the matrix will be two dimensional. When this matrix is normalized and, optionally, a colormap is applied to it, the acoustic heat map is obtained.Examples of the SRP are depicted in The DaS beamforming technique properly delays the input audio samples to generate constructive interference for a particular steered direction. These time delays are obtained based on Equation  and rounite case . Due to ite case .This kind of degradation is shown in ng stage . FractioEquations  and 4)  4) are rBased on the floor and the ceiling of the With the double-precision weighing coefficient n used in the fraction and rounded to the nearest natural number.Double-precision computations demand a great amount of computational power, which is unavailable on constrained devices such as embedded systems and can lead to intolerable execution times and low frame rates. Luckily, double-precision delays can be changed into fractional delays The fractional delays In Equation , the nomEquation . The main allows a more fine-grained DaS computation. For values of n beyond four, the resulting response is almost equal to the response of the DaS with double-precision interpolation.The effects of the fractional delays are demonstrated in In addition to a single-response result, the effect of fractional delays is also visible in the waterfall diagrams of the microphone array . When noThis section first introduces the characteristics of the target acoustic camera  and the Acoustic cameras, such as the Multi-Mode Multithread Acoustic Camera (M3-AC) described in , acquireThe type of microphone, the sampling methods, and the signal processing methods influence the final outcome of the beamforming. For instance, the microphone array of M3-AC is composed of MEMS microphones with a PDM output. Despite the benefit of using digital MEMS microphones, there is a need for a PDM demodulation in order to retrieve the acquired audio. Each microphone converts the acoustic into a one-bit PDM signal by using a Sigma-Delta modulator . This moTo retrieve the encoded acoustic signal, a set of cascaded filters are applied to simultaneously decimate and recondition the signal into a Pulse Coded Modulation (PCM) format . Both thTraditional datasets for SR consist of high-resolution images that are downsampled and blurred to generate the low-resolution image, or they use different cameras to generate two images of the same scene . When usThe CABE is used to generate the acoustic images. The CABE can emulate the behavior of the traveling sound, microphones, and stages that are required for generating the acoustic heat map. The main advantages of using an emulator over real-life acoustic heat maps are consistency and space. First, considering the consistency, using an emulator makes it is possible to replicate the exact same acoustic circumstances multiple times for different resolutions and different configurations. One could generate the same acoustic image with two different microphone arrays for example or use a different filtering stage.Second, considering space, an emulator eliminates the requirement of having access to anechoic boxes or an anechoic chamber. This allows one to generate acoustic images with sound sources that are several meters away from the microphone array. In order to achieve the same results in a real-world scenario, a large anechoic chamber is required. If one has access to such a chamber, the CABE could still be used. The CABE has the option to use PDM signals from real-life captured recordings to generate the acoustic heat maps. Doing so will omit everything that comes before the filtering stage and replace it by the PDM signals from the real-life recording.In order to have a representative dataset, the same architecture as in  was usedScale: The key application of our proposed acoustic map imaging dataset is to upscale the spatial resolution of a low-resolution image by multiple scale factors . To realize this, eight different sets of images were generated, each containing 4624 images. Four different resolutions were used: the HR ground truth images of size  where the distance between the two sound sources became too small to distinguish one from the other. For instance, when both sound sources were placed at 90°, they overlapped and became one sound source.Normalization: Natural images in SISR problem are logged in the uint8 data type, which is in agreement with the most recent thermal image benchmarks. Although acoustic and thermal sensors allow the generation of raw data logged in float representation, which consists of rich information, this could produce a problem in the validation consistency between benchmarks. To avoid technical problems with the validation, the proposed benchmark was standardized with the current research line to be compatible with published datasets in other SISR domains.I and The registered sound amplitude in acoustic map imaging depends on the sound volume, the chosen precision during the computation, and the number of microphones. The more microphones that are used during the computations, the higher the amplitude will be in the registered map. This can generate high variant values in the minimum-maximum value range. Due to this, it is not possible to preserve the local dynamic range of the images and to normalize them using fixed global values. It may also cause a problem with the unknown examples. Any method of contrast or histogram equalization could harm the images and cause the loss of important information. Consequently, instance min-max normalization as in Equation  and the Baseline approach: The authors believe that this is the first work to provide as large a dataset of acoustic map imaging pairs captured with four different scales in an emulator and in the real world. The PSNR and the Structural Similarity Index Measure (SSIM) metrics are reported for reference in SISR problems. A bicubic algorithm was used as a baseline model for validation comparison for the super-resolved images. To reduce quality degradation caused by subsampling errors in phase delay estimation, a Gaussian kernel with different kernel sizes was used on top of the bicubic output to reduce the artifacts. Train and test set: Because of the large number of images in the dataset and to avoid possible overfitting due to shared similarities in the images, ninety-six samples were drawn for the test set from the images with low PSNR value. The test set sampling procedure was processed on the PSNR distribution built on the 8× bicubic upscaled images. The sampling toward the low PSNR value images was processed using a Kernel Density Estimator (KDE) skewed distribution. In the end, the test set became biased with more complex examples.X CFCs, as each cycle contained one RFE module. The value of x was an odd number since two consecutive cycles were mandatory for the final results. The model used only one convolutional layer (Encoder (E)) to extract the shallow features F from the low-resolution input image The baseline model of the proposed method was comprised of two main modules: Cycle Features Correction (CFC) and Residual Features Extraction (RFE). As shown in Equation . The preD) of only one convolutional layer uses the final features X to reconstruct the super-resolved image The term The CFC module serves as a feature correction mechanism. It was designed to supply the encoded features of the two parallel feature spaces to its RFE module for further feature extraction and corrects the encoded features once at a time. The output of its RFE module is backprojected by addition to one of the two parallel feature spaces. The backprojection serves to correct the previous features’ location in both encoded features’ manifold in contrast to . By haviU) before adding the correction to match the scale of the features, as shown in Equation FLRU) is a resize-convolution [The Upsampler  in each CFC depicted in I)) responsible for extracting deep features from the two parallel spaces. One (I) for each of the feature spaces of only one convolutional layer, with strided convolution in the high-resolution space to adapt to the different resolution scale. The two deep encoded features are then concatenated, and a pointwise convolution layer [As depicted in on layer  transforL of double activated convolution layers connected sequentially. The output of the three L levels, defined as the inner skip connections, are concatenated together, creating dense residual features before a pointwise convolution layer returns them to their original channel space size. Finally, the output of the merger layer is fed to a channel attention module, inspired by the RCAN model [The main core of the RFE module consists of three levels AN model , to weigThe final XCBP proposed model was constructed with X = 8 cycles. All convolution layers were set to 3 × 3 except for the channel reduction, whose kernel size was 1 × 1, to transform the concatenated features to their original channel space size. All features were encoded and decoded with 128 feature channels. Convolution layers with kernel size 3 × 3 used the reflection-padding strategy and the PReLU  activatiThe experiments were implemented in PyTorch 1.3.1 and performed on an NVIDIA TITAN XP. Ninety percent of the training images were selected, with a total of 4067 image pairs. Data augmentation was performed on the training images with random rotation of After every epoch, the validation metric was run on the current model, and the model with the highest PSNR value was recorded for inference. The model of scale ×2 was trained first. Subsequently, the model was frozen, and the model of scale ×4 was added and trained. The same procedure was performed for the model of scale ×8.Despite the quality of the experimental quantitative comparison, the super-resolution problem demands an analytical comparison, as it is not possible to rely only on quantitative standardized metrics such as the PSNR/SSIM. To confirm further, a row profile test was run on the images to observe the output with the closest result to the ground truth with fewer artifacts. In The obtained images were indeed very similar to the original ground truth HR images. Though this was subjectively confirmed in the cropped regions, the same conclusions can be drawn after observing the entire image such as in Both the increase in resolution and the reduction of artifacts helped to improve the quality of the acoustic images and acoustic cameras. First of all, the reduction in noise helped the overall image quality and readability for humans of set images. A second improvement was the frame rate of acoustic cameras. In order to increase the resolution of an acoustic camera without upscaling, one needs to compute more steering vectors and perform more beamforming operations. Beamforming operations are computationally intensive, meaning that increasing the number of steering vectors or the resolution of an acoustic image also increases the computational load and time to generate one acoustic image, giving a trade-off between resolution and frame rate. Using super-resolution to both upscale and reduce the artifacts of nonfractional delays tackled these two problems at the same time. The acoustic camera could generate the acoustic image at a lower resolution and higher frame rate without the need for fractional delays because the super-resolution improved the resolution and quality, without affecting the frame rate.The super-resolution also allowed better pinpointing the direction sound was coming from and distinguishing sound sources. Outliers caused by the fractional delays were removed, and the increased resolution helped to better estimate the angle of arrival of the sounds. In order to determine what the origin of the sound was instead of the angle of arrival, acoustic images were combined with images from RGB cameras. RGB cameras have a higher resolution. To overlay acoustic images with RGB images, both must have the same resolution by increasing the resolution of the acoustic image, decreasing the resolution of the RGB image, or a combination of both. Here, the super resolution can help to match the resolution of the acoustic image with the resolution of the RGB camera.The similarities in the acquisition of the developed acoustic map imaging dataset may lead to characteristics similarities in the image distribution and to overfitting the data. Given this, the proposed model was tested on real captured images to study its ability to generalize over an unseen data distribution. Although the real captured images still shared characteristics with the simulated images in terms of the number of sound sources and their frequency, they were recorded using different equipment and in another environment, which reduced the possibility of overfitting. This work proposed the XCycles BackProjection model (XCBP) for highly accurate acoustic map image super-resolution. The model extracted the necessary residual features in the parallel (HR and LR) spaces and applied feature correction. The model outperformed the current state-of-the-art models and drastically reduced the sub-sampling phase delay error estimation in the acoustic map imaging. An acoustic map imaging dataset, which provided simulated and real captured images with multiple scale factors , was also proposed. The dataset contained low- and high-resolution images with double-precision fractional delays and sub-sampling phase delay error. The proposed dataset can encourage the development of better solutions related to acoustic map imaging.


This paper proposes a method to embed and extract a watermark on a digital hologram using a deep neural network. The entire algorithm for watermarking digital holograms consists of three sub-networks. For the robustness of watermarking, an attack simulation is inserted inside the deep neural network. By including attack simulation and holographic reconstruction in the network, the deep neural network for watermarking can simultaneously train invisibility and robustness. We propose a network training method using hologram and reconstruction. After training the proposed network, we analyze the robustness of each attack and perform re-training according to this result to propose a method to improve the robustness. We quantitatively evaluate the results of robustness against various attacks and show the reliability of the proposed technique. The hologram is a record of a fringe pattern that occurs due to interference between a reference wave as a reference and an object wave reflected on an object and contains two-dimensional (D) data of a complex plane  but contains 3D image information. The digital hologram may be generated by obtaining an analog hologram in which the interference phenomenon is realized by an optical device with a digital photographing device or sampling the analog hologram to generate a digital hologram. However, in recent years, digital holograms are generated by mathematically modeling and calculating the interference between two waves and by generating them using deep learning ,2,3.Recently, digital holograms have been widely used due to the digital hologram compression standard of JPEG Pleno , the devArtificial intelligence technology using deep learning has been widely used in all fields. Similarly, in the area of digital watermarking, studies using deep learning are increasing. There is no published technology for digital holograms yet, but deep learning-based technologies show good performance in the digital watermarking of 2D images. Zhu et al. proposed a method to embed the watermark globally by duplicating 1D watermark data in 3D . This meWe intend to try holographic watermarking using deep learning. We show that the deep neural network can adequately interpret the components of the hologram and prove that the deep neural network can hide the watermark invisibly in the hologram. It is shown that the watermark embedded in the hologram can be very robust by including a malicious attack in the deep neural network’s training process. Since the image quality of hologram reconstruction is essential, we train the invisibility of reconstruction simultaneously. We propose a deep neural network (DNN) that performs watermarking of digital holograms and a training method that considers the characteristics of holograms. In addition, if the network is vulnerable to a specific attack after the train, we propose a method to improve the robustness of the attack through re-training.This paper is organized as follows. In In this section, we explain the previous studies about hologram watermarking and deep learning-based watermarking in depth.Javidi et al. proposed a technique to use digital holography to hide a 3D object stored in the form of a digital hologram within another 3D object, which generates the digital hologram of both the hidden 3D object and the host 3D object optically using phase shift interferometery. Then, the hologram of the hidden 3D object is encoded using double phase encoding. The encoded hologram of the hidden 3D object is embedded within the hologram of the host 3D object. The watermarked hologram is double phase encoded again using a different set of random phase codes. The second double phase encoding process provides a higher level of security and will guarantee the whiteness of the transmitted hologram. The watermarked hologram is robust to pixels occlusion in terms of reconstructing different poses of the 3D object due to the whiteness of the final transmitted data. Mathematical analysis and digital experiments showed that the proposed system has a reasonable performance . Kim andOur team has developed several hologram watermarking methods. We proposed spatial-domain and a frequency-domain electronic watermarking schemes. The spatial-domain scheme was only to compare the results to the ones from frequency-domain scheme and the frequency-domain scheme used two-dimensional mallat-tree discrete wavelet transform. Both of them showed very high imperceptibility and quite high robustness against the attacks. Especially the MDWT-domain scheme was very high in robustness such that the error ratio at the worst case was only 3%. Thus, we expect that it is used as a good watermarking scheme of digital hologram with high performance. However, the spatial-domain scheme turned out to be useless when data compression process is necessary after watermarking . Our teaZhow et al. proposed a spatiotemporal consistent embedding algorithm for the holographic video watermarking. Imperceptibility requirement in the holographic video watermarking is more challenging compared with static holograms because of the temporal dimension existing in videos. The embedding algorithm should not only consider spatially embedding strength for each frame of the video but also take the temporal dimension into account in order to guarantee the visual quality of the moving object. Before embedding, to defend the imperceptibility of the watermark from the holographic moving object, the embedding parameters were evaluated by the salient object from the interframe and intraframe. To ensure the robustness, 3-D watermark converted data (QR code) were embedded in the cellular automata (CA) domains using 3-D CA filters. The QR codes can be extracted from the watermarked holographic frames, and the final 3-D watermark can be digitally reconstructed with different depths’ cue using the computational integral imaging reconstruction algorithm. The experimental results demonstrated that the proposed method exhibits superior performance compared to several methods in the literature, especially the robustness against additive noise and compression attacks .NN-based watermarking schemes have been proposed ,17,18,19This section explains the proposed hologram watermarking network and training methodology.The proposed DNN consists of three sub-networks; resolution-converting network (RCN), watermark-embedding network (WMN), and watermark-extracting network (WXN). All networks use 3 × 3 convolutional layer. We show the structure of the DNN for hologram watermarking in The RCN consists of the hologram RCN and the watermark RCN. The hologram RCN has a convolution layer that keeps the resolution of channels constant and expands the host hologram to 64 channels. The watermark RCN consists of four convolutional layers, which increases the resolution of a watermark up to the same resolution of the host hologram. The former three layers have the structure of convolution-batch normalization-ReLU(activation)-AP(average pooling), and the last fourth layer has the structure of CONV-AP. After multiplying the strength factor to the output of the last layer, the feature maps are output. The strength factor is decided by experiment. The structure of the RCN is shown in The hologram and watermark feature maps with the same resolution are concatenated and formed to a channel in the WMN. The WMN consists of five CNN blocks, which keeps the resolution of the input feature map unchanged. The WMN does not include a pooling layer. The former four layers of the WMN consist of CONV-BN-ReLU, and the last fifth layer consists of CONV-tanh. The reason that the last layer uses the tanh as the activation function is to normalize the output to the range of  to be the same as the host data. The structure of the WMN is shown in The input of the WXN is the hologram in which the attack was applied to the watermarked hologram. The input hologram is normalized as in the embed process, and watermark information is extracted from the normalized hologram. This result is denormalized again, converted into binary numbers with only −1 and 1 values, and the final watermark is extracted.The WXN consists of four CNN layers, and the resolution is reduced in each layer so that the output of the last layer has a watermark resolution. The first three layers consist of CONV-BN-ReLU, and the fourth layer uses CONV-tanh so that the output has a binary value of . If the result range of the fourth layer is , 1 is assigned, and if the range is , a value of −1 is assigned. The structure of the WXN is shown in We propose a training method for hologram watermarking together with the DNN. We try to increase the robustness of hologram watermarking by including attack simulation in the training process. The signal processing-based watermarking algorithm has rules to defend against every predicted attack after analyzing them, but deep learning-based watermarking considers attacks in the training process. The host hologram itself is very important, but the reconstruction of the hologram is more critical for invisibility because users observe not the hologram but the reconstruction. Therefore, we insert the reconstruction of holograms in the training process. Including attacks for the watermarked holograms.Considering reconstruction error of the watermarked holograms.Considering stereoscopic quality in reconstruction.As previously explained, the attack for holograms is included in the training process to add robustness to the watermarking algorithm. The reconstruction process is also included because of its importance for invisibility. The pixel-by-pixel correlation between the real and imaginary parts may be expressed as phase information, and the phase information corresponds to the stereoscopic information of a hologram. If the watermarking algorithm does not consider the relationship between real and imaginary parts, it affects the stereoscopic property regardless of the robustness of watermarking. Therefore, the amplitude and phase component should be considered together with the real and imaginary parts to consider the stereoscopic property of the hologram after reconstruction:The integrated training method for the DNN is shown in The first loss function The second loss function Since the data used in Equation  is binarGenerally, the invisibility of the embedded watermark has trade-off relationship with robustness. In other words, if a watermark is weakly embedded with small strength factor, invisibility is increased, but robustness is decreased. Conversely, if the watermark is strongly embedded with a large strength factor, invisibility decreases and robustness increases. Since watermarking pursues both invisibility and robustness, two factors cannot be separated. Therefore, in this paper, the loss function Equation , and theEquation .(3)LEMBIn Equations (3) and (4), The proposed DNN can improve its robustness through various training methods. The structure and depth of the network can be variously adjusted, and the dataset for a train can be variously modified. In addition, the form and type of attack can be variously modified, and the performance of the network can be improved by adjusting hyperparameters. We propose a method to improve the robustness by performing re-training for a specific attack after analyzing the robustness against an attack. After training the network, the inference is performed using the trained network. An attack with weak robustness may be selected from the inference result compared to other attacks, or an attack requiring higher robustness may be selected. This process is expressed as robustness analysis in The proposed DNN was programmed using Python and Tensorflow and implemented in the computing environment of Intel(R) Xeon(R) Gold 5120 CPU @ 2.20 GHz with 180 GB RAM and NVIDIA Tesla V100-SXM2-32GB GPU. A hundred digital hologram patches were used for mini-batch. The training was carried out up to 400 epoch, and the value of 0.0001 was used for the learning rate. For optimization the Adam optimizer  of Equation , where mJPEG Pleno is an organization that studies the standard compression framework for new image representation formats such as texture-plus-depth, light field, point cloud, and hologram . JPEG plAmong datasets provided by JPEG pleno, three datasets  used in this paper include 46 kinds of holograms. We regard the real and imaginary part of the complex hologram with R, G, and B channels as an independent hologram, which means that we can use 172 holograms for our study. First, we explain the training result of the DNN for watermarking. The training is mainly classified into two cases of reconstruction included and reconstruction not included. In the test (inference) results for invisibility in From the results of This paper showed that watermarks could be successfully embedded and extracted in digital holograms using deep learning. For digital holograms, the quality of the reconstructed image is essential, and the three-dimensional effect of the hologram is essential. We included the image quality of reconstruction in training to improve invisibility in the reconstruction of digital holograms. In 2D or 3D images, the quality of the domain itself in which the watermark is embedded is essential. Since the user observes the information reconstructed through the interference and diffraction of the hologram, not the hologram itself, we proposed a training model for the deep neural network by reflecting the unique characteristics of the hologram. We conducted training for both the case where the reconstruction domain was not included in the training and the case where the reconstruction domain was included. As a result, it was confirmed that the robustness was higher when the reconstruction domain was included. In addition, according to the hologram format, it was confirmed that embedding in the amplitude component showed higher robustness and invisibility than embedding in the complex part. According to the result of analyzing the robustness of the attack, it was shown that performance could be improved by re-training the weak attack. Therefore, it was shown that the proposed method can have high versatility for watermarking and it has adaptive characteristics against various attacks.


In this paper, a neural adaptive fault-tolerant control scheme is proposed for the integrated attitude and position control of spacecraft proximity operations in the presence of unknown parameters, disturbances, and actuator faults. The proposed controller is made up of a relative attitude control law and a relative position control law. Both the relative attitude control law and relative position control law are designed by adopting the neural networks (NNs) to approximate the upper bound of the lumped unknowns. Benefiting from the indirect neural approximation, the proposed controller does not need any model information for feedback. In addition, only two adaptive parameters are required for the indirect neural approximation, and the online calculation burden of the proposed controller is therefore significantly reduced. Lyapunov analysis shows that the overall closed-loop system is ultimately uniformly bounded. The proposed controller can ensure the relative attitude, angular velocity, position, and velocity stabilize into the small neighborhoods around the origin. Lastly, the effectiveness and superior performance of the proposed control scheme are confirmed by a simulated example. Nowadays, with the rapid development of sensing and control technologies, space missions have become increasingly complicated. The spacecraft proximity operation plays an important role in various space missions, such as rendezvous and docking, active debris removal, and on-orbit servicing. The relative attitude and position control is a critical technique for spacecraft proximity operations. During the proximity operations, the chaser and target are inevitably affected by uncertain parameters and disturbances. Even worse, the parameters of the target may be fully unknown for noncooperative proximity operations. In addition, the chaser also frequently suffers from actuator faults due to the harsh space environment. The presence of unknown parameters, disturbances, and actuator faults bring great difficulty to the relative attitude and position control of spacecraft proximity operations. Traditionally, the spacecraft relative attitude and position control systems are often designed independently. However, the inherent couplings between the relative attitude and position are neglected in this way and these controllers cannot be directly applied to the spacecraft proximity operations, especially when high control accuracy is required. The integrated attitude and position control based on the six-degree-of-freedom (6-DOF) dynamic model of spacecraft proximity operations is an effective solution to this problem.Until recently, many relevant results have been reported for the integrated attitude and position control of spacecraft proximity operations. Singla et al.  designedIt should be noted that most of the above controllers require prior knowledge of nominal model information for feedback. Nevertheless, the physical parameters of the chaser and the target may be fully unknown in some extreme cases. The intelligent approximation is an efficient tool to construct the model-free controllers, owing to the powerful learning capability of the neural network (NN) and fuzzy logic system. By adopting the NNs or fuzzy logic systems to approximate the lumped unknowns, the intelligent control does not need any model information for feedback. In ,29, robuBoth the relative attitude control law and relative position control law are designed by integrating with the neural approximation. Benefiting from this design, the proposed controller is model-free and strongly robust against the lumped unknowns in 6-DOF dynamics;Rather than the conventional intelligent approximation ,29,30, iLyapunov analysis shows that the overall closed-loop system is ultimately uniformly bounded. The proposed controller can ensure that the relative attitude, angular velocity, position, and velocity stabilize into the small neighborhoods around the origin.Motivated by the above discussions, this paper proposes a neural adaptive fault-tolerant control scheme for the integrated attitude and position control of spacecraft proximity operations in the presence of unknown parameters, disturbances, and actuator faults. The proposed controller is made up of a relative attitude control law and a relative position control law. In comparison with most of the existing investigations, the main contributions of this research are summarized as follows:The remainder of this paper is arranged as follows: Consider the spacecraft proximity operation system depicted in The modified Rodrigues parameters (MRPs) are utilized to represent the attitude orientation of the chaser. Then, the attitude and position dynamics of the chaser can be expressed in frame Similarly, the attitude and position dynamics of the target can be expressed in frame According to the geometric relationship in Note that Remark 1.From the 6-DOF dynamic model of spacecraft proximity operations (8) and (9), the relative translational motion of the target with respect to the chaser is heavily affected by the relative rotational motion due to the inherent coupling between the relative attitude and position.The purpose of this research is to design a controller for the spacecraft proximity operation system such that relative attitude Lemma 1.For any continuous nonlinear function,, it can be approximated by a radial basis function NN (RBFNN) asRef.  For any whereis the ideal RBFNN weight,is the basis function vector,is the identification error satisfying,is a positive constant, andis the number of RBFNN nodes. Moreover,is commonly chosen as the Gaussian functionwhere, andare the center and width of the Gaussian function, respectively.The structure of the proposed neural adaptive fault-tolerant control scheme is shown in First, consider the relative attitude subsystem described as (6) and (7). Introduce the following filtered error:Theorem 1.When the relative attitude control law (18) and the adaptive updating law (19) are employed to the relative attitude subsystem described as (6) and (7), the overall closed-loop system is ultimately uniformly bounded and the relative attitude can stabilize into the small neighborhoods around the origin.Proof. Introduce the following Lyapunov function:Substituting the relative attitude control law (18) and the adaptive updating law (19), we haveConsider the following inequalities:Then, consider the relative position subsystem described as (8) and (9). Introduce the following filtered error:Theorem 2.When the relative position control law (31) and the adaptive updating law (32) are employed to the relative position subsystem described as (8) and (9), the overall closed-loop system is ultimately uniformly bounded and the relative positioncan stabilize into the small neighborhoods around the origin.Proof. Introduce the following Lyapunov function:Remark 2.In the conventional intelligent approximation ,29,30, t Alternatively, the indirect neural approximation is exploited in this paper by adopting NNs to approximate the upper bound of the lumped unknowns. In this way, only two adaptive parameters,, are required for theindirect neural approximation, and the online calculation burden of the proposed controller is therefore significantly reduced. Actually, theindirect neural approximation makes the proposed controller more suitable for practical engineering, especially considering the onboard computer has limited online calculation capability.Remark 3.According to Theorems 1 and 2, the proposed controller can ensure the relative attitude, angular velocity, position, and velocity stabilize into the small neighborhoods around the origin. From (26) and (39), it follows that the small neighborhoods around the origin are adjustable. If we set the parameters as large as required, the small neighborhoods can be made sufficiently small.Remark 4.It is noteworthy that the RBFNN utilized for intelligent control in this paper can also be replaced by some other approximation tools, such as recurrent NNs, wavelet NNs, and fuzzy logic systems. Moreover,adaptive dynamic programming is an effective methodology for the optimal control of unknown nonlinear systems with the help of critic NNs ,33,34,35. Future investigation building on this research will focus on extending the present results by embedding them with an adaptive dynamic programming approach.A simulated example is performed to illustrate the proposed control scheme. The sampling frequency for feedback is set as The masses of the chaser and the target are chosen as Due to the actuator saturation, the acceptable maximum control torques and forces are set asIn addition, the actuator faults are also considered. The actuator health factor matrices are given asThe commonly used proportional-derivative (PD) controller is employed for performance comparisons. The compared PD controller is also made up of a relative attitude control law and a relative position control law. The relative attitude control law is designed asThe parameters of the proposed neural adaptive fault-tolerant controller are given as The translational motion of the chaser and the target for proximity operation is provided in From Furthermore, some crucial indexes are introduced to quantitatively compare the performance between the proposed controller and the PD controller. Specifically, the integrated absolute errors (IAEs) are defined as In summary, the simulation results indicate that the proposed neural adaptive fault-tolerant controller can realize superior performance and good uncertainty rejection capability, which guarantees the successful implementation of the spacecraft proximity operation.This paper aimed to propose a neural, adaptive, fault-tolerant control scheme for the integrated attitude and position control of spacecraft proximity operations in the presence of unknown parameters, disturbances, and actuator faults. The proposed controller is made up of a relative attitude control law and a relative position control law. Both the relative attitude control law and relative position control law were designed by adopting the NNs to approximate the upper bound of the lumped unknowns. By introducing the indirect neural approximation, the proposed controller is more suitable for practical engineering, especially considering the onboard computer has limited online calculation capability. The ultimate uniform boundedness of the overall closed-loop system can be achieved through Lyapunov analysis. The proposed controller can ensure the relative attitude, angular velocity, position, and velocity stabilize into the small neighborhoods around the origin. Lastly, simulation results indicate the effectiveness and superior performance of the proposed control scheme.


As a part of the multi-source cooperative navigation scheme, data fusion has a significant impact on the quality of state estimation. Particle filtering has gradually become the focus of many fusion methods due to its unique theoretical advantages in nonlinear non-Gaussian systems. However, the particle degradation and the resulting sample impoverishment restrict its application in complex engineering scenarios. In this paper, a robust cubature fission particle filter (RCFPF) is proposed to deal with these problems. First, in the framework of cubature rule, Huber function is used to combine the L2 norm and L1 norm to improve the importance density function(IDF), suppress the observation noise. Meanwhile, the proposed distribution(PD) is further optimized by combining the Gaussian distribution with Laplace distribution to alleviate particle degradation. Second, the particle swarm is fissioned before resampling, and the particle weight is reconstructed by fission of high weight particles and covering low weight particles to inhibit sample impoverishment. The vehicle experiments of multi-source cooperative navigation show that the proposed algorithm achieves better test results in accuracy and robustness than extended Kalman filter (EKF), strong tracking particle filter (STPF), and cubature particle filter (CPF). Therefore, it has attracted extensive attention in recent years. In order to improve the efficiency of position perception and obtain higher precision cooperative navigation solutions, scholars have made a lot of research work and gradually improve the quality of navigation parameters7. However, most of these studies integrate new navigation modes from the perspective of measurement and update the cooperative navigation system to obtain better solutions. There are few improvements on the data fusion methods that play an important role in the solution. The improvement of relevant literature remains within the framework of Kalman filter (KF) requiring linear Gaussian system8. Because of the inherent theoretical deficiency, its accuracy in dealing with multi-source complex systems is always limited to some extent. Unlike Kalman filter, particle filter (PF) has almost no requirements for the system and noise characteristics9, so it becomes a possibility of multi-source cooperative navigation data fusion.The relative position provided by cooperative navigation is the basis of vehicle intelligent transportation and location-based service10. Gordon et al. performed resampling after Monte Carlo importance sampling, so that PF began to have a basis for practical applications11. Since then, PF has gradually become a focus in the field of data fusion and improved with various theories18. Although particle filtering has made great progress in recent years, particle degradation and the resulting sample impoverishment problem have always restricted the further development and application of the above-mentioned various improved PFs18. Particle degradation means that after multiple iterations, only a small number of particles in the particle swarm have large weights, while most of the particles have negligible weights, so that a lot of calculation time and resources are wasted on particles with small weights, which affects the performance of the algorithm21. Kong has proved that particle degradation is an inherent problem of PF22. Doucet and Wang further pointed out that the variance of particle weight increases with time24. However, the proposed distribution (PD) generated by the importance density function (IDF) in PF is the direct approximation of the particle to the posterior probability distribution, and its simulation accuracy has an important impact on the estimation performance. Selecting an appropriate IDF can suppress the weight variance and alleviate the degradation problem25. Therefore, researchers have carried out a lot of researches around IDF.As one of the realization methods of Markov-Bayesian recursion, PF uses a set of weighted particles to iteratively update to approximate a posteriori probability density distribution of the system model. In theory, it can carry out probability reasoning and recursive estimation for any system26. This choice is easy to implement and will not increase additional computational load. However, using the state transition probability density represented by the state equation as the PD does not consider the current measurement information. Since the particle weight is proportional to the likelihood function, choosing the equation of state as the IDF will make the variance of the particle weight larger, especially when the likelihood function is relatively "steep" or located at the tail of the state transition probability distribution, it will accelerate the degradation and even cause the filter divergence27. To solve this problem, related scholars introduced the annealing algorithm to optimize the probability density29. Although the number of particles in the likelihood region is increased, the loop process introduced by the annealing optimization also greatly increases the computational load and limits the practical application of PF. Another idea of IDF selection is to introduce measurement, i.e., using existing filtering methods to make the a priori distribution and the likelihood function have a larger overlap area through the measurement so as to better match the posterior distribution. According to different Bayesian integral approximation strategies, commonly used improved PF includes extended particle filter (EPF)30 based on EKF, unscented particle filter (UPF) based on Unscented Kalman filter (UKF)31, and cubature particle filter (CPF) based on cubature Kalman filter (CKF)32. Since the use of existing filtering as IDF can greatly improve the calculation performance of PF under the condition of appropriately increasing the amount of calculation, this method is more and more favored by researchers and has achieved considerable development. Liu and Hu have obtained a strong tracking particle filter (STPF) based on the improved EKF, which was used to compensate the model error and apply it to fault detection34. Zhang uses the truncated adaptive theory to generate the proposed distribution under the CPF framework to optimize the accuracy of the algorithm in strongly nonlinear systems35. Havangi uses a particle swarm optimization algorithm to improve UKF, adjust the position and velocity of the particle swarm to improve the accuracy of the posterior probability of particle simulation36.The state equation is the earliest and most extensive choice of IDF because of its relative simplicity and easy accessibility18. In fact, for more complex multi-source systems, the posterior distribution is often affected by multiple factors, which is difficult to approximate it with a single distribution as "irregular". The definition of PD in Gaussian distribution limits the superiority of PF in dealing with general nonlinear non-Gaussian systems. At the same time, various studies focus on the approximation of the real model of the system and few literatures pay a attention to suppressing the interference of various noises of the sensor itself to the PD.However, the various studies mentioned above all use different methods to improve the KF to produce PD to approximate the posterior probability density within the PF framework. Although its accuracy is continuously improved through the combination of various theories, the theoretical basis of these methods are based on the central limit theorem and limited to the Gaussian distribution of L2 norm estimation. The fusion of the L1 norm only revolves around its sparsity and fails to make full use of other properties of the L1 normIn this paper, for the above-mentioned deficiencies and limitations the Huber M estimation based on the third-order cubature law is applied to generate PD in PF. After the cubature rule is applied to transfer the system states, the algorithm combines the L2 norm and L1 norm using the Huber function for a better approximation of the posterior probability distribution. The posterior probability distribution of the system is redefined through their corresponding Gaussian distribution and Laplace distribution. The proposed idea also can make full use of the robustness of the L1 norm to suppress the interference of observation noise to PD. In order to further deal with the sample impoverishment problem, the algorithm uses the fission method to derive the high weight particles before resampling, and uses the offspring particles to cover the low weight particles in the parent to reconstruct the particle swarm weight. Finally, the robust cubature fission particle filter is formed. The vehicle experiments of multi-source cooperative navigation show that RCFPF performs better in accuracy and robustness than EKF, STPF, and CPF. It provides a new idea for optimizing PD, alleviating particle degradation and sample impoverishment, and also provides a new algorithm for data fusion of multi-source cooperative navigation.The rest of the paper is organized as follows: the second part introduces the cooperative navigation system, mainly the measurement of each subsystem that needs data fusion. The third part introduces the specific methods of the algorithm, including the process of classical PF, the construction of IDF and the specific process of particle swarm weight reconstruction. In the fourth part, the vehicle experiment of cooperative navigation is carried out, then the results comparison and analysis are given. The fifth part summarizes the paper and gives the conclusion.a and b (receiver a and receiver b) in VANET as an example, and its algorithm and conclusion can be extended to ad hoc networks with more vehicles7. In this article, the cooperative positioning system uses the pseudo range and Doppler frequency shift acquired by the receiver, the relative distance measured by Ultra Wide Band(UWB) to realize it. The workflow of relative position sensing of cooperative navigation is shown as follows: vehicle a obtains additional information through UWB communication, i.e., the pseudorange and Doppler frequency shift of vehicle b, makes double difference in combination with local pseudorange and Doppler frequency shift, measures the distance between two vehicles with UWB, and performs data fusion of the navigation system with the help of RCFPF to obtain the final cooperative navigation positioning solution. In order to fully demonstrate the algorithm process, we first introduces the fusion measurements required by each navigation system. Second, the system equations of cooperative navigation are given, and the data fusion is carried out in its framework.Vehicle cooperative navigation is generally realized on the platform of vehicle ad hoc networks (VANET). In order to simplify the system and facilitate interpretation, this paper takes the working logic between vehicles t as shown in Eq.  to satellite e and satellite j, respectively. a and receiver b.The single difference definition of the real distance between the receiver and the satellite is given by Then the double differenced true range g to Eq. , as showg to Eq. :5\documeSubstituting Eq.  into Eq.37.It should be noted that since the positioning error of Global Positioning System (GPS) is generally within tens of meters, which is almost negligible compared with the distance from the navigation satellite to the earth's surface, the unit line-of-sight vector between the receiver and satellite can be obtained by using the approximate receiver position and the known satellite ephemerise by receiver a at time t is shown in Eq.  provided by Joseph et al38. The experimental results by Shenshowed that the probability density of the UWB ranging error can be assumed to obey the zero-mean Gaussian distribution with variance of 0.3 a and b, respectively; UWB takes the half of the product of signal propagation time and propagation speed as the relative distance, and uses at least three reference stations with known coordinates to obtain the three-dimensional coordinates of the target point through resection. Under normal circumstances, the vehicles run on the same road and the driving track is regular, this paper assumes that there are no obstacles between UWB transceivers. In Shen's workThe state equation of the cooperative navigation system is defined as:The measurement equation of the system is showed as the Eq. :12\docum of Eqs. ; \documeIn order to obtain the covariance matrix of observation noise, we first need to define the observation error variance of pseudo range n in Eq. :14\docum39. In the state prediction stage, the algorithm predicts the particle state through the kinematic model of the system. Liu et al. have proved that 40. However, in the actual algorithm implementation, because it is difficult to realize, a suboptimal approximation N is the number of particles and Filtering can fuse a variety of measurement data, and use these data to finally obtain the estimated target parameters. Its purpose is to estimate, but the process can be regarded as the fusion of multiple measurement data. The particle filter uses weighted particles to perform Bayesian estimation based on Monte Carlo idea. Under the assumption of the first-order Markov process, the algorithm logic is divided into two stages, namely state prediction and time updateConsidering the problem of particle degradation, resampling technology is used in classical particle filters. First, the effective particle number The corresponding threshold 41. For further explanation, we take CKF as IDF as an example. CKF is an L2 norm estimation based on Gaussian distribution. Although the cubature law compensates some errors from the perspective of system nonlinear error in the generation stage of PD. The L2 idea of minimum variance will square the error in the calculation process and amplify the accuracy loss caused by outliers. Therefore, the measurement outliers far away from the sample population have an obvious impact on the residuals squares sum of the sample population, resulting in the decline or even divergence of the overall estimation accuracy of CPF. Different from L2 estimation, the error probability distribution of L1 estimation belongs to Laplace distribution, which has more significant "thick tail" characteristics44, when the variance of Laplace distribution is equal to that of Gaussian distribution, Laplace distribution has a higher probability density near and far from the mean than Gaussian distribution. So it is more suitable for the likelihood description of significant noise. At the same time, the error growth rate of L1 norm estimation is "linear" and significantly slower than the quadratic growth of L2 norm error, which makes L1 estimation have better suppression effect and stronger robustness against colored noise and outliers.As mentioned earlier, IDF has a crucial impact on the performance of a particle filter. Li et al. has proved that the measurement with significant noise will produce a less precise PD when introduced into the IDF, causing interference to the simulation of a posteriori distributionIn this paper, we first fuse L2 and L1 estimation based on Huber function in the more advanced CKF algorithm for nonlinear systems, and combine Gaussian distribution and Laplace distribution to better approximate the posterior probability of the system under the cubature law; secondly, the measurement update of CKF is transformed into linear regression, and solved by Huber M estimation. The M estimation uses Huber weight function as its weight function, which can give full play to the robustness of L1. So the algorithm will reduce the weight of the disturbed measurement, truncate and average the filtering innovation to achieve robustness. n is the state dimension; n-dimensional space on each axis; CKF approximates the probability density function by weighting the cubature points. In order to realize function transfer, the algorithm first needs to generate a cubature point set with equal weight according to the cubature criterion:First, the cubature point Second, based on the system dynamics model, the function is transferred through the state cubature points, and the approximation Finally, the system priori state is obtained by equal-weighted fusion of the state cubature point information, and the priori error covariance matrix is estimated:Measurement update needs to use In order to obtain the measurement likelihood description of each cubature point, it is necessary to use the new cubature point Second, perform equal-weighted fusion estimation on the measured cubature points to obtain the measured estimated value Finally, the measurement variance matrix After the measurement update is completed, a linear regression equation needs to be constructed, and the measurement update is further transformed into solving the linear regression problem.Using state prediction and measurements combined with observation equation to construct linear regression equation. The relationship between true state and valuation can be expressed as:The measurement Eq.  can be aning Eq.  and Eq. Each matrix variable is specifically defined as:M is the total dimension of the state and measurements, and Huber M estimation is used to solve the above linear regression. Define loss function:The Huber function as Function Defining Using the above-mentioned matrices to simplify Eq. , we obtaDefining olve Eq. , a posteThe corresponding a posteriori error covariance matrix Resampling is a method to eliminate the low weight particles to avoid the waste of computing resources caused by particle degradation. At the same time, by selecting and copying the high weight particles, resampling can maintain the confidence level of the particle swarm while keeping the total number of particles unchanged. However, this will destroy the diversity of particles. After several iterations, the algorithm will approximate the posterior distribution by only a few high weight particles in the initial particle swarm. The most extreme case is that the offspring particles are the offspring of a "most reliable" particle in the primary generation, i.e., the sample impoverishment. The simplest way to avoid sample impoverishment is to increase the number of particles, but this method is not efficient and will greatly increase the computational load.45. First, when the number of effective particles is less than the threshold, the particle swarm is arranged in the order of weight from small to large. Second, fission: take the current particle as the parent, select the first In order to further alleviate the sample impoverishment and minimize the increased computational load, fission technology is adopted to derive particles before resampling7. In order to obtain the original data of the algorithm, the relative distance between vehicles was obtained by UWB based on MSSI. Meanwhile, in order to obtain the reference truth value, the Leica GS10 receiver and the Novatel INS-LCI system integrating GNSS-INS were equipped on vehicles a and b, respectively, and the real time kinematic (RTK) positioning of them was implemented to obtained high-precision position estimations, which was used as the truth value to evaluate the algorithm performance. The pseudorange, Doppler frequency shift obtained by satellite receiver and UWB range were used to directly execute the proposed algorithm. Figure In order to verify the performance of the RCFPF algorithm, compare and analyze its impact on the relative position perception of cooperative navigation, the data used in the experiment were consistent with that in the experiment by ShenThe data time in the experiment was about 9 min. During this time, the two vehicles transmitted measurement information through UWB communication. The data sampling rate of GPS was 1 Hz. At the same time, the Differential Global Position System (DGPS) correction was executed through the GPS observations by a reference station of Nottingham Institute of geospatial research. In order to ensure a wide field of vision in the sky during data acquisition and ensure that the receiver can observe as many satellites as possible, the acquisition location is Clifton Boulevard between Derby road and Loughborough Road Road, Nottingham, UK. In the experiment, the initial relative position of the two vehicles is determined by GPS single point positioning, and the initial relative speed is set to zero. The initial state covariance matrix is set to  the Eq. . The expFigure As shown in Figs. ith diagonal element of the covariance matrix In order to compare the performance of four data fusion algorithms in more detail, and simultaneously prove the robustness and accuracy of RCFPF. According to the filter estimation error In order to further express the improvement of algorithm ACC and PRE of the four algorithms, the percentage of RCFPF improvement over EKF, CPF and STPF in the three standards is obtained as follows.Utilizing It can be seen from Table Based on cooperative navigation as a platform, this paper proposes a robust cubature fission particle filter for multi-source data fusion. The algorithm can fit the a posteriori probability density of the system more flexibly through the Gaussian distribution and Laplace distribution. At the same time, it can use the theoretical advantages of the fusion distribution itself to suppress the interference of the observation noise on the proposed distribution without adding additional computational load. The improved importance density function combines the advantages of L2 and L1 norm estimation. Then the resampling combines particle fission smoothing weights to alleviate particle degradation and sample impoverishment. The vehicle experiment of multi-source cooperative navigation shows that compared with EKF, RCFPF improves RMS, Accuracy, and Precision by 26.12%, 30.67%, and 18.46%; compared with CPF, it increases by 45.68%, 27.22%, and 56.85% in RMS, Accuracy, and Precision, respectively, compared with STPF, the RMS, Accuracy, and Precision are increased by 39.56.%, 29.76%, and 47.85% respectively, which achieves the theoretical optimization effect and further improves the accuracy and robustness of the particle filter. The experimental results confirm the superiority of the proposed algorithm, provide a new system state estimation strategy for the multi-source cooperative position, and also provide a way of thinking for the further improvement of the particle filter performance.The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.