Restoring the correct or realistic color of a cultural heritage object is a crucial problem for imaging techniques. Digital images often have undesired color casts due to adverse effects caused by unstable illuminant conditions, vignetting, and color changes due to camera settings. In this work, we present an improved color correction method for color cast images that makes the color appear more realistic. It is based on a computational model of the human visual system that perceives objects by color constancy theory; it realizes illumination non-uniformity compensation and chromaticity correction for color cast images by taking into account the color stability of some pigments. This approach has been used to correct the color in Cave 465 of the Mogao Grottoes. The experimental results demonstrate that the proposed method is able to “adaptively correct” color cast images with widely varying lighting conditions and improve the consistency efficaciously. It can achieve improved consistency in the mean CIEDE2000 color difference compared with the images before correction. This colorimetric correction methodology is sufficiently accurate in color correction implementation for cast images of murals captured in the early years. The human visual perception system has the ability to match objects’ colors in scenes taken under ambient lighting conditions; the color appears to be approximately constant to human observers. A scientific theory can be defined as color constancy; some explanation of color constancy has been confirmed by observation or experiment , 2019. IColor-accurate image archives of cultural heritage items have played a vital role in their preservation, and scientific studies have been carried out during restoration and renovation. Color constancy is a complex problem for cultural heritage digitization because the image color depends on the spectral reflectance function of the object and the spectral distribution function of the incident light, both of which are generally unknown. Therefore, color constancy became the problem of removing the color of the light that illuminated a scene . PreciseThanks to advancements in technology, such as high-resolution imaging , hyperspThe discussion above highlights several issues associated with color-accurate image archives. One prime goal is to capture the colors of cultural heritage objects accurately. A high-resolution panoramic image was stitched together from small overlapping pictures captured along the customed rail tracks. Color accuracy is affected by illumination conditions, lighting uniformity, geometry conditions, white balance, and camera settings. Specifically, the following issues are addressed:(a)The color appearance of the object shifts depending on the lighting conditions present when the image is captured and the object’s intrinsic properties, i.e., color is an unstable visual feature .(b)The imaging system setup is inadequate because of limited space or non-optimum lighting conditions, which can cause spatial non-uniformity in the images.(c)The lens vignetting or light falloff with spherical aberration could result in an image that is brighter in the middle and darker around the edges because the stronger the refracted light, the more significantly the imaging signal will be reduced; this effect is especially prominent for wide-angle lenses. In most cases, the illumination is brightest in the middle of the image, with a steady decrease toward the edges .(d)Most color constancy algorithms assume that the incident illumination remains constant across a scene, but this assumption is very often not valid for real images . Illumin(e)White balancing is widely applied to avoid color distortion caused by illumination changes. In general, estimates of illumination chromaticity are from space-average chromaticity or the brightest patches across scenes ; it can To address these issues and concerns, the specific objective of this study is to develop an image correction solution that employs a computational model of the human visual system to make the color appear more realistic. We specifically focus on chromatic transfer for color cast images using the color constancy theory proposed in this study. To solve the illumination non-uniformity problem, we compensated the non-uniform illumination in the scene using homomorphic filtering to eliminate illumination variations. The superior performance of this proposed method is evaluated and compared with existing methods in cave 465 of the Mogao Grottoes. The key contribution of this work is the solution it provides to “adaptively correct” color cast images with widely varying lighting conditions, and it improves color consistency efficaciously. It is hoped that this research will contribute to a deeper understanding of cultural heritage digitization.The methodology for producing a color correction of color cast image is diagrammed in Path 1 began with the mural image acquisition system and included image denoising, image enhancement, geometric distortion correction, and image stitching. Next, the image was converted to hue, saturation, and value (HSV) color space from red, green, and blue (RGB) space, which was defined for the international commission on illumination (CIE) 1931 standard observer and standard illuminant D65 to resemble how humans tend to perceive color. Then, the non-uniform illumination was compensated using low-pass filtering in the brightness  channel . It should be noted that the separability of the scene’s illumination and reflection components were processed in the image frequency domain.In Path 2, the palette was defined using analytical techniques and documentation by conservation scientists. Samples with different particle sizes and concentrations were prepared for the palette following the essential techniques of Chinese mural painting, and the chromaticity and spectral properties were measured to identify the pigments . Image masks were made for pigment segments of murals that were corrected. The HSV color coordinates were transformed into a pigment map mask using chromaticity characteristics. The size of the image mask is a compromise between minimizing the range of colors and maximizing the accuracy of the pigment map (Section “Color cast correction based on color constancy”). Moreover, the selected pigments must result in a colorimetric match for the specific illuminant and observer.An assumption was that the concentration of the colorant was constant throughout the masks. The change in the hue component channel of the HSV color space between the mask area and the reference pigment was calculated next. These differences were converted to CIEDE2000 color differences. Pigments were selected from the optical database resulting in the closest chromaticity match for the specific illuminant and observer. The color cast correction model was constructed to complete the color compensation of relevant pigment colors. Finally, all the colors in the mask area were translated into realistic colors.The Mogao Grottoes is a world cultural heritage site located in northwest China that is famous for the exquisite murals and Buddhist sculptures kept inside the caves. The Mogao Grottoes have 735 caves, more than 45,000 square meters of murals, and 2,415 colored sculptures of different sizes. They provide abundant vivid materials depicting various aspects of medieval life, such as farming, textiles, war, architecture, marriage, funerals, daily dress, arts, and commercial activities in ancient China.The approach in this paper is used to correct the mural color of the three-paved double-body mandala on the south wall of Cave 465, Mogao Grottoes, Dunhuang, Yuan Dynasty [1271–1368 anno domini (AD)], which is the earliest and most complete Tibetan-style mural outside of the Tibetan area, with a size of 13.13 m by 5.3 m, as shown in The developed image acquisition system included a commercial trichromatic camera (Canon EOS 1D X Mark III) with 14-bit digitization and 21 MP pixels. The lens was a Canon EF 50 mm USM. Two Elinchrom Ranger Free Lites mounted with translucent diffuse reflectors were arranged at an angle of approximately 45° to the capturing area of the murals. The imaging plane of the digital camera was set to be approximately parallel to the sample placement plane. Ten percent of the illumination deviation falls within the mean distribution. The camera was mounted on a rail track slider to capture images horizontally; each image corresponded to 0.15 square meters. After white balance, color management, and geometric distortion correction, 578 overlapped adjacent pictures were digitally stitched into a large image with a resolution of 150 dpi.2.A database of chromaticity and spectral reflectance for the mural palette began with the image cast correction of Cave 465, Mogao Grottoes. The palette was approximated using several color pigments . The redTo adaptively and accurately correct the mural, some reference pigments commonly used in the mural are essential. The reference pigment selection is significant in at least two major aspects of consideration, chemical and chromaticity properties.Of the many colorants in the mural, red earth, lazurite, mineral blue, mineral green, limestone, and calcite, are all capable of maintaining the same color in chroma with time. However, ochre and ultramarine are faded or darkened; some pigments, such as miniumite, lithargyrum, massicot, and white lead, have poor chemical stability; hence, their color would change under ultraviolet light, high temperatures, and humidity. These changes have dramatically influenced the mural’s color appearance, because these fading pigments cause the colors in the faces and skin to change from pink to brown. Some palette pigments are summarized in Dry pigments, such as vermilion, orpiment, litharge, cinnabar, azurite, ultramarine, and mineral green were dispersed in animal-derived gelatin-producing paints. Some pigments have many particle sizes, such as the minerals green, mineral blue, ultramarine, and cinnabar. Each pigment was mixed with titanium dioxide to achieve a different concentration of tints and applied to a smooth white substrate at a thickness resulting in opacity. For each pigment, there were at least ten mixture patches with white paint at weight concentrations of 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, and 100%. Additionally, tint ladders were created following the essential techniques of Chinese mural painting.The spectral reflectance and chromaticity of all the patches were measured by an X-Rite Ci64 UV portable integrating sphere spectrophotometer from 400 to 700 nm at intervals of 10 nm over the wavelength range. Measuring the specular component included (SCI) would capture accurate color data from the sample and negate the effect of surface appearance to measure only color. It makes little or no difference if the patches are mirror-like or matte in appearance. The measurement aperture of the spectrophotometer was 3.5 mm in diameter, and colors within the aperture were averaged spatially.In terms of chromaticity, an image observed by people is a visual perception of color stimuli formed by light irradiated on the object’s surface; the light is typically absorbed predominantly at some wavelengths and reflects or transmits light at other wavelengths; it is well-known as the illumination-reflectance model. That is, images captured in complex scenes can be highly degraded due to unstable lighting conditions. Illumination distributions are typically very slow across an image compared to reflectance, which can change quite abruptly at object edges ; this diThe HSV  color model is normally used because of its similarities to how humans tend to perceive color; the brightness is roughly analogous to the stimulus after illumination irradiation and object reflection. We, therefore, corrected colors by estimating the scene’s overall illumination in the HSV brightness channel.The first step in this process is to convert the RGB image into HSV color space, and the brightness channel is used to separate image luminance from color information. In general, the light stimulus can be regarded as the product of the illumination of the scene and the reflectance of the objects; i.e.,log domain.where I is the light stimulus at each point , L is the scene illumination resulting from the lighting conditions at the time of image capture, and R is the reflectance arising from the properties of the scene objects themselves. In homomorphic filtering, we first transform the multiplicative components into additive components by moving to the The Fourier transform is then used on both sides of the upper expression;orL and FR are the Fourier transforms of ln) and ln), respectively. F is the Fourier transform of the image being filtered.where FAs previously stated, with the low frequencies of the Fourier transform of an image with illumination and the high frequencies with reflectance, a separation can be gained over the illumination and reflectance components with a homomorphic filter. This control requires the specification of a low-pass filter H in the frequency domain to extract the low-frequency illumination component while preserving the high-frequency reflectance component. Then, we process F using a Gaussian low-pass filter function H fromwhere G is the Fourier transform from the image that has been processed. The H filter used in this procedure is the Gaussian low-pass filter defined asD0 is the cutoff distance measured from the origin, which determines the bandwidth of the low-frequency band that will be filtered out. To obtain the actual results, the following formula must be returned to the spatial domain:where D is the distance from the origin of the center transform at point  in the frequency domain and The next step is to apply an exponential function to invert the log-transform at the beginning of the process and obtain the homomorphic filtered image g, which is denoted byFinally, the illumination compensation model can be expressed as follows.output is the corrected brightness image, and İ is the mean value of the brightness channel.where I is the brightness image in the HSV color space, IFinally, the processed image is converted back into the RGB color space.The human visual system exhibits some color constancy that can keep the color perception of a scene constant when the illumination changes. Since cameras do not intrinsically have this ability, white balancing is widely applied to avoid color distortion caused by illumination changes. However, white balancing is not enough to achieve a more accurate and professional color correction, resulting in a lack of consistency among lighting conditions. Color constancy, therefore, is a feasible approach to correcting the mural’s color in a way that uses pigments with stable optical properties in chromaticity. As indicated previously, the HSV color space does a substantially better job mimicking how humans interpret color than the standard RGB color space. Hue is the primary chromaticity property that allows distinguishing pigments of different colors. To address these color casts, the following steps are taken:The first step in this process is to convert an input image from the RGB color space into the HSV color space.A pixel-wise mask is then created based on the angular character in the hue coordinate system dividing an image into its constituent parts or pigment colors. That is, image masks are made of areas identified for hue-based color cast correction.In the follow-up phase, the mean and standard deviation (SD) of hue channels are computed for the cast color and reference color. For a mask image of pigments, formal definition of mean and SD are given by Eqs 11, 12:mask represents the pixel value in the hue channel of the color cast image,  denotes the coordinates of the image, Ḣmask represents the mean value of hue channels, σ represents the SD. M,N presents the size of the pixels in mask, respectively. Then, we scale the hue channel by the ratio determined by the respective SDs and add in the mean of the hue channel to obtain an image of the cast. The color correction model is defined aswhere Hcorr represents the corrected value in the Hue channel, Ḣpatches represents the mean hue value of reference patches, σmask and σpatches represent the SD of the target image and reference patches, respectively, and the reference patches are of the mineral green pigment in the pigments database. The illuminant chromaticity change is given by:where HAfter that, in order to estimate the scene’s illuminant chromaticity distribution, a two-order polynomial surface fitting model of illumination chromaticity change △H is performed based on the mask defines the list of pixels.poly represents the scene’s illuminant chromaticity distribution, a1, a2, …a6 represents the surface fitting parameters. In particular, the chromatic components are changed by moving their deviation distributions caused by illumination changes. The corrected image Hnew is obtained from the original image H by:where,  represents the pixel position of the image, respectively, Hpoly is the mean value of the polynomial image.where, ḢFinally, the channels are merged back together and converted back into the RGB color space from the HSV space.In this section, the proposed method is implemented and compared with the currently existing methods. We conducted experiments to confirm the effectiveness of the proposed method.In this experiment, the effectiveness of illumination compensation was demonstrated by using homomorphic filters. To verify the proposed approach, the performance of non-uniform compensation with a different method [including the Gaussian process  and a moi denotes the probability associated with brightness level. It can be concluded that the proposed method produces better illumination uniformity and contrast in the enhanced images.where i denotes brightness level and pAnother common problem raised in the previous section concerns removing color casts based on color constancy theory. It was feasible to correct the mural’s color in a way that uses pigments with stable optical chromaticity properties. Since the hue channel in HSV space is an intrinsic property of surfaces and remains approximately constant under variations in illumination, this should make image segmentation easier on the basis of quantities. A pixel-wise mask was then created based on the hue in HSV space, and the image was divided into its constituent parts or pigment colors.x-axis represents hue values and the y-axis represents the number of pixels. The histogram follows a multimodal distribution, and each peak represents the most common color of the pigments. The hue of the mural image is mainly composed of red, orange, cyan, blue, and purple. The mural image was then segmented into several different regions, each corresponding to one of the peaks. Despite considerable variations in brightness between the mineral green pigments in the original image, the result gave rise to one major region, as shown in The color cast correction based on color constancy requires that the reference pigment has color stability chromatic properties; i.e., the chromaticity value of the pigment does not change significantly due to the granularity and concentration. As indicated previously, the mineral green pigment meets the requirements .To verify the proposed method, 20 color regions containing mineral green were chosen for the evaluation of color correction, all areas were located in the surrounding images of the portrait in Cave 465, Mogao Grottoes, as shown in The results section consists of tables and figures. The proposed method was implemented and compared with currently existing methods. These include the principal component analysis (PCA) method , max-RGBBuilding on the color correction work applied to the mineral green color, the color of the light that illuminated a scene over the whole image was constructed, and the further correction was then achieved to calibrate the chromatic value of the entire image. As illustrated in To compare the performance of the color correction method. The above algorithm was tested on several mural images. Restoring the correct or realistic color of a cultural heritage object is a crucial problem for imaging techniques. Currently, most methods fail to consider the intrinsic properties of pigments. In this study, we propose a new method to improve the color correction accuracy for early mural digital images with color casts. The results illustrate that the tint consistency of the corrected image can be significantly improved. The illumination compensation based on homomorphic filtering could be an effective approach to move non-uniformity illumination. The color correction model based on color constancy theory can transform a cast image into a more realistic state. The factors that have contributed to the improvement in color correction accuracy are as follows.(1)The results from the studies imply that the spectral reflectance is an intrinsic property of objects independent of illumination; this should make image segmentation easier on the basis of chromaticity value and should not affect the overall luminance level of an image.(2)For conservators and conservation scientists of cultural heritage, assessing color accuracy in digital images is useless when lacking concern for some pigments with color stability, such as mineral green and carbon black. Our approach is to estimate the illuminate chromaticity when mineral green pigments with hue-value stability properties as a reference are available, the experimental results show that the corrected color is more realistic.(3)Our study demonstrates the effectiveness of this method under color constancy theory and supposing non-uniform illumination of the scene. Moreover, due to its scientific evidence, the result remains speculative and is suitable for color correction implementation.(4)The color appearance restoration of early mural images cannot be verified and, at best, is an informed approximation when first executed. We should not lose sight of the purpose of color correction—to provide an informed impression of how a mural may have looked.The main goal of the current study was to develop a color correction solution for cast images of murals in the Mogao Grottoes. This study firstly compensated the non-uniform illumination using the homomorphic filter in the brightness channel of HSV space. Then the color cast was removed from the mural image using color constancy based on some pigments with stable optical chromaticity properties. The experiment results demonstrate that the proposed method can significantly improve the consistency of the tint and the non-uniform illumination compensated in the image frequency domain; it can achieve improved consistency compared with before correction. The contribution of this study may assist in understanding how images looked when first created to extend our knowledge of cultural heritage. These results highlight the potential usefulness of image restoration and color simulation for conservators and conservation scientists of cultural heritage. Further studies need to be carried out to evaluate the color gamut of the mask area. This colorimetric correction methodology is sufficiently accurate in color correction for cast images of murals from the early years.https://zenodo.org/record/7013743#.YwYi9XbMI2w.The original contributions presented in this study are publicly available. This data can be found here: ZL: methodology, data collection and analysis, and writing. Y-XL: compile programs. G-AG: data collection and writing – reviewing. YK: data collection. BW: data analysis. J-XL: methodology, funding acquisition, and writing – reviewing. All authors contributed to the article and approved the submitted version.


Salter-Harris type VI physeal fracture is a rare injury. This case study aims to present a novel method for treating a rare entity of Salter-Harris type Salter-Harris VI physeal injury of the medial malleolus.A 6-year-old boy with Salter-Harris type VI physeal injury was successfully treated using the two-stage procedure. In the first stage, the patient was treated with intravenous antibiotics, a series of debridement and lavage followed by a skin graft that left a defect in the medial malleolus. In the second stage, an autogenous iliac crest apophyseal graft was transplanted to reconstruct the medial malleolus, and the ankle joint was stabilized by an external fixator. An additional anticipatory Langenskiold procedure was performed for the physeal bar resection. Although the complete radiological development of medial malleolus compared to the contralateral side was not evident at the last follow-up, the functional and cosmetic outcomes were satisfactory.The reconstruction of medial malleolus using an autologous iliac crest apophyseal graft and stabilization of the ankle joint with an external fixator is a novel reconstruction technique in treating Salter-Harris type VI physeal injury of the medial malleolus. This technique provides satisfactory functional and cosmetic outcomes in such a fracture pattern; however, a further clinical study using a larger sample size is warranted in order to find the definitive outcome of the technique. Physeal injury of the distal tibia is the second most common among all physeal injuries having a tremendous risk of premature growth arrest resulting in angular deformity and leg length discrepancy . InitialA case of a 6-year-old boy with an open Salter-Harris type VI physeal injury and a complete loss of medial malleolus was presented in this case report. The deformity was treated using the two-stage procedure. In the first stage, the patient was treated with intravenous antibiotics, a series of debridement and lavage followed by a skin graft that left a defect in the medial malleolus. In the second stage, an autogenous iliac crest apophyseal graft was transplanted to reconstruct the medial malleolus, and the ankle joint was stabilized by an external fixator. An additional anticipatory Langenskiold procedure was performed for the physeal bar resection. Finally, this patient achieved a satisfactory functional and cosmetic outcome. This case report was approved by the ethical review board of the author's institute, and written informed consent was obtained from the patient's legal guardian.A 6-year-old boy was brought to the orthopedic emergency department following a motor vehicle accident (MVA) with an injury to his right ankle. He complained of pain on the right side of his foot and difficulty in walking, with obvious deformity of the injured ankle. The clinical and radiological examination revealed the loss of overlying skin and an osteochondral fragment from the medial malleolus. The condition was diagnosed as the Salter-Harris type VI physeal injury of the medial malleolus of the right ankle.The patient has been treated with a two-stage procedure. In the first stage in emergency department, the wound was meticulously washed, debrided, and kept in a vacuum sealing drainage for 5 days. Intravenous antibiotics were administered as a part of the management of open fractures. The wound bed was prepared for granulation tissue formation. After the formation of healthy granulation tissue, the split-thickness skin graft was grafted over it with a cast for immobilization by injury surgeons. The boy was discharged from the hospital 7 days after skin grafting and requested follow-up at the outpatient clinic. However, the boy lost the follow-up. At 20 months after the primary surgery, the boy again appeared in the hospital complaining of significant ankle pain, deformity, and abnormal gait. On examination, there was an obvious varus deformity of the ankle joint. Radiographic examination revealed the medial malleolus defect with cavovarus deformity of the right ankle .In the second stage, the boy was treated with the reconstruction of the medial malleolus using an autogenous iliac crest apophyseal graft  by pediaThe external fixator was removed after 3 months when the radiological evidence of graft union was present , and thePhyseal injury is common in children that account for ~15–18% of all pediatric fractures . AnatomiPeterson and Jacobsen  describeIt is challenging to reconstruct Salter-Harris type VI physeal injuries in one stage because an open injury has a high risk of complications such as infection. To avoid such complications, we used a two-stage technique. In the first stage, only wound care was performed with intravenous administration of antibiotics, wound lavage, and debridement. The wound bed was prepared for the healthy granulation tissue to grow so that the skin graft could be possible later. Another issue is that even if the soft tissue condition is relatively good in the acute stage, it's challenging to perform a free flap and malleolar reconstruction simultaneously. To avoid such complications, reconstruction is usually performed in the second stage after the complete healing of the wound with no evidence of the infection.The age of the patient and degree of bony loss are crucial factors for developing deformity and growth arrest. For optimum outcomes, the surgeon should always consider these factors before planning the reconstructive surgery. Our patients' age was only 7.5 years at the time of reconstructive surgery, so the risk of post-operative deformity was almost inevitable. The medial malleolus was reconstructed with an iliac apophyseal autograft covering the medial aspect of the talus. The physeal line of the autograft was placed parallel to the physeal line of the distal tibia with the hope that the autograft would heal appropriately and allow for potential growth.In contrast to Abbo et al. , an exteAt the last follow-up, the age of the patient was 10.5 years. No length discrepancy in our case might be because of the early excision of the bony bar, which was <20% of the distal tibia physeal scope. Although the complete union was evident at the last follow-up, the medial malleolus did not develop as on the contralateral side. However, the external deformity was not apparent, and the patient had satisfactory cosmetic and functional outcomes. Epiphysiodesis is a valuable option to prevent further deformity.The reconstruction of medial malleolus using an autologous iliac crest apophyseal graft and stabilization of the ankle joint with an external fixator is a novel reconstruction technique in treating Salter-Harris type VI physeal injury of the medial malleolus. Our patient reported satisfactory functional and cosmetic outcomes at the final follow-up; however, a further clinical study using a larger sample size is warranted in order to find the definitive outcome of the technique.The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.The studies involving human participants were reviewed and approved by the Ethics Committee of Tongji Medical College, Huazhong University of Science and Technology (IORG No: IORG0003571). Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin. Written informed consent was obtained from the individual(s), and minor(s)' legal guardian/next of kin, for the publication of any potentially identifiable images or data included in this article.BS were involved in data collection and follow-up assessments. XT, SR, and HZ were responsible for literature search, study design, and finalized the manuscript. CZ drafted the manuscript. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


In order to solve the tracking accuracy problem of the redundant manipulator, a PI control method with Henry gas solubility optimization parameter regulator (PI-HGSO) is proposed in this paper. This method consists of the controller and the parameter regulator. The characteristic is that the position deviation of a manipulator is equivalent to a specific function; namely, the proportional-integral (PI) controller is used to adjust the deviation input. The error can be better corrected by the processing of the PI controller so that the inverse kinematics solution of the minimum error can be realized. At the same time, the parameter selection of PI controllers has always been a difficulty in controller design. To address the problem, Henry gas solubility optimization (HGSO) is selected as a parameter regulator to optimize the parameters and obtain the optimal controller, thereby achieving high-precision trajectory tracking. Experiments on 9-DOF redundant manipulator show that our method achieves competitive tracking accuracy in contrast with others. Meanwhile, the efficiency and accuracy of the PI controller are greatly guaranteed by using HGSO to automatically optimize controller parameters instead of making approximate adjustments through infinite manual trial and error. Therefore, the feasibility and competitive superiority of PI-HGSO is fully proved in trajectory planning of redundant manipulators. With the development of technology and industry, redundant manipulator that can achieve high flexibility and high precision is gradually developed to meet the needs of modern intelligent industry. Although the redundant manipulator can complete more complex tasks, it is more difficult to solve the problems of related kinematics planning and control. Trajectory planning of redundant manipulators includes planning a set of end trajectories and completing the movement of the end trajectory under the manipulator’s limited conditions of the tracking accuracy, joint velocity, and joint angle during the movement.Trajectory planning methods of the redundant manipulator are mainly divided into the geometric method, iterative method, and numerical method ,6,7,8,9.In this paper, a PI control method with an HGSO parameter regulator (PI-HGSO) is proposed to perform trajectory planning f a 9-DOF redundant manipulator. Proportional-integral (PI) controller is designed to set the optimum joint angles from the current to the desired end pose. Meanwhile, Henry gas solubility optimization (HGSO) is used to solve the appropriate controller parameters to achieve the trajectory planning of the redundant manipulator. PI controller ,24,25,26The proposed PI-HGSO method enables the designed PI controller to obtain accurate gains during the trajectory planning process for redundant manipulators, achieving a highly precise position tracking of the end-effector.The design can obtain PI controller parameters more quickly and efficiently by using the HGSO parameter regulator instead of the traditional empirical trial and error, improving the controller parameter tuning cycle.The PI controller is used to replace the traditional gain compensation system to achieve the end-effector position compensation of the deviation part efficiently. The design structure is simple, and the error is effectively suppressed. Besides, the feasibility of the design has been guaranteed since this paper has proved that PI-HGSO is asymptotically stable in the Cartesian deviation trajectory of the endpoint.The specific contributions of the PI control method with HGSO parameter regulator (PI-HGSO) for trajectory planning of 9-DOF redundant manipulator can be summarized as follows:The structure of the rest is organized as follows: the second part describes the PI controller with joint limits for trajectory planning; the third part describes the design of parameter regulator by HGSO; the fourth part completes the simulation and results of analysis under the 9-DOF redundant manipulator; the fifth part is conclusion and summary.In order to solve the inverse kinematics of the redundant manipulator, the algebraic method needs to calculate the joint angles through the pose deviation. The deviation calculation of the end-effector needs to satisfy a certain functional relationship, and this characteristic has a strong fit with the PI characteristic. Therefore, in this paper, the expected path planning of the end-effector is equivalent to a PI controller ,24,26. TTheorem 1.The stability of the equivalent PI controller method can be proved by the Lyapunov stability theorem ,25,32.We denote the end-Cartesian space deviation by We define the Lyapunov function as:Which is obtain In the inverse kinematics solution process of the redundant robot, the value that needs to be acted on the end-effector is mapped to the joint space through the Jacobian matrix. Whereupon, the velocity of joint space is obtained, which is expressed as:Minimum norm method with weighting coefficient ,33,34,35The Jacobian matrix with weighting coefficient is expressed as:To avoid approaching the joint limit, the objective optimization function For the redundant manipulator, its Jacobian matrix is a long square matrix, and the inverse cannot be calculated. So, the pseudo-inverse can be used. However, the pseudo-inverse of the Jacobian matrix ,36,37 wiThe Jacobian inverse with singularity robustness and weighting coefficient is expressed as:Thus, the inverse kinematics solution of using the weighted minimum norm method is expressed as:By combining the weighted Jacobian minimum norm and singularity robustness inverse, the joint velocity norm can be reduced by For the joint angle constraint problem and joint angle singularity problem, the Jacobian pseudoinverse with weight coefficient is applied to complete the transformation of joint angular velocity and joint angle. At the same time, in order to solve the problem of manual selection of PI controller parameters by experience, HGSO is selected to optimize PI parameters, ensuring the requirement of a small error value in the process of redundant manipulator trajectory planning. Thereupon, A PI control method with an HGSO parameter regulator is designed to perform trajectory planning of the 9-DOF redundant manipulator, and the specific scheme is presented in A novel physics-based algorithm named Henry gas solubility optimization (HGSO) was proposed by Fatma et al. in 2019 . FurtherThe Henry’s constant Step 1: InitializationThe HGSO is established on the basis of Henry’s law of gas solubility, and related mathematical modeling formulas and steps are presented as follows:At the beginning of the HGSO algorithm, Furthermore, gas Step 2: ClusteringThere are Step 3: Fitness evaluationAccording to the evaluation index function, the gas particles in each cluster Step 4: Update Henry’s coefficientThe updating formula of Henry’s constant can be described as:Step 5: Update gas solubility.The gas solubility Step 6: Update positionThe relevant formula of gas position update strategy is as follows:Step 7: Escape from local optimum and reset the position of the worst agentsIn order to avoid the algorithm falling into the local optimum, the total particles are sorted according to the fitness value and the number of the worst particles is defined. Then these worst particles are selected, and their positions are reset. The position reset strategy is shown in Formula (21), and the relevant formula of the worst particle number is denoted as:Traditionally, the parameter selection of the PI controller is mainly based on the experience and the trend of the result to adjust and choose the approximate value ,25,26. TThe sum of the deviations between the actual and the expected trajectory points, is designed as the evaluation index, and the formula of the evaluation function is expressed as:Step 1: InitializationThe main principles of PI control method based on HGSO parameter adjuster for trajectory planning are as follows: Firstly, PI parameter Initialize population particle size Step 2: The setting of PI controller parameters:The position information Step 3: Evaluation and update:Complete the evaluation of the current position Step 4: Update the worst particles:Select the poor particles according to Equation (21) by fitness Step 5: Repetition and iteration:Return to Step 2 and continue to complete the next loop until the iteration termination condition is met. The optimized parameters Algorithm 1 PI control method with HGSO parameter regulatorOutput: PI parameters and tracking trajectoryInitialization: Initialize HGSO parameters, whilet < the maximum number of iterations do Set the  Complete the evaluation of the current  for each search gas particles do Update the position information  for end Update Henry’s coefficient of each gas type clusters using Equation (19). Update solubility of each gas using Equation (20) Select the poor particles according to Equation (22) by  Reset the position of these worst particles by Equation (19) to Equation (21). t = t + 1while endreturnImport: trajectory pointsThe Algorithm 1 implementation pseudocode of the PI control method with HGSO parameter regulator for trajectory planning is as follows:In this paper, based on the model of the 9-DOF hyper-redundant manipulator, the feasibility and advantages of the proposed PI-HGSO are verified. The structure of the 9-DOF hyper-redundant manipulator is shown in In the experimental part, the linear trajectory and the curve trajectory are taken as examples to prove the effectiveness of PI-HGSO by simulation. The related parameters of algorithms are listed in The gain results of parameter optimization obtained by PI-HGSO are as follows: linear trajectory gain is The trajectory planning results of the 9-DOF manipulator through the linear trajectory and curve trajectory planning are presented by PI-HGSO in Furthermore, for each algorithm, 30 iterations of experiments were carried out, and corresponding data analysis values were obtained. As shown in Meanwhile, it is obtained that the average errors of PI-HGSO are In order to complete the practical test of the proposed algorithm in this paper, the experimental verification of the PI-HGSO was carried out with the 9-DOF hyper-redundant manipulator entity of our laboratory. The above simulation data of the elliptic curve was used for the experiment, and the experimental trajectory result is shown in In this paper, a PI control method with HGSO parameter regulator for trajectory planning of 9-DOF redundant manipulator is proposed. In this method, the PI controller was selected to optimize the motion deviation of the manipulator end-effector. Then, the Jacobian matrix with weight coefficient is used to limit the joint angles. Meanwhile, HGSO was used to adjust PI controller parameters, and appropriate parameters were obtained to minimize the trajectory planning error. Experimental results demonstrate that the proposed method can not only obtain a small error value but also, the fluctuation of the error of each point is smaller than other algorithms. Besides, PI-HGSO can greatly improve the adjustment time and parameter accuracy compared with the original manual trial with experience.However, there are some areas where this approach needs to be improved. For example, the iteration time of superior intelligent algorithms is limited, and the real-time performance after different trajectory changes cannot be realized. That could be the future research direction to realize the fast response of parameter adjustment.


Edge detection of ground objects is a typical task in the field of remote sensing and has advantages in accomplishing many complex ground object extraction tasks. Although recent mainstream edge detection methods based on deep learning have significant effects, these methods have a very high dependence on the quantity and quality of samples. Moreover, using datasets from other domains in detection tasks often leads to degraded network performance due to variations in the ground objects in different regions. If this problem can be solved to allow datasets from other domains to be reused, the number of labeled samples required in the new task domain can be reduced, thereby shortening the task cycle and reducing task costs. In this paper, we propose a weakly supervised domain adaptation method to address the high dependence of edge extraction networks on samples. The domain adaptation is performed on the edge level and the semantic level, which prevents deviations in the semantic features that are caused by the overgeneralization of edge features. Additionally, the effectiveness of our proposed domain adaptation module is verified. Finally, we demonstrate the superior edge extraction performance of our method in the SEGOS edge extraction network in contrast to other edge extraction methods. The extraction of ground objects from high-resolution remote sensing images plays an important role in many fields, such as urban planning and change detection ,2. In adTo reduce the need for the number of samples, domain adaptation methods reuse samples that are not in task domains by reducing the distribution shift between different task domains . In thesCurrently, the mainstream domain adaptation methods are adversarial learning ,13,14,15We used one extractor and two discriminators for adversarial learning. The extractor is used to detect edges and output an edge strength map (it describes the confidence of the predicted edge). Then, the first discriminator inputs the edge strength map to perform domain adaptation on the edge level. The second discriminator inputs the mean map of the ground truth and the edge strength map to perform domain adaptation on the semantics level. In addition, during training, we set different weights for each part, especially the dynamic parameters used inside the extractor. In the early stage of training, overfitting caused by too few samples in the source domain is prevented, and the training weight of the source domain is increased so that the network can fully learn enough edge features. In the later stage of training, the training weights of the source domain are reduced so that the source domain samples are fine-tuned to provide accurate semantic features. As a result, our method achieves its performance with few target domain samples compared with fully supervised edge extraction networks that use sufficient source domain samples.Domain adaptation is applied to edge detection for the first time.A weakly supervised domain adaptation method for high-resolution remote sensing object edge detection is proposed. This method performs domain adaptation at the edge level and the semantic level to reduce the difference between two domains in the edge detection network.The main contributions of this method are as follows:The rest of the paper is organized as follows. This section introduces edge detection and domain adaptation, respectively.Previous edge detection methods use low-level features to determine whether each pixel of an image belongs to a contour. Convolving the image with a local filter detects the pixel with the highest gradient magnitude in its local neighborhood as an edge. To extract discontinuous features that are generated by step edges, linear filters such as Canny  and SobeDue to the success of CNNs, many edge detection methods based on deep learning have been proposed. Liu, Cheng, et al.  combinedThe development of edge detection methods based on deep learning has brought more solutions to the information extraction of high-resolution remote sensing images. Wei et al.  used theDomain adaptation methods reuse the labeled samples in the source domain by aligning the distribution offset between the source domain data and the target domain data, which reduces the high dependence of deep learning methods on the number of labeled samples in the target domain.One of these ideas is adversarial learning ,13,14,15Another idea is self-training ,18,19,20Many methods apply domain adaptation to the high-resolution remote sensing image information extraction field. Song et al.  designedX; Y.Our method uses two datasets in different regions, in which the semantic edges of the same type of ground objects are labeled. The source domain dataset contains a large number of samples; the target domain dataset contains a small number of samples. In our method, Adversarial learning is performed between the edge detection module and the two adaptation modules, each of which is continuously enhanced. If the features generated by the edge detection module can confuse the identification of the adaptation module, the distribution of the features extracted from the source domain image and the target domain image is considered consistent.Each part of the network will be described in detail below. For the edge detection task of high-resolution remote sensing images, connectivity is also one of the important indicators to evaluate the quality of the edge. The edges of objects such as cities and roads are often widely distributed, so they should have a large receptive field size to retain detailed spatial information. Therefore, SEGOS , which iAfter inputting By considering that the distribution of the edge pixels and nonedge pixels is extremely unbalanced, the class-balanced cross-entropy loss function is usually used, and the influence of this unbalanced distribution on the network training process is reduced by introducing the edge scale parameter λ. The loss function formula is as follows:The edge detection network outputs an edge strength map to describe the confidence in predicting object boundaries. However, edge pixels are not independent, and cross-entropy loss does not take into account the continuity of the edges. Therefore, the mean-square error (MSE) loss is added to mitigate this effect:α and β are their weights. The loss function formula is as follows:Therefore, the edge detection loss is the weighted sum of the above loss functions, and The module consists of multiple convolutional layers whose number of channels, kernel, and stride can be adjusted according to the resolution of the remote sensing imagery. Except for the last layer, each convolutional layer is followed by a weakly ReLU , which fHowever, since the edge strength map focuses more on the edge information, the semantic information of the semantic edge is weakened. Therefore, only using the edge adaptation module will lead to more generalization of the transferred edge feature, which will increase confidence in the nontarget object edges and result in no significant changes. Therefore, we should also consider how to better transfer the semantic features.Considering that weakly supervised domain adaptation has a small number of target domain sample labels in contrast to unsupervised domain adaptation, the ground truth contains correct semantic information, so the edge strength map can be combined with the ground truth. First, their mean map is obtained and then input into the network with the same structure as the edge adaptation module; thus, more semantic features can be obtained during the adaptation process. The semantic distribution loss function can be expressed as:P and Y are input into the SA to obtain their respective semantic distribution loss functions To make the edge detection module and the adaptation module perform adversarial learning, Each module has its own weight, and there is also a dynamic parameter δ inside the edge detection module to adjust the role of the samples of the two domains in the network training process. The weight of the source domain samples is increased in the early stage of network training to prevent overfitting caused by too few samples in the target domain. At the end of network training, the weight of the source domain samples is reduced to prevent them from providing too much inaccurate semantic information, and the network is fine-tuned to detect more accurate semantic edges of the target domain.Thus, for ED, the loss function is:For EA, its loss function is:For SA, the loss function is:Based on our goal of adversarial learning, we optimize the following min–max criterion:The goal is to minimize edge detection loss while maximizing the probability that the target domain predictions are regarded as the source domain predictions.In this section, we present the experimental details and the results.The data sets selected in this paper are two sets of multi-category semantic edge datasets, both of which are data sets that were created in our research laboratory. The first set of data sets was produced through the GF2 PMS imagery in Yangyuan County, Zhangjiakou City, Hebei Province. Each image is 1000 pixels × 1000 pixels × 4 channels, for a total of 500 images. Edges of water, crops, fruit trees, forests, buildings, grass, and roads are labeled in the sample. The second set of data sets was produced through the GF2 PMS imagery in Jiashan County, Jiaxing City, Zhejiang Province. Each image is 1000 pixels × 1000 pixels × 4 channels, for a total of 309 images. The edges of the same aforementioned objects are labeled in the samples. A total of 103 samples were selected from each of the two sets of data for testing, and the remaining samples were used for training.As shown in We implemented our method using the PyTorch  deep leaOur method is compared with AdaptsegNet, AdvEnt, and ASS for domain adaptation and with BDCN, DexiNed, and SEGOS for edge detection. The compared methods use optimal parameters for training. For the domain adaptation methods, the loss is the same as the loss function used in our method. The ground truth of the target domain samples is not used in the baselines and unsupervised methods. For the edge detection methods, since they do not use multi-domain samples, they are used for training: only the source domain samples, only the target domain samples, and mixing the two domain samples.The evaluation is performed using the optimal dataset scale (ODS) and optimal image scale (OIS) proposed by HED , which hIn the following experiments, except for the last set of experiments, all samples in the Yangyuan training set were selected as the source domain samples, and 51 samples were selected from the Jiashan training set as the target domain samples to compare the extraction performance of the methods with a small number of target domain samples.The last set of experiments verifies the edge detection of our method by using different numbers of target domain samples in different transfer directions. Our method performs domain adaptations from Yangyuan County to Jiashan County and from Jiashan County to Yangyuan County. Additionally, SEGOS is trained only with samples from the target domain. All samples from the source domain will be used, and 25%, 50%, and 100% of the target domain samples will be used for training.The results of all experiments are predicted on the test set of the target domain.As shown in As seen in As seen in From the experimental results of SEGOS in This study aims to make domain adaptation more effective in edge extraction of high-resolution remote sensing images, thereby obtaining better edge extraction results using a small number of target domain samples. The effectiveness and advancement of this method are verified from multiple perspectives through four sets of experiments.As observed from the results  of the aWe compared our method with other domain adaptation methods in different edge detection networks. The results  show thaThe comparison results  of edge In summary, most of the current domain adaptation methods for semantic segmentation only perform domain adaptation at a single level. Some multi-level domain adaptation methods are also not suitable for semantic edges. The semantic edge contains low-level edge information and high-level semantic information, which easily makes the adaptive process biased and affects the training of edge extraction networks. Therefore, our method performs an adaptation process at both the edge level and the semantic level, using them together to constrain the transfer of semantic edges. At the semantic level, we further introduce ground truth to adapt the entire image so that the network is guided by more accurate semantic information. From the results, it can be seen that our method has an excellent performance in both regions with large domain differences and regions with small domain differences. The overall evaluation results are also higher than the current state-of-the-art methods in all aspects.However, it can be observed from the experimental results that even though adversarial learning is used to improve the edge extraction performance of the network, there are still some pixels lost in the result, resulting in broken lines. If these results are used as pseudo-labels, there is a clear difference from the manually labeled samples. Therefore, the self-training method using pseudo-labels is limited by the difference between the extracted edge and the manually labeled edge and cannot play an effective role. If pseudo-labels similar to human labels can be made by broken-line repairing methods, they will go one step further to provide effective semantic information for the target domain. However, for complex tasks, such as multi-category semantic edge detection, this is still more difficult, so further research into self-training learning is needed.In this paper, we apply the domain adaptation method to edge detection for the first time and reduce the new task domain’s high dependence on the number of samples through weakly supervised domain adaptation. First, by considering the distribution characteristics of ground objects in remote sensing images, we choose SEGOS as the edge detection network. By understanding that the UDA methods lack correct semantic information guidance in the edge detection, we add a small number of target domain samples to make full use of the ground truth of the target domain to provide the correct semantic information. Simultaneously, domain adaptation is performed on the edge level and the semantic level, and dynamic parameters are introduced to adjust the influence weights of the source domain samples on the network training in different training stages. Finally, we conduct sufficient experiments on high-resolution remote sensing data sets that were developed in our research laboratory. The effectiveness of our proposed domain adaptation module is verified. Furthermore, we demonstrate the superior edge extraction performance of our method in the SEGOS edge extraction network in contrast to other edge extraction methods. In addition, if we can further study how to repair the broken lines on the edges, the self-training learning will further improve the edge extraction effect using a small number of target domain samples.


Introduction Persistent left superior vena cava (PLSVC) is a rare vascular malformation, with several cases reported in the English literature. The diagnosis is made incidentally, during cardiovascular imaging or when a catheter is placed in the left jugular or subclavian vein. They are without associated hemodynamic alterations, except if they have left atrial drainage or an associated dilation of the coronary sinus. If necessary, long-term PSLVC catheterization with right atrial drainage is safe.Case Presentation We report the case of 40-year-old man, admitted for placement of totally implantable vascular access device (TIVAD) on the same day of his first chemotherapy. A disease localized to the right neck made it impossible to puncture on the right. During the puncture of the left internal jugular vein, the diagnosis of PLSVC was made. Postoperative investigations confirmed the diagnosis and showed the presence of the right superior vena cava to which it was connected by the left brachiocephalic vein. They also confirmed the drainage of PLSVC into the coronary sinus. In addition, they demonstrated the presence of an associated right aberrant subclavian artery of direct aortic origin. Chemotherapy was administered safely and the port was removed 9 months after insertion without any problem.Conclusion This is one of the rare cases reported in the English literature of PLSVC diagnosed during TIVAD insertion and the first to report an associated vascular malformation. We publish it to encourage physicians to think about this differential diagnosis and to carefully perform the appropriate investigations before using the port. Many physicians involved in the treatment of cancer patients, including surgeons, interventional radiologists, and other specialists, routinely use central venous access devices,The central venous catheter should be placed over the right ventricle in the superior vena cava (SVC). Due to the rapid flow at this level, thrombogenicity is minimized.Persistent left SVC (PLSVC) is a rare vascular variation that is identified by chance during central venous access procedures when the left internal jugular vein (IJV) is used,Here, we present a case of 40-year-old man, who was diagnosed with PLSVC and right aberrant subclavian artery diagnosed during portacath placement.A 40-year-old man presented 2 months ago for a lump on the right cervical level, whose ultrasound showed the presence of a right jugulocarotid lymphadenopathy. A biopsy was performed which revealed the presence of a mixed lymphocyte population and the presence of Reed–Sternberg cells, confirming the diagnosis of mixed Hodgkin's lymphoma. Computed tomography and positron emission tomography scan showed disease localized in his right cervical lymph nodes.Therefore, the patient was scheduled for a totally implantable vascular access device (TIVAD) placement. The patient's preoperative work-up was normal, and he was admitted to the operating room on the same day of his first cycle chemotherapy administration. Intraoperative ultrasound of the neck confirmed the presence of an enlarged right cervical lymph node of 5 cm, with compression of the IJV and consequently inability to puncture the vein. The decision was therefore made for a left catheterization.Local anesthesia is administered, followed by ultrasound-guided puncture of the left IJV, insertion of the guidewire, and control of its position in the SVC by fluoroscopy, according to our own institutional technique.The postoperative frontal chest X-ray preformed, demonstrating the unusual path of the catheter in the left hemithorax rather than the normal anatomical right side of the SVC, confirmed the presence of the left SVC .A computed tomography of the chest was performed to confirm the diagnosis and look for any other associated abnormalities. The following images confirm the presence of the catheter in the PLSVC in addition to the right vena cava in place .It also highlights the communication between the PLSVC and the right one through the left brachiocephalic vein , and coInterestingly, another anatomical variant was discovered in this patient; it was an aberrant right subclavian artery that arises directly from the aortic arch, passing posteriorly to the esophagus, compressing it slightly anteriorly and having no correlation with the trachea .Since the drainage of the PLSVC is confirmed in the coronary sinus and no dysrhythmia developed, the decision was made to start his first dose of chemotherapy, without the necessity of further investigations.The entire chemotherapy protocol was administered through the portacath without any problem. It was removed 9 months after its insertion.PLSVC is the most common thoracic venous anomaly, despite its rarity among all vascular malformations. In the general population, the prevalence varies from 0.2 to 3%. In people with congenital heart disease, the prevalence ranges from 1.3 to 11%.John Marshall first reported PLSVC in 1850,About 20% of the total return of venous blood from the left arm, left side of the head and neck is handled by PLSVC, and in 80 to 90% of cases, right atrial drainage is found,It is difficult to diagnose on the basis of medical history alone,The presence and thickness of the two SVCs, the presence of both azygos veins, and the anastomotic ramus were used to classify PLSVC.PLSVC is found incidentally in 75% of cases.A chest X-ray reveals an atypical catheter path in the left hemithorax.Thermodilution,Aside from misplacement in a PLSVC, improper placement in minor veins might create an anomalous position of the catheter tip near the left edge of the mediastinum. The left internal thoracic vein, the left superior intercostal vein, the left pericardiacophrenic,Simple ligation, provided that the right SVC is well-developed intracardiac conduit and anastomosis of the PLSVC to the right atrium or the pulmonary artery, can all be used to repair a PLSVC draining into the left atrium. Polytetrafluoroethylene grafts are also used as a venous replacement.When traditional SVC catheterization is not possible, long-term PSLVC catheterization is effective, especially after determining the drainage pattern,After reviewing the English literature, only 17 cases of PLSVC, diagnosed after insertion of a portacath, were reported between 2003 and 2021 . The maPLSVC is a rare congenital vascular malformation, detected incidentally in the majority of cases and can lead to morbidity and mortality during its puncture and use. We urge all physicians to keep this differential diagnosis in mind and carefully revise any available preoperative imaging that may help detecting any vascular anomalies such as PLSVC.We reported here the first case of association of vascular anomalies in patient where we discovered a PLSVC during the placement of TIVAD.


As a powerful feature extraction tool, a convolutional neural network (CNN) has strong adaptability for big data applications such as bearing fault diagnosis, whereas the classification performance is limited when the quality of raw signals is poor. In this paper, stochastic resonance (SR), which provides an advanced feature enhancement approach for weak signals with strong background noise, is introduced as a data pre-processing method for the CNN to improve its classification performance. First, a multiparameter adjusting bistable Duffing system that can achieve SR under large-parameter weak signals is introduced. A hybrid optimization algorithm (HOA) combining the genetic algorithm (GA) and the simulated annealing (SA) is proposed to adaptively obtain the optimized parameters and output signal-to-noise ratio (SNR) of the Duffing system. Therefore, the data optimization based on the multiparameter-adjusting SR of Duffing system can be realized. An SR-based mapping method is further proposed to convert the outputs of the Duffing system into grey images, which can be further processed by a normal CNN with batch normalization (BN) layers and dropout layers. After verifying the feasibility of the HOA in multiparameter optimization of the Duffing system, the bearing fault data set from the CWRU bearing data center was processed by the proposed fault enhancement classification and identification method. The research showed that the weak features of the bearing signals could be enhanced significantly through the adaptive multiparameter optimization of SR, and classification accuracies for 10 categories of bearing signals could achieve 100% and those for 20 categories could achieve more than 96.9%, which is better than other methods. The influences of the population number on the classification accuracies and calculation time were further studied, and the feature map and network visualization are presented. It was demonstrated that the proposed method can realize high-performance fault enhancement classification and identification. Modern mechanical equipment develops towards complexity and automation; hence, minor malfunctions of parts may bring serious chain reactions. Rolling bearings, as a type of important component of rotating machinery, are widely used in wind turbines, compressors, high-speed railways, and other modern mechanical equipment. Bearing failure may cause the failure of the whole equipment, resulting in significant economic loss and even casualties. Therefore, it is necessary to monitor the working condition of rolling bearings and diagnose bearing faults in time to prevent potential accidents, which can be realized by intelligent fault diagnosis methods widely studied in recent years ,2.The framework for an intelligent fault diagnosis method can be divided into three parts : signal In practical applications, the fault features in the vibration signals are always weak when the fault is in its early stage or the faulty part operates in a terrible working environment. The weak features are difficult to extract through a normal feature extraction method, thus limiting the weak fault diagnosis performance of the corresponding intelligent fault diagnosis methods. Therefore, a feature extraction method with the capability of weak-signal enhancement should be selected in these cases. The stochastic resonance (SR) -based methods are a type of weak feature extraction method that can obtain a higher signal-to-noise ratio (SNR) compared to other traditional feature extraction methods . StochasHowever, SR systems need to meet the small-parameter conditions   and simulated annealing (SA) was used to achieve stochastic resonance (SR) in this system and obtain the optimal parameters. To make the SR outputs able to be processed by the CNN for classification, a mapping method based on a noise intensity sequence is further proposed to convert the time-domain output of the Duffing system into an image that can be further processed in classification.The bistable Duffing system, which is a typical form of nonlinear system that can achieve SR, can be described as :(1)x¨(t)In Equation (1), the term of Previous research results show that due to the adiabatic approximation theory, the bistable Duffing system can only achieve SR under small-parameter conditions, i.e., the amplitude, frequency and noise intensity should be small ; howeverThe output signal-to-noise ratio (SNR) of the Duffing system can be regarded as an objective optimization function to decide whether the system achieves SR. The output SNR of the system is defined as:To achieve SR adaptively in a bistable Duffing system under an input signal with fixed parameters, an optimization algorithm is needed to obtain a group of appropriate system parameters  based on both the GA and the SA combining their advantages was utilized in this work for high-performance multiparameter optimization of SR. In the HOA, the Metropolis criterion of the SA was added in the parents’ selection of cross and mutate stages based on the GA framework. Hence, the HOA can get a better optimization solution through the capacity of the SA that jumps out of local optimization in a short time. As a result, we can get an acceptable SNR for the Duffing system in a short time using the HOA.According to previous analyses, the proposed HOA provides an effective approach to achieving multiparameter-adjusting SR in a Duffing system, thus improving the quality of the input raw signal, and enhancing its SNR. The relevant data optimization method is presented in this subsection.It is noted that the fourth-order Runge–Kutta algorithm is adopted in this work to solve the Duffing system. The HOA used in this paper adopts a binary encoding format, and each parameter consists of 15-bit binary numbers to guarantee sufficient resolution. Moreover, the value of In order to use roulette to select the crossed parents and mutation parents of the HOA see , a fitneWhen using the SR-based methods, engineers are required to have extensive experience, hence, it takes a lot of time and manpower to find out the characteristic frequency of the bearing fault vibration signal. This can be solved by using an intelligent diagnosis method proposed in this work that combines SR with a neural network classifier. For this purpose, the SR output of the Duffing system should be converted into a grey image, which can be further used for feature extraction and fault classification by the neural network. The process of the mapping method is as follows.First, Next, by inputting Hence, for each detected signal, a grey image can be obtained by adding a group of noises with different intensities into the input signal and then being processed by the adaptive multiparameter adjusting Duffing system. More detected signals can produce more grey images, which can be further used by the convolutional neural network (CNN) for feature extraction and fault diagnosis.In this section, basic knowledge of a CNN is briefly introduced. The network architecture used for fault classification was obtained by modifying the parameters of the conventional visual geometry group (VGG) net architecture to satisfy the resolution of grey images obtained by the proposed mapping method. It is noted that compared to the conventional VGG, the batch normalization (BN) modules are added in the convolution layer and the dropout modules are added in the full connected (FC) layer to enhance the generalization of the VGG in this work.The architecture of CNN is briefly introduced in this subsection. A CNN consists of some filter stage and one classification stage . The filThe convolutional layer convolves the input local region with kernel filters, and the following activation layer generates a feature map. The kernel that extracts the local features keeps the same in each filter, thus reducing the complication of CNN. The convolution process is described as follows:In the activation layer, a nonlinear activation function of Rectified Linear Unit (ReLU) is widely used to improve the expression ability of the whole network, which means the functions that can be expressed are more abundant. The ReLU can prevent the occurrence of overfitting by making the output of some neurons to be zero, thus resulting in the sparsity of networks.A BN layer is further designed to speed up training and convergence of the network and reduce the shift of internal covariance. The pooling layer generally adopts the max-pooling layer, which enhances the generalization of the model by reducing the parameters while retaining the main features.Moreover, the classification stage is composed of several FC layers. The FC layer is used for enhancing the generation of the model after convolution, and the number of neurons in the output layer denotes the types of bearing health conditions.The whole architecture of the CNN used in this work is shown in In the process of training, the number of iterations was set as 300, and an Adam optimizer was utilized to minimize the loss function, where the learning rate was set as The proposed method, which can be used for fault classification and fault identification in practical applications, is verified in this section. It is necessary to point out that the computer used for numerical simulations had a CPU of Intel(R) Core (TM) i5-10400 and RAM with 16.00 GB as its main configurations.For practical noisy fault signals, the characteristic frequency In this subsection, the advantage of the HOA is verified by processing a simulated signal with different injected noises. The signal was set as a pure harmonic signal with amplitude Before optimizations, the value of research , the valBesides, it is convenient to set a large population number to obtain a local optimization close to the global optimization with a large In this subsection, the vibration signals of the rotating bearing from the bearing data center of Case Western Reserve University (CWRU) were processed by the proposed method, thus verifying its feasibility in bearing fault data classification and fault diagnosis. The test rig is shown in In the test rig, the test bearings, which were deep groove ball bearings of type 6205-2RS JEM SKF, were used to support the motor shaft. The bearing details are listed in The bearing data used in the experiment was sampled at the end of the drive with a sampling frequency of 12,000 Hz. As the location of the fault relative to the bearing load area affected the vibration response of the whole motor system, the bearing data at 6 o’clock, 3 o’clock and 12 o’clock directions of the bearing load area were listed, respectively. In this work, the bearing data at 6 o’clock was used for verification.However, the bearing data set only contains one bearing signal of each fault type, which is not enough for data training. To obtain more fault signals to make the classification results more generalized, each bearing signal was expanded to 400 samples, as shown in The details of the datasets are shown in In this subsection, the samples of dataset A are taken as examples to be processed by adaptive optimization of SR. As we mentioned at the beginning of In the optimization processes, the scale transformation parameter was set as Hence, the The classification accuracies of the bearing signals are presented in this subsection. It is noted that the batch size has a significant influence on the coverage speed and classification result. In this work, the batch size was set as 32 to obtain the highest classification accuracy with a relatively high convergence speed. Through simulations, an accuracy of 100% can be obtained in However, an accuracy of 100% is difficult to achieve in practical engineering as the number of fault categories is far more than 10. To study the classification performance of the proposed method for more fault categories, the datasets A, B, C and D presented in In addition to the classification accuracies, the calculation time is another important index to evaluate the performance of a classification method. Both indexes are affected by the population number, which is studied in the subsection.The datasets shown in Generally, CNN is an efficient tool to extract features, but it is hard to understand how CNN processes grey images. In this subsection, the feature maps and networks are plotted for a better understanding of the powerful feature extraction and classification capacities of CNN.For dataset A, A novel intelligent fault diagnosis method combining the adaptive multiparameter stochastic resonance (SR) as a weak-fault feature extraction method and the normal convolutional neural network (CNN) as a fault classification method is proposed in this paper for bearing fault enhancement diagnosis. A multiparameter bistable Duffing system that can realize SR for large-parameter signals is introduced as a weak-fault feature extraction model, whose optimal output signal-to-noise ratio (SNR) can be achieved adaptively using a proposed hybrid optimization algorithm (HOA) combining the genetic algorithm (GA) and simulated annealing (SA) with high optimization speed. Therefore, poor-quality raw data can be pre-processed by the adaptive multiparameter-adjusting SR of the Duffing system. A mapping method that can convert the SR outputs of the Duffing system into grey images is proposed. The obtained grey images are further processed by a normal CNN with batch normalization (BN) and dropout layers, thus achieving fault enhancement classification. The research shows that a relatively high output SNR can be obtained from the adaptive multiparameter-adjusting Duffing system using the proposed HOA that has a better optimization performance compared to the GA. Therefore, weak features of raw data can be significantly enhanced. The bearing fault dataset from the CWRU bearing data center was pre-processed by the feature extraction method and was further processed by the proposed CNN model. Classification accuracies of 100% were achieved for 10 categories of bearing signals and those of more than 96.9% were achieved for 20 categories of bearing signals, which is better than other methods. The influences of the population number on the classification accuracies and calculation time were further studied, indicating that the proposed method can realize a rather good classification result in a short time. The feature map and network visualization are presented by t-SNE to show how the features were identified by the CNN. Hence, it was demonstrated that the proposed method can realize high-performance fault enhancement classification and identification. Moreover, to make full use of the advantage of SR in detecting weak signals, the abundant information of SR output, such as the frequency domain output, can be further combined with other appropriate intelligent methods for fault classification or prediction in future work.


The problem of nowcasting extreme weather events can be addressed by applying either numerical methods for the solution of dynamic model equations or data-driven artificial intelligence algorithms. Within this latter framework, the most used techniques rely on video prediction deep learning methods which take in input time series of radar reflectivity images to predict the next future sequence of reflectivity images, from which the predicted rainfall quantities are extrapolated. Differently from the previous works, the present paper proposes a deep learning method, exploiting videos of radar reflectivity frames as input and lightning data to realize a warning machine able to sound timely alarms of possible severe thunderstorm events. The problem is recast in a classification one in which the extreme events to be predicted are characterized by a an high level of precipitation and lightning density. From a technical viewpoint, the computational core of this approach is an ensemble learning method based on the recently introduced value-weighted skill scores for both transforming the probabilistic outcomes of the neural network into binary predictions and assessing the forecasting performance. Such value-weighted skill scores are particularly suitable for binary predictions performed over time since they take into account the time evolution of events and predictions paying attention to the value of the prediction for the forecaster. The result of this study is a warning machine validated against weather radar data recorded in the Liguria region, in Italy. Autumn events, when deep Atlantic troughs more easily enter the Mediterranean area and activate very moist and unstable flow lifted by the mountain range, may cause catastrophic flooding on these coastal areas, which are characterized by a high population density . Just as an example, the November 4th 2011 flood in Genoa caused six deaths and economic damage up to 100 million euros7). A common feature in these extreme events is the presence of a quasi-stationary convective system with a spatial extension of few kilometers12One of the most interesting problems in weather forecasting is the prediction of extreme rainfall events such as severe thunderstorms possibly leading to flash floods. This problem is very challenging especially when we consider areas characterized by a complex, steep orography close to a coastline, where intense precipitation can be enhanced by specific topographic features: this is the case for the example of Liguria, an Italian region located on the northwest Mediterranean Sea and characterized by the presence of mountains over 2000 m high at only a few kilometres away from the coastline. This specific morphology gives rise to several catchments with steep slopes and limited extension19.Medium and long range either deterministic or ensemble Numerical Weather Prediction (NWP) models still struggle to correctly predict both the intensity and the location of these events, which can be triggered and enhanced by very small-scale features. High resolution convection-permitting NWP models manage to partly return a more realistic description of the dynamics of severe thunderstorms. Many studies addressed the role played by different components or settings of NWP models in order to better describe severe convective systems over the Liguria area, such as model resolution, initial conditions, microphysics schemes or small-scale patterns of the sea surface temperaturenowcasting models, e.g. in order to feed automatic early warning systems, which may support meteorologists and hydrologists in providing accurate and reliable forecasts and thus reducing the consequences of these extreme events. These forecasting systems typically rely on two kinds of approaches. On the one hand, either stochastic or deterministic models are formulated utilizing partial differential equations in fluid dynamics, and numerical methods are implemented for their reduction, nesting hydrological models into meteorological ones22. On the other hand, more recent data-driven techniques take as input a time series of radar (and in case satellite) images belonging to a historical archive and provide as output a synthetic image representing the prediction of the radar signal at a subsequent time point; this approach can rely on some extrapolation technique, e.g. based on a storm-tracking system23 or a diffusive process in Fourier space24, or on deep learning networks36. Mixed techniques have been also proposed, blending NWP outputs with data-driven synthetic predictions37. The aim of these studies is to make time series prediction by exploiting image-based deep learning techniques, such as U-net26, Convolutional Long Short-Term Memory (ConvLSTM)28, improvements of ConvLSTM as Trajectory Gated Recurrent Unit (TrajGRU)34, and Generative Adversial Networks (GANs)36, which produce reflectivity images in the next future. From the predicted reflectivity images the rainfall quantity can be extrapolated but no indication of the presence of lightning can be provided. In our work, we focus on the forecasting of extreme thunderstorm events and therefore previous methods mentioned above do not directly apply to our problem. On the contrary, we present a novel method which recasts the problem into a classification one by using the lightning density as a fundamental feature for characterizing an extreme event. Towards this aim, we exploit a deep neural network, originally conceived for video classification, to predict the probability that an extreme event occurs. We use as input time series of multichannel radar images and we define the labels on the basis of a certain level of precipitation and lightning density. The deep-learning model combines a convolutional neural network (CNN) with a long short-term memory (LSTM) network39 in order to construct a long-term recurrent convolutional network (LRCN)40. The prediction assessment is performed by means of the recently introduced value-weighted skill scores41 which allows ranking prediction errors on the basis of their distribution along time, preferring to show up a warning well in advance of the actual occurrence of an event rather than not to show it at all. Finally, we exploit the iterative nature of the network training process to collect a set of predictions from which we select a subset of valuable ones on the basis of their value-weighted skill score. This procedure falls within the class of ensemble learning techniques. We remark that the term “ensemble” as used here refers to deep learning methods and not to the NWP algorithms. The main methodological novelties of this approach are the following. The prediction problem is reformulated into a binary classification one in which labels depend on both heavy rainfall conditions and lightning density;forecasting verification is performed by the use of value-weighted skill scores on the basis of an automatic ensemble strategy.Other works have been translated the forecasting problem into a binary prediction , but the focus was on moderate rain, i.e. when the rainfall is beyond a certain threshold, mainly However, the intrinsically limited predictability of convective systems requires the use of shorter-term The paper is organized as follows. In “42; such a representation is particularly useful for compositing reflectivity data measured by different radars over overlapping regions, returning a reflectivity field for the larger area covered by a radar network.Precipitation activity and locations of rain, showers, and thunderstorms are commonly monitored in real-time by polarimetric Doppler weather radars; return echoes from targets (such as hydrometeors) allow the measurement of the reflectivity field on different conical surfaces, one at each elevation angle of the radar; however, reflectivity values at a certain height can be interpolated to 2D maps, which are also known as Constant Altitude Plan Position Indicator (CAPPI) imagesIn our study CAPPI reflectivity fields measured by the Italian Radar Network within the Civil Protection Department are considered. CAPPI images, measured in dBZ, are sampled every 10 minutes at a spatial resolution of 43 combine radar rain estimates and rain gauge measurements with an hourly frequency and provide the amount of rainfall integrated over 1 hour  data and lightning data. MCM dataMCM data must contain at least 3 contiguous pixels exceeding 50 mm/h within the selected area;at least 10 lightning strikes must occur in a 10-minute time range in the area comprising 5 km around each one of the MCM pixels with over-threshold content.It is worth noticing that 50 mm/h is regarded as a threshold for heavy rain in the Liguria region; however, the first condition accounts for the fact that an over-threshold value associated to an isolated pixel may be associated to spurious non-meteorological echoes like, for instance, the passage of a plane. On the other hand, the second condition implies that the extreme events considered must always involve the occurrence of thunderstorms.The labeling process associates each CAPPI video to the concept of severe convective rainfall event, whose definition relies on the following two items:40 we propose the use of a Long-term recurrent convolutional network (LRCN) which combines a convolutional neural network (CNN) and a long short-term memory (LSTM) network to create spatio-temporal deep learning models46. In this application, the input is made of time series of 10 radar reflectivity images (representing a video 1.5 hours long) at the three CAPPI 2, CAPPI 3 and CAPPI 5 levels, which refer to 2 km, 3 km and 5 km ASL, respectively. Images have been resized to a The idea of this work is to address the prediction of extreme events in the short term as a radar image video classification problem. Following the work of47 is adopted as an activation function and the max pooling operation with size  and stride  is applied. We initialize all the convolutional weights by sampling from the scaled uniform distribution48. The three convolutional layers are characterized by 8, 16 and 32 kernels with size ,  and , respectively. The input are sequences of size , where T represents the number of frames in each movie, 128 and 256 correspond to the image size (in pixel) and 3 represents the three levels of CAPPI data. In all operations we take advantage of the “Timedistributed” layer, available in the Keras library49, which allows the in parallel training of the T convolutional flows. Figure 50: the dropout value is set to 0.5, meaning that In this work the CNN architecture of the LRCN model consists in three blocks, each one composed by a convolutional layer with stride , followed by a batch normalization layer to improve stability; the Rectified Linear Unit (ReLU) functionX to a probability outcome X is a video of CAPPI reflectivity images and X within the selected area . In the training process we consider an optimization problemOnce the architecture of the NN is set up, we can denote with During the iterative optimization process a set of deep neural networks The result of a binary classifier is usually evaluated by computing the confusion matrix, also known as the contingency table. Let us denote with 34; and the true skill statistic (TSS)51. The CSI varies from , while the TSS varies from A specific classical  skill score is given by a map 41. The aim of the value-weighted approach is to mitigate errors such as false positives that precede false negatives (the case of predictions well in advance) and false negatives which are preceded by true positives  as they have little impact on the prediction from the point of view of the forecaster. In fact, a value-weighted confusion matrix is defined asw isi, then T elements before T elements after i, then However, such metrics do not account for the distribution of predictions along time and are not able to provide a quantitative preference to those alarms that predict an event well in advance with respect to its actual occurrence, and to penalize predictions sounding delayed false alarms. To overcome these limitations, value-weighted confusion matrices have been introducedThe introduction of this value-weighted confusion matrix allows the construction of the associated value-weighted Critical Success Index wCSI and the value-weighted True Skill Statistic wTSS, respectively.N epochs of the training process of the deep neural network j computed from the training set. The procedure has been introduced in41, and it can be summarized in the following steps: j we select the classification threshold For each epoch Choose the subset of valuable predictions by selecting the predictors with a skill score higher than a given a quality level X, we have m indicates the median function. In the case where the number of zeros is equal to the number of ones, we assume We define the ensemble prediction as the median value of all the selected predictions. Given a new sample In the second step of the above scheme, the parameter 41, where the above procedure was introduced and (i)For each (ii)Select the optimal parameter As a result of this procedure, the estimated value of We consider an ensemble procedure to provide an automatic classifier from the probability outcomes provided by the deep NN. Consider the first mentclasspt{minimaWe show the pipeline diagram explaining the ensemble method in Fig. M times by randomizing the initial values of the weights, i.e. by training the deep neural network M times and we take the best ensemble prediction on the validation set. The best prediction is in the sense of the highest preferred skill score k-th random initialization, we define the optimal weights ask-th random initialization of the training process.In order to ensure statistical robustness of the entire ensemble procedure, we repeat it In the following we show the performance of the ensemble deep learning technique when the LRCN network is used for the problem of forecasting extreme rainfall events in Liguria.52 with learning rate equal to 0.001 and mini-batch size equal to 72. The class balanced cross-entropy defined in . It is worth noticing that the aim here is not the prediction of the exact location and intensity of a heavy rain event, but rather the probable occurrence of a severe thunderstorm over a reference area in the next hour.The realization of warning machines able to sound binary alarms along time is an intriguing issue in many areas of forecastingThe crucial point in our approach relies on the kind of evaluation metrics adopted. In fact, the TSS can be considered a good measure of performance in forecasting, since it is insensitive to the class-imbalance ratio. However, such a skill score, as all the ones computed on a classical quality-based confusion matrix, does not account for the temporal distribution of alarms. Therefore, we propose to focus on value-weighted skill scores, as the wTSS, which account for the distribution of the predictions over time while promoting predictions well in advance. We focused on the problem of forecasting extreme rainfall events on the Liguria region, and we showed that the performance of our ensemble technique in the case when wTSS is optimized, is significantly better than the performance when the model is trained to optimize a standard quality-based score.57, whose minimization in the training phase allows the automatic maximization of the corresponding skill scores. Possible future studies of this work concern (1) the investigation of other ensemble techniques as59, (2) the use of feature selection methods which allow individuating the most relevant subset of features extracted by CNN models as in60, (3) the use of dynamic graph modeling approaches to learn spatial-temporal representations in radar reflectivity videos61. Further, deep hashing methods62 could be used to exploit more information for the prediction, like the density and type of lightning (such as cloud-to-cloud and cloud-to-ground strikes).Next in line in our work will be the application of a class of score-driven loss functions