Developing a user interface (UI) suitable for headset environments is one of the challenges in the field of augmented reality (AR) technologies. This study proposes a hands-free UI for an AR headset that exploits facial gestures of the wearer to recognize user intentions. The facial gestures of the headset wearer are detected by a custom-designed sensor that detects skin deformation based on infrared diffusion characteristics of human skin. We designed a deep neural network classifier to determine the user’s intended gestures from skin-deformation data, which are exploited as user input commands for the proposed UI system. The proposed classifier is composed of a spatiotemporal autoencoder and deep embedded clustering algorithm, trained in an unsupervised manner. The UI device was embedded in a commercial AR headset, and several experiments were performed on the online sensor data to verify operation of the device. We achieved implementation of a hands-free UI for an AR headset with average accuracy of 95.4% user-command recognition, as determined through tests by participants. Augmented reality (AR) is one of the hottest issues in the information and communication technology (ICT) industry. In the field of AR, developing a user interface (UI) device that is optimized for AR headsets is one of the key challenges. This is because the headset environment, which differs greatly from a personal computer or mobile phone, makes it difficult to use conventional UIs . Pressing buttons on the side of the headset or holding remote controllers are the most typical solutions currently adopted for headset UIs. However, such UIs not only have limitations in supporting complex and dangerous operations, such as surgery or manual labor, but are also insufficiently convenient for general usage. Voice recognition techniques and physiological sensors, such as electrooculography (EOG), electroencephalography (EEG), or electromyography (EMG), are promising alternative means of providing a hands-free UI for AR headsets, however, optimal sensing devices and methods for implementation of headset UI have yet to be developed. To address this unsolved requirement for a hands-free natural user interface (NUI) for AR/VR devices, we propose a method to detect skin deformation using a custom-made sensor. The purpose of the proposed technology is to provide a hands-free UI system for an AR headset, which exploits the facial gestures of users as the input commands of the UI. The proposed technique facilitates the detection of facial skin movements in a noninvasive manner and supports hands-free interaction between the headset user and the AR system by mapping a detected facial gesture of a user to an input command.The custom-made sensor monitors the deformation of the facial skin of headset users and uses the facial-skin movement to recognize the facial gestures of the user. The skin deformation is detected by measuring spatially resolved diffuse reflectance (SRDR) ,2,3,4 ofTracking predefined feature points on the human face is a widely used method for facial-gesture detection . In thisSTAE is frequently exploited for studying video anomaly detection ,10,11,12Implementation of an IR sensor for skin-deformation detection;Design of a classifier neural network for spatiotemporal data processing;Conception and realization of a hands-free UI for AR headsets.The proposed network was trained using the sequential images of the NIR SRDR dataset collected from experimental volunteers. We installed the proposed UI device on an Epson BT-350 AR headset and then implemented an AR headset UI based on the proposed device. Several experiments were conducted for verification of the operation of the proposed UI. The contributions of our work can be summarized as follows:The remainder of this paper is organized as follows. In Many researchers in academia or industry have already developed advanced devices and solutions enabling user interaction with headset-type devices. Hand gesture detection or eye-gaze tracking technologies are some of most popular methods for this purpose. Using eye gaze to control an AR system was tried . Those rFacial gesture recognition technology is a great alternative solution for developing UI for headset device environments. The best-known method for the facial gesture recognition is facial image processing with an external camera. Liu et al. presented a boosted deep belief network (BDBN) for the facial expression recognition from facial pictures . Their nMcFarland et al. used an electroencephalogram (EEG) to sense brain activities and facial expressions by recording brain waves for brain–computer interfaces and control . This meAn optical-based sensor exploits infrared (IR) reflection or diffusion characteristics for recognition of facial gestures. Masai et al. integrated photo-reflective sensors on the inner surface of a glasses frame to recognize the user ’s facial expression . The arrDeformation of human skin can be detected by measuring changes in the SRDR of IR transmitted underneath the skin. “Diffuse reflectance” (DR) is the ratio of the amount of flux reradiated by diffuse reflection, and its intensity is affected by the microstructure of collagen fibers distributed under the skin. “Spatially resolved” (SR) implies using the radial distance from a light incident spot ,4,31. NiOne of our goals in this study was to implement a general-purpose UI device as reliable as a mouse  for the AR headset environment by utilizing facial gestures combined with head movements. Similar to the basis of a mouse UI, the proposed headset UI consists of input motions for moving a cursor and clicking a pointer on the display. By combining intentional facial wink gestures and user head movements, we could implement user input commands for an AR headset. An on-board gyro sensor detects rotational movement of a user’s head and the proposed IR sensor module determines the intention of the user to click pointers. The author’s design principle of the proposed UI is to make inputting the command-set of an AR headset as simple as using a mouse with personal computers. We only detected simple wink gestures, even though it is not difficult to detect subtle expressions of the user, because too many subtle gestures may increase the complexity of the user interface to the extent that it becomes inconvenient as a UI.2 at human pupil is categorized as a class 1 laser, which is considered to be safe under all conditions of normal use for NIR laser. The output power of our laser emitter is 0.4 mW, which is much smaller than the acceptable standard. Further, we have used the 10% transmittance filter in front of our emitter module to get ten times lesser output power from the emitter to guaranty the safety of users. The NIR illumination did not interfere with the user ’s visual field, and we limited the NIR dose to a safe level to prevent any potential harmful effects to the human eye.The proposed hands-free UI system included a sensor module that detected IR diffusion patterns from the sensor wearer ’s skin surface and a sensor data classifier based on a DNN structure. The NIR band has the deepest skin penetration depth of irradiation in the harmless light spectrum . This isAs illustrated in the previous section, the data input to the data classifier was in the form of NIR SRDRs images taken from human skin. The camera-captured SRDR shape may have person-to-person variations due to many hardly modeled factors. The proposed sensor data classifier had to detect the transition of the captured SRDR shape to recognize the user’s gesture. We utilized unsupervised learning methods for the neural network sensor data classifier. The unsupervised learning methods provided us an easy way to calibrate when we had to cope with person-to-person variation, because we could avoid data annotation tasks for generation of the training dataset.Unsupervised learning for extracting features from the dataset using an autoencoder;Clustering and fine-tuning the extracted features to boost the performance of the classifier.The process for training the sensor data classifier is composed of two phases:We used the in-house IR sensor module shown in Data input to the classifier network were a series of four 320 × 240 pixel IR diffusion snapshot images, which were taken sequentially by the camera and stored in an input image queue. The network performed preprocessing before sending the images to the feature extraction part of the network. In the preprocessing procedure, the preprocessing unit highlighted the transition of the SRDR contour shape to make the classifier network focus on variation of the SRDR contour shape, and then grouped the four sequential images to meet the RNN window size of the STAE. The unit first converted the input image to a black-and-white binary image with an empirically determined threshold value to highlight the contour shape of each SRDR. In addition, the threshold process reduced the sensor noise due to unwanted external IR light sources, such as the sun or fluorescent lamps. Then, the unit calculated pixel-wise differences in the SRDR binary images between certain time steps  and the previous time (n-2), in practice) to highlight the variation of the SRDR contour shape. After that, the images were resized into 28 × 28 pixel images to save computational resources. The past images were stored in the image queue. Finally, the unit composed four sequential images as a set of input image sequences for input to the feature-extraction part. The image at the end of the input sequence indicated the latest image captured from the camera. The explained preprocessing procedure is illustrated in The camera of the proposed sensor converted user facial gestures into sequential NIR SRDR images. Those images were the data input to the classifier network, which decided whether the data implied a winking gesture. Image clustering is a promising method for image classification in an unsupervised way; however, this is still a hard task for the following two reasons. The first issue is the dimensionality of the input image data. Images are often of very high dimensionality, and this should be reduced to avoid the “curse of dimensionality” , which sk clusters represented by a centroid j = 1, …, k, STAE was able to reduce the dimensionality of each We adopted STAE  to reducThe autoencoder is a type of artificial neural network that learns the data representation in an unsupervised manner and typically is part of an encoder–decoder pair. The encoder is trained to compress the input data, while the decoder is trained to restore the compressed data to the original input data with minimum loss. In this context, the encoder was induced to learn data representations to reduce redundancy in the input data. As illustrated in Clustering is one of the most commonly used unsupervised learning techniques, and it is used to group data that have similar characteristics. Compared to supervised learning-based classification, clustering has the advantage of not needing a background answer for the data. For the implementation of clustering, many researchers have adopted the K-means algorithm  because The sensor adopted deep embedded clustering (DEC) method, which was introduced by Xie et al. , to reinQ and the target distribution P using the functions mentioned in [Q remained unchanged within a predefined threshold value. After the fine-tuning process, the weights of the encoder and centroid coordinates of the clusters were optimized for the classification of the input image stream. DEC clusters data by simultaneously refining a pretrained set of k cluster centroids ioned in  as the fInstead of conducting clustering validation with label information, we first validated the quality of cluster results by utilizing internal clustering validation statistics. For the validation, we calculated the silhouette coefficient  and S-Dbw , which were promising internal clustering validation statistics reported in , and theFor evaluation of the accuracy of the proposed classifier network, we conducted additional experiments  with users wearing the proposed AR headset. Because we had not collected datasets for validation and testing, we utilized a person’s online sensor data as a validation dataset. In the experimental validation step, we applied the trained weights to the network, depending on the number of iterations of learning, and then measured the classification accuracy. In the third set of experiments, participants were able to see the results of the gesture detection in response to their facial gestures in real time, and when the wink gesture was detected by the sensor, they received a score of one point. Each participant needed to earn 100 points to complete her or his experiment. We counted the number of intended wink gestures made by each participant until completing each experiment. These experiments were conducted using ten people who were all fresh participants . The proposed wink-gesture classifier achieved 95.4% average accuracy, with the worst case of entropy of 0.24, in regard to the ten participants listed in Even though the proposed method utilized an IR camera as an input device, it was completely different from conventional image-based classifications of facial expressions. The proposed sensor detects the movement of muscles underneath the skin surface by tracking the distorted alignment of collagen fiber in the skin based on IR diffuse reflectance. It detects the skin deformation of the ROI without contact and does not require whole facial images. Because humans normally blink and wink intentionally, using different facial muscles, the proposed SRDR sensor can distinguish the two actions. Another strong point of this sensing method is that the proposed sensor could generate classification features from a plain surface of facial skin where no features could be found by camera.We achieved the clustering of IR diffusion images generated by AR headset-user winking gestures with a custom-made facial gesture detecting sensor and a classifier network. The proposed sensor provides an efficient way to implement a UI for the AR headset. Many commands could be applicable with a combination of facial gestures and head rotation data, the latter of which are captured by the gyro sensor commonly integrated in most AR headsets. For example, the user could select a button by winking, or the user could move some objects in virtual space by rotating the head and manipulating it with a wink gesture, analogous to the drag-and-drop command commonly used in the desktop environment. Our training database consists of two facial gestures  from 10 people between 23 to 36 years old. Our training database is limited, so 70% of the subjects were male, and the training dataset was generated from Korean people. However, we had verified that this interface device operates quite well regardless of the ethnic group of users in the demonstration   should bWe focused on building a simple, robust, and cost-effective mouse-like, hands-free interface for AR headsets. As shown in the application demonstration in We developed a system that provides a hands-free UI for an AR headset, which consists of an IR-based facial-gesture tracking sensor and a corresponding sensor data classifier. The goal of the proposed technique was to provide a mouse interface-like simple, robust, and cost-effective UI for AR-based assistance. A wink gesture of the headset wearer was captured as an input command by a custom-designed skin-deformation-detection sensor. We implemented an unsupervised learning-based sensor data classifier network to classify the sensor signal. We collected 81,758 IR images for training of the proposed classifier network, and the network achieved 95.4% user-intention-detection accuracy in the proposed UI test application.


The research findings highlight the near-dry machining process robustness over the dry machining routine while its great potential to resolve the heat transfer concerns in this manufacturing method was demonstrated. The results confirm other benefits of these methods  linked to the sustainability aspects in terms of the clean process, friendly environment, and permits as well as in terms of improving the manufacturing characteristics.With regard to the manufacturing of innovative hard-machining super alloys , a potential alternative for improving the process is using a novel cutting fluid approach. Generally, the cutting fluids allow the maintenance of a better tool topography that can generate a superior surface quality of machined material. However, the chemical components of fluids involved in that process may produce harmful effects on human health and can trigger environmental concerns. By decreasing the cutting fluids amount while using sustainable methods , Near Dry Machining (NDM) will be possible in order to resolve these problems. This paper discusses the features of two innovative techniques for machining an Inconel-800 superalloy by plain turning while considering some critical parameters such as the cutting force, surface characteristics ( The Inconel-800 alloy that is part of a large group of nickel-based super alloys is identified in some specialized aeronautical applications especially on the turbine and aero-components. This material is employed in this field due to its excellent properties such as good resistance to corrosion and high chemical as well as physical strength against elevated temperatures . HoweverSo-called, dry machining generates the manufacturing process while the cutting fluids may completely be eliminated. Yet, this process can generate high temperatures during the machining process and when producing leads to greater concern about the machining of nickel super alloys  . Hence, The Inconel-800 superalloy made as a bar with its nominal dimension of 50 mm diameter and a length of 150 mm length was employed in this research. X, Y and Z axis).The NDM  apparatus is a professional NOGA NDM system (as exhibited in cF). The morphology of the main tool flank wear (bmaxV) was detected by a microscope  while details of surface roughness (aR) value measurements were made by using a surface roughness tester  for a cut-off length of 0.8 mm. For a consistent evaluation, the finished work piece was as a set on three different points with respect to the tool displacement while a mean calculation was later implemented.A TeLC DKM2010 dynamometer  connected to XKM software was used to measure the main cutting force  were selected in conformity with the literature statements that vary as 0.10, 0.15, and 0.20 mm/rev. The depth of the cut was settled to fix 1 mm in all the experiments.The main problems in manufacturing a dedicated machine tool apparatus may arise as the cause of cutting forces responsible for different deformation, which can affect the work-piece and are estimated by tolerance violations and errors. Their major impact is in a close relationship with some material structural parameters  and/or conditions of cutting. The values of the cutting force obtained in dry and near dry conditions by using alternative cutting parameters were introduced in The difficulty of machining the Inconel-800 alloy is demonstrated as well as by an excess of heat produced during dry machining that can severely change the work-piece into a more brittle material, which definitely raises the cutting forces . HoweveraR) [aR) evaluated when it was employed with a dry and near dry cooling/lubricated strategy. Furthermore, morphological details of machined surfaces obtained by SEM microscopy  and produced for different conditions such as dry and NDM are introduced in By using the ISO 4287 standard guidance, it was possible to estimate the average of the surface roughness (Ra) . Figure This fact  causes the potential vibration at the machine tool assembly and the rubbing between the tool and work-piece may produce a local increase of the temperature on the cutting zone that later on will generate loss of surface integrity of the work-piece. Dry and near dry machining outcomes in terms of surface roughness from  prove thThe NDM process generated smaller values of surface roughness with no cavities and metal debris at all cutting conditions. In fact, when the pressurized air is introduced together with cutting fluid, it generates a significantly better cooling condition on the cutting zone. On both works, the materials were emphasized with a small thermal softening that has decreased to a certain extent by reducing the friction between the tool and the work-piece interface. Hence, a high surface finish is observed in NDM conditions. It can also be noted that the variation is more obvious under dry conditions. The means, the higher tensile strength (465 MPa), and the hardness (42 HRC) of the Inconel-800 can produce some undesirable effects on the surface quality by increasing the built-up-edge (BUE) tendency when considering dry conditions. Nevertheless, this phenomenon was reduced by applying near dry machining that allows as well as diminishs the tendency of the BUE formation and generates better surface quality . The resMechanistically and physicality demonstrated that the machining process cannot occur without generating a certain amount of heat. The hard contact between the tool-chip interfaces can produce the maximum heat. The heat produced there can affect directly the integrity of tool life, the variation of cutting forces, and/or the geometry and its morphology of the chip. It was demonstrated that the main type of tool failure is of adhesion, abrasion, or diffusion. The value of tool wear with variable cutting speeds at different feed rates is demonstrated in bmaxV grew. In addition, it was observed that partial de-bonding of the protecting coating that covers the tool and is caused by the high speed–feed combination may further increase the amount of flank wear. The comparisons of tool wear considering dry and near dry conditions in It has been observed from the plot that the tool wears rapidly and grows with the rise in cutting speed combined with the growth of the feed rate. The reasons might be linked to the fact that, when the cutting speed together with the feed rate was increased, an extension of the chip area occurred while the friction between tool-chip interfaces increased. Therefore, The higher hardness values of the Inconel-800 (42 HRC) can be detrimental to the tool life of the dry machining, which reduces the strain rate of the tool such as, when it reaches a certain limit, the damage will be clearly described by the accentuated tool wear. The mechanism observed is proven by Wang et al.  in the mIn metal cutting operations, the chip control is one of the most vital perspectives. The type of chip formed significantly affects the efficiency of the industrial machining process. The generation of chips that includes its aspect of breaking is prominent in machining since it can affect the tool life and the geometry of the surface finish that is linked to the accuracy of the work piece produced . Hence, (1)For the cutting forces, an application of the NDM produced less magnitude of cutting forces  than those of the dry machining environment. The cutting fluids that are of mist form a pass with a high-speed jet through the nozzles, which will help to reinforce the lubrication at the capillary interface between the tool and chip, which reduces the friction and cutting forces by forming the boundary lubricating layer.(2)In NDM conditions, the machined surface shows a greater quality  because a proper usage of the cutting fluid within a very limited quantity i.e., near dry machining can possibly enhance the surface quality by reducing the gradient temperature in the cutting area and avoids early damage of the tool-tip. Moreover, an enhanced lubrication due to the oil droplets presumably reduced the friction between the fraying surfaces.(3)After analyzing the tool wear, the NDM reveals better performances in terms of the tool wear value (reduction of 4% to 11%) in contrast with dry machining. The use of compressed air with cutting fluid has almost eliminated the amount of the crater on the rake surface by using a proficient dispersion of the coolant between the rake surface and the chip formed.(4)For the analysis of chip control, the application of near-dry machining generates small fragments of chips with a bulk average ratio of 8.1 while some unbroken, very long continuous chips that have a bulk ratio of approximately 68 were produced under machining dry conditions.(5)The near dry machining process generates dry chips that can be considered as an initiative towards the cleaner production.Near-dry machining or NDM represent a sustainable strategy that could reduce several machining issues related to its performance i.e., cutting forces, surface roughness, tool wear etc. The main findings of the present investigations are summarized below.


Gait deficits in cerebral palsy (CP) are often treated with a single-event multi-level surgery (SEMLS). Selecting the treatment options (combination of bony and soft tissue corrections) for a specific patient is a complex endeavor and very often treatment outcome is not satisfying. A deterioration in 22.8% of the parameters describing gait performance has been reported and there is need for additional surgery in 11% of the patients. Computational simulations based on musculoskeletal models that allow clinicians to test the effects of different treatment options before surgery have the potential to drastically improve treatment outcome. However, to date, no such simulation and modeling method is available. Two important challenges are the development of methods to include patient-specific neuromechanical impairments into the models and to simulate the effect of different surgical procedures on post-operative gait performance. Therefore, we developed the SimCP framework that allows the evaluation of the effect of different simulated surgeries on gait performance of a specific patient and includes a graphical user interface (GUI) that enables performing virtual surgery on the models. We demonstrated the potential of our framework for two case studies. Models reflecting the patient-specific musculoskeletal geometry and muscle properties are generated based solely on data collected before the treatment. The patient's motor control is described based on muscle synergies derived from pre-operative EMG. The GUI is then used to modify the musculoskeletal properties according to the surgical plan. Since SEMLS does not affect motor control, the same motor control model is used to define gait performance pre- and post-operative. We use the capability gap (CG), i.e., the difference between the joint moments needed to perform healthy walking and the joint moments the personalized model can generate, to quantify gait performance. In both cases, the CG was smaller post- then pre-operative and this was in accordance with the measured change in gait kinematics after treatment. Cerebral Palsy (CP) is the most common cause of motor deficiency in young children with a prevalence of 2–3 cases per 1,000 live births  based models has been extensively demonstrated  into the musculoskeletal models. The need to account for the musculoskeletal deformities of the individual CP patient and the bony deformities in particular, dictates the use of The altered muscle parameters  in patients with CP compared to a healthy population  need to be accounted for into the musculoskeletal model. Whereas the effect of muscle-tendon lengthening and muscle transfer on the moment-generating capacity, lengths and velocities of the muscles have been described  and medical images. To this aim, the framework contains several building blocks . First, In order to provide the information needed for the complete personalization of the models, inputs from several sources are required. First, imaging data, such as CT scans and MRI-images, allow defining the musculoskeletal geometry. Second, three-dimensional gait analysis data, including marker trajectories and ground reaction forces as well as EMG signals from the most important muscles in the lower limb, are needed for the personalization of the muscle parameters and definition of the motor control model. Third, clinical examination reporting the passive range of motion of the patient contains useful information to refine the personalization of the muscle parameters.The musculoskeletal models are based on a generic SIMM  model and are composed of 14 bodies and 21 degrees of freedom that are actuated by 86 muscles. The musculoskeletal geometry of this generic model is adapted to reflect the patient's musculoskeletal geometry. The model is then further personalized by tuning the muscle-tendon parameters (section Muscle Parameters) and by adding a model of the patient's motor control (section Motor Control).Musculoskeletal geometry is derived from MRI-images. The workflow to create the MRI-based models has been published previously .Performing virtual surgery alters the musculoskeletal geometry and hence generates a new model, which is linked to the lmoand lts)  and, optionally, the clinical examination of the passive joint range of motion . The undfmincon function in Matlab.To perform this tuning, we extended a static optimization problem since this allows us to optimize the fit between the computed activations and joint moments, and, respectively, the EMGs and inverse dynamic joint moments. Static optimization computes muscle activations that produce the inverse dynamic joint moments underlying a measured movement while optimizing a performance criterion . Here, we allow optimal fiber lengths and tendon slack lengths to change during the optimization while imposing constraints on the allowable muscle lengths that represent the tuning criteria described above. In contrast to the typical static optimization approach that is solved for each time frame separately, here all time frames are coupled to obtain a single set of muscle-tendon parameters. It is important to note that static optimization neglects muscle dynamics by assuming that tendons are rigid but allows accounting for the muscle force-length-velocity relationship  (fL), depending on normalized muscle fiber length (a), and a passive component, (fP) depending only on fiber length:To cope with the scarcity of input data , we decided to tune the parameters only in a set of major muscles (M) . In addiM) , was takf is the maximum isometric force the muscle can exert and the subscript i defines the instant in time. For the remaining muscles (N), the generated force is proportional to activation:Where j) during the gait cycle . Reserve moments are generated by ideal actuators and are added to the muscle moments to guarantee that the inverse dynamic joint moments can be matched even when the muscles are not sufficiently strong. Since they are not physiological, their use is heavily penalized in the cost function to keep their contribution to a minimum:Hence, the resulting estimation problem has the following structure. Optimization variables consist of muscle activations and reserve moments  for which EMG was collected see  reflect Where ε represents the experimental EMG envelope. The scaling factor σ was introduced as an optimization variable to impose similarity between activations and EMG patterns irrespective of signal amplitude since, in the absence of maximum voluntary contraction tests, the relation between signal amplitude and muscle activation cannot be accurately derived.The cost function was minimized subject to the following constraints. A first set of constraints describes that the muscles should produce the inverse dynamic joint moments:m with respect to joint j at time i.Where A second set of constraints imposes bounds on the muscle fiber lengths during gait and the clinical exam of the range of motion. To ensure that normalized muscle fiber lengths during gait are within 0.4 and 1.5, we constrain the minimal and maximal fiber lengths. In addition, to ensure that muscles operate around their optimal length during gait, we constrain the maximal fiber length to be above and the minimal fiber length to be below optimal fiber length:Maximal normalized fiber lengths during the clinical exam  and, the final cost function becomesl) and normalized fiber length  while the musculoskeletal model contains many more muscles . A multi-step procedure is proposed to derive muscle coordination for all muscles in the model based on EMG data from a limited number of muscles  using non-negative matrix factorization (NNMF)  of these resampled signals by the synergies extracted from the original signal is computed. Ns is the lowest number of synergies for which VAF is higher than 90% for at least 95% of the resampled signals. The number of synergies Ns is used in the subsequent steps.First, muscle synergies are derived from the pre-operative EMG data acquired during three gait cycles to define the complexity of the patient's motor control by the number of synergies (mσ. The cost function is obtained by combining (Equations 3 and 4):Next, an EMG-informed static optimization analysis is performed on the patient's pre-operative gait data, where the optimization variables are Cϵ term enables us to account for the pathological characteristics of muscle activations, such as antagonistic muscle co-contractions, which are very common in children with CP. This cost is minimized subject to constraints describing the equilibrium between inverse dynamic and muscle moments (Equation 5).The Ns + 1. The extra synergy is included to take into account muscles for which no surface EMG was collected. For instance, Allen and Neptune (Hpre) that define the motor control model of the patient and describe the activations of all the muscles. This motor control model is later used in the computation of the gait performance of the patient (section Capability Gap and Muscle Report).Finally, muscle synergies are extracted by performing a new NNMF on the muscle activations computed with static optimization. The number of synergies for this analysis is  Neptune  found thExtension and Derotation Osteotomy, Derotation Osteotomy, Muscle Transfer, Patella Advancement and Botulinum Toxin Injection. Only the Muscle Transfer and Botulinum Toxin Injections influence the muscle parameters.We developed a set of virtual surgeries and a GUI that allow to directly manipulate the musculoskeletal models by leveraging the Matlab-OpenSim application programming interfaces. The surgeries implemented in the current version of the GUI are Extension and Derotation Osteotomy  can generate. We use a synergy-based static optimization approach to compute the capability gap. Here, the reserve moments appearing in the moment equilibrium function (Equation 5) represent the torque deficit and hence the capability gap. Subject-specific musculoskeletal geometry and muscle parameters are described in the musculoskeletal model. Impaired motor control is imposed through additional constraints on the activations based on the patient's muscle synergies.The desired motion used for the computation is derived by scaling average TD walking data to the patient's dimensions . First, CSO as defined by Equation (3). The moment equilibrium (Equation 5) has to be satisfied. Inputs to Equation (5), Hopt. The optimization variables are hence mW, which is a deviation from the pre-operative synergy weights (see below). Individual muscle activation patterns (including both subsets N and M) are then computed from the synergy activation patterns using the synergy weight vectors that describe the patient's motor control Wpre (see section Motor Control):Afterwards, the synergy constrained static optimization is performed. The cost function to be minimized is Wpre) are used to compute the pre- and post-operative CG, since we hypothesize that the orthopedic intervention does not alter muscle co-contraction patterns. Hence, we assume that the neural system will respond to the altered musculoskeletal geometry by changing the timing and magnitude of the pre-operative activation patterns H. A small deviation (W) from the original weights is allowed because the synergy matrices computed using NNMF typically do not capture 100% of the signal variability. In our formulation, W is normalized so that the maximum value in each vector equals one and W ≤ 0.05.The same muscle co-contraction patterns . Muscles active at short lengths are those whose activations are >0.25 when operating at normalized fiber lengths smaller than 0.6, meaning that they are producing relatively little force.Furthermore, we provide a muscle report  summarizThis framework is originally intended to work with the personalized neuro-musculoskeletal models as described above but it is also possible to exclude one or more of the personalization blocks. In fact, this procedure can be of value when assessing the importance of the different factors contributing to the impairment, as highlighted in section Case Studies. It is for instance possible to import a scaled, generic model, and perform the same analyses and surgeries, as envisaged for the personalized models, in order to evaluate the importance of the bony deformities in defining the impairment of the patient. Alternatively, it is possible to investigate alternative causes of the functional impairment by excluding the muscle force/length relationships or the constraints imposed by the motor control on the muscle activations when computing the gait performance (CG).We analyzed two representative patients with the proposed framework . Both weWe created pre-operative personalized models using the experimental data  and post-operative personalized model by performing virtual surgeries according to the surgical plan of the actual intervention. For Patient 1 the following interventions were modeled: bilateral rectus femoris transfer, distal femur extension, and derotation osteotomy, patella advancement. Patient 1 also received a derotation of the tibia, but this was not modeled due to the fact that the MRI from which the musculoskeletal model was built did not include images of the feet and distal tibiae. For Patient 2 the following interventions were modeled: bilateral rectus femoris transfer, left distal femur derotation, right distal femur extension, and derotation osteotomy, patella advancement, right gastrocnemius and psoas release.Extension and Derotation Osteotomy intervention, the angle between the two cutting planes in the femur was modeled based on the knee extension deficit observed when testing the passive range of motion. For the femur Derotation Osteotomy, the anteversion angle was corrected to be equal to 0° in the simulated post-operative model in agreement with information provided by the orthopedic surgeons. Patella advancement was modeled by shortening the patella ligament by 2 cm, as reported in the surgical plan. Rectus Femoris Transfer was modeled in two steps. First, a via point in the femur reference frame was introduced in the middle of the muscle-tendon unit. Second, the insertion site was transferred to the semitendinosus tendon, while keeping the original length of the musculotendon unit unchanged. Our approach replicates the surgical procedure in which the rectus femoris is detached and reattached distally but left attached proximally, thus maintaining its function as a hip flexor. Muscle Release interventions were modeled by completely removing the muscle contribution from the generated moment, i.e., by setting the maximum isometric muscle force to zero. Although both patients additionally received botulinum-toxin injections, these were not included in the postoperative model given that their effect can be considered small given the time between the pre- and post-operative observations (10 and 13.5 months).To simulate the We compared the predicted motor performances of the patients in terms of the capability gap for the pre-and post-operative conditions . We testThe patient had near normal range of motion at the hip and ankle, but a bilateral knee extension deficit in both limbs, bilateral spasticity in most muscles, including rectus femoris, good strength in most muscles bilaterally but slightly lower strength in hip and knee extensors, as well as hip abductors, with overall good selectivity . The preSynergy analysis revealed that three synergies were sufficient to describe the pre-operative EMG signals in both legs. For comparison, previous work of the group found that, during walking, 57% of TD children use four synergies, whereas the remainder of the subjects uses three. Therefore, to take into account muscles for which no EMG were collected, four synergies were used for the CG computation.An important CG was found bilaterally at the level of the knees and to a lesser extent hip adduction , 6B. TheThe tuning of the muscle parameters was a necessary step to perform the aforementioned analyses. After applying the bony deformities to the model, most of the muscles would have operated at excessive values of l~ . Therefol~ .Within the framework, it is possible to evaluate the isolated effect of motor control deficit, by in- or excluding the motor control model when computing the post-operative CG. In The patient's right side was more involved in terms of passive ROM, spasticity and muscle weakness . More spSynergy analysis revealed that four and three synergies could explain the pre-operative EMG signals for the left and right leg, respectively, thus leading to the use of five and four synergies in the CG computations. The CG in the pre-operative condition reflected the reduced range of motion of the right leg. CG was higher for the right then for the left leg with muscles gastrocnemii, hamstrings, iliacus and psoas being excessively stretched , leadingThe simulated treatment had a very different impact on the CG for the two legs. For the right leg, surgery massively reduced the CG. The simulated extension and derotation osteotomy in isolation reduced the CG generated by the hamstrings at the hip and knee at the beginning and end of the gait cycle. The use of a trapezoidal wedge, most commonly used to correct large extension deficits, reduced the CG most by reducing excessive stretch in the muscles . The effWe introduced a novel and promising musculoskeletal modeling and simulation-based framework to assist clinicians in the treatment selection process to improve gait function in patients with CP. The salient feature of this framework is a comprehensive personalization of the models comprising subject-specific musculoskeletal geometry and muscle parameters as well as motor control. Furthermore, we introduced a GUI to simulate different orthopedic interventions and interactively modify the musculoskeletal models. As a result, the effect of several candidate orthopedic interventions on the gait performance, evaluated in terms of the patient's capability gap, can be evaluated. In comparison with a number of other studies that aimed to predict the outcome of orthopedic treatments , which will enable the biomechanical community to create post-operative models in an easy way and therefore foster future research related to orthopedic interventions and pathological gait. Furthermore, the GUI as well as the concept of the CG can be applied to different populations  and research questions .A beta version of the developed GUI is freely available ]. In accordance with the Declaration of Helsinki, written informed consent was obtained of the participants' parents prior to the experiment. The participants parents supervised the measurement session. The protocol was approved by the Medical Ethics Committee UZ KU Leuven/Research.LP, IJ, FD, KD, and AH contributed conception and design of the study. EP collected the data. AV and GM provided clinical insights for the methods development. LP, HK, AF, MW, SV, and HH contributed in developing the methods. LP, IJ, and FD wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The burnup credit application is examined with respect to both existing concepts: taking into account actinides only and taking into account actinides plus fission products. The criticality safety calculations are integrated with uncertainty quantifications that are as detailed as possible, accounting for the uncertainties in the nuclear data used, fuel assembly and disposal canister design parameters and operating conditions, as well as the radiation-induced changes in the fuel assembly geometry. Furthermore, the most penalising axial and radial burnup profiles and the most reactive fuel loading configuration for the canisters were taken into account accordingly. The results of the study are presented with the help of loading curves showing what minimum average fuel assembly burnup is required for the given initial fuel enrichment of fresh fuel assemblies to ensure that the effective neutron multiplication factor, This paper presents preliminary criticality safety assessments performed by the Paul Scherrer Institute (PSI) in cooperation with the Swiss National Cooperative for the Disposal of Radioactive Waste (Nagra) for spent nuclear fuel disposal canisters loaded with Swiss Pressurized Water Reactor (PWR) UO The Swiss National Cooperative for the Disposal of Radioactive Waste (Nagra) plans to submit a general licence application for a deep geological repository for the disposal of spent fuel and high-level waste (HLW repository) and for low- and intermediate-level waste (L/ILW repository) by 2024. One of the requirements for the design of the HLW repository is the safety of the installations (encapsulation facility and repository) from the point of view of a possible criticality excursion over a 1,000,000-year lifetime. Were it to occur, criticality would affect the properties of the engineered barrier system, namely the canister, and the backfill material and the near-field of the host rock.For the above reasons, the criticality safety issue for the disposal of spent fuel canisters was preliminarily investigated by Nagra already in 2002 in the context of the safety assessment of a repository for spent fuel and high-level waste in the Opalinus Clay . The res® MCNP® or MCNPX Software .) using criticality benchmark experiments keff valuferences ). The apferences ,43.For an additional illustration, The Monte Carlo sampling method employed to obtain the estimated uncertainty in 2 fuel and for the employed ENDF/B-VII.1 CM. Details of the calculations performed are given in Reference [eference .2 fuel . The deuel .It should be noted that the uncertainty components ferences ,29.Next, to be on the conservative side, the total ND-related uncertainty will be composed of the ing time .The performance of the decay module of the SERPENT2 code and the employed nuclear data library was investigated and benchmarked in Reference . Later, The Monte Carlo statistical uncertainty The analysed random uncertainty components are summarised in To better illustrate the impact of the considered burnup profile penalties and the uncertainty components, The final target of this work was to assess a minimum average burnup for individual fuel assemblies required for a full loading of the disposal canister without exceeding the defined upper subcritical limit. This goal is accomplished by the development of specific loading curves for discharged spent fuels, where the initial enrichment and final burnup of a fuel bundle will function as the acceptance criteria for the loading of the disposal canister. The development of the curve is done as follows: the left part of Equation (1) is represented by a curve depending on the burnup, while the right part of Equation (1) is a constant corresponding to the given USL value. If the burnup-dependent curve of Equation (1) and the burnup-independent USL line intersect, the burnup at the point of the intersection becomes the point on the loading curve corresponding to the given fuel enrichment. If the burnup-dependent curve of Equation (1) is always below the USL value, then for the given enrichment, the minimum burnup equals zero on the loading curve. As presented in To give an outlook on the general behaviour of the curve over the burnup, It can be seen again that the most conservative case for the credit of AC+FP is zero cooling time after discharge, while in the case of only AC, the most conservative results generally correspond to the cooling time of about 30,000 years  are also demonstrated in 2, 4.94 w/o fuel, the saving in terms of the minimum required burnup as compared between the cases of AC+FP vs. AC is about 24 GWd/tHM (this has been illustrated in more detail in Reference [It can be seen from eference ). This reference : for theThe classical practice for criticality safety calculations for applications out of the reactor has relied on the fresh fuel assumption. The advantages of taking credit for changes in the actinide compositions after irradiation and the build-up of fission products in the spent fuel, with a net effect of decreasing the In line with this, a preliminary criticality assessment study has been performed taking into account the burnup credit within the PSI/Nagra BUCSS-R research project recently conducted at PSI. Ways for further methodological improvements and the verification of the present results are discussed below.the nuclear data (ND) combined impact . Also, it would be relevant to assess the SIMULATE-3/BOHR/CASMO5 methodology component approximations, namely the infinite lattice calculations of the pin-wise SNF compositions with CASMO5 code. With respect to this limitation, the transition to the SIMULATE5 code in combination with another Studsvik code “SNF” (in quotation marks to distinguish from the earlier spent nuclear fuel abbreviation) could provide intra-assembly pin-wise isotopic distributions from the real core-follow calculations, as discussed in the next section. Such work is currently ongoing at PSI outside the present research collaboration with NAGRA.Furthermore, a comparison of the current methodologies, which can be classified as a combination of the best estimate plus uncertainties (BEPU) approach with certain conservative assumptions, against more traditional methodologies can be recommended. In particular, the comparison of the BEPU SNF compositions versus analogue data but with application of the so-called isotopic correction factors  obtainedFinally, the inclusion of the correlation analysis in the CSE methodology, as discussed in Reference , should The presently developed BUCSS-R calculation scheme is considered at PSI to be sufficiently detailed and accurate for the construction of the BEPU-type loading curves, provided that statistically representative samples can be obtained for FAs with all considered fuel types and enrichments. A drawback of this system is that it is rather demanding in terms of calculation resources. Important to mention is that the SIMULATE/BOHR/CASMO calculations required for the BUCSS-R scheme are basically not needed for any other type of simulation, so they are practically solely dedicated to the BUCSS-R methodology for the loading curve derivations . Since the CASMO calculations generally have to be redone every time an updated/new version of the CASMO code or/and its cross section library are released, such a method for the derivation of the loading curves becomes costly and inefficient.2M method has been presented in Reference [However, an alternative way to derive the loading curves recently became available based on the exploitation of the advanced features of the additional CMS code “SNF”, which can be seamlessly integrated into the PSI CMSYS system. A description of the new methodology developed at PSI outside the BUCSS-R project, which is based on the CASMO/SIMULATE/SNF/COMPLINK/MCNP system of codes and which was named the CSeference , togetheeference  for detaFurthermore, as already mentioned, it is recommended in the future to use the SIMULATE5 code, as it has a more accurate inter-assembly SFC characterisation capability. In doing so, it will be in principle possible to extract the SFC for every individual fuel assembly ever operated in Switzerland and thus to avoid all undesirable approximations to realise the BEPU calculations. The uncertainty propagation can be done with the same tools as used before in the BUCSS-R scheme. This is illustrated with the shaded boxes in the top part of Therefore, although the new CS2M approach is also time-consuming, it does not require any complementary and unnecessary simplified assessments, and for that reason, it is more efficient and transparent for verification and validation. Finally, since both approaches being developed at PSI to derive the loading curves for the Swiss reactor spent fuel assemblies differ from the more conventional “conservative” approaches . Theoretically, mixed configurations of SNF with different burnup levels could also be a solution for the most problematic cases.The loading curves must be treated as preliminary since there is still room for improvement in the assessment of different components of Equations (1)–(3). However, valuable findings have already been obtained concerning the identification of the most problematic scenarios and loading schemes, which will serve as guiding information for the next phases of the research and development at PSI and Nagra, in particular for the optimisation of the canister design and the mixed burnup/fuel assembly design loading schemes.Among the topics for further improvement of the BUCSS-R methodology , other aspects can be proposed for consideration, such as the extension of the analysed FA sample to yield 95%/95% tolerance bounds for the loading curves ; a refinement of the treatment of the uncertainties with respect to the burnup axial and radial profiles, allowing a consistent “Total Monte Carlo” assessment (“seamless” calculations of both depletion/decay and criticality models using the same ND perturbation factors, as outlined in 2M scheme [In parallel, assessments of an alternative option for the currently used BUCSS-R methodology are ongoing independently at PSI, for which Studsvik’s code “SNF” is integrated into the CSM scheme  to subst


Low-rank representation-based frameworks are becoming popular for the saliency and the object detection because of their easiness and simplicity. These frameworks only need global features to extract the salient objects while the local features are compromised. To deal with this issue, we regularize the low-rank representation through a local graph-regularization and a maximum mean-discrepancy regularization terms. Firstly, we introduce a novel feature space that is extracted by combining the four feature spaces like CIELab, RGB, HOG and LBP. Secondly, we combine a boundary metric, a candidate objectness metric and a candidate distance metric to compute the low-level saliency map. Thirdly, we extract salient and non-salient dictionaries from the low-level saliency. Finally, we regularize the low-rank representation through the Laplacian regularization term that saves the structural and geometrical features and using the mean discrepancy term that reduces the distribution divergence and connections among similar regions. The proposed model is tested against seven latest salient region detection methods using the precision-recall curve, receiver operating characteristics curve, F-measure and mean absolute error. The proposed model remains persistent in all the tests and outperformed against the selected models with higher precision value. Salient Region Detection (SRD) is a procedure to confine the image according to the human visual attention and discovers the most useful and informative portion of an image. This procedure tries to approximate the possibility that the image region that is taking more attention comes out as a salient object. It is also a very helpful step because it is applied in many computer vision applications to reduce the computational complexity by only focusing on the salient parts of the image. The conventional saliency methods are separated into two groups as the bottom-up  and top-Existing techniques reveal the efforts of the researcher in finding the different features for SRD. The major portion of SRD literature is comprised of the bottom-up approaches , as thesThe previously designed background dictionary-based methods , 11–14 uThe local methods compute the saliency by the rarity of neighbors or surrounded regions . While tTo save the local and global image information, and to evade the unrelated background regions from embedding with the salient object, we exploit a Graph Regularize Term (GRT) and a Maximum Mean-discrepancy Regularization Term (MMRT). The GRT preserves the locality of the salient object and homogenously treats with all the parts of salient objects. While the MMRT decreases the distribution divergence between the similar regions. In this way, the background regions, which are mistakenly highlighted are suppressed by treating with MMRT. On the other hand, the salient object regions that are erroneously concealed during pre-processing are decorated again by exploiting this MMRT. In short, theses regularization terms transform the low-rank representation into a new appearance, which produces the saliency results more smoother, locally and globally homogeneous. The contributions of our designed method are given as:In this proposed framework more structural information of the image is incarcerated, for this purpose we concatenate four feature spaces to construct a new feature space that is consist of CIELab, RGB, HOG and LBP features.We propose a discriminative background dictionary that is constructed through the background and the foreground templates. This discriminative dictionary has more distinctive power because in our method the salient object is projected through the background and the foreground templates instead of the background template.We regularized the low-rank representation to save the similarity and locality of the regions. We introduced a GRT and a MMRT for the representation coefficients and the representation errors, respectively. By using, these terms the representation coefficients and the representation errors of similar regions contain similar saliency values when sparsely encoded with the discriminative dictionary.Our designed framework has more discerning power and more effective appearance as compared to the current low-rank representation-based methods because the low-rank representation is presented in a new way through the MMRT.II. The details of our method like the feature extraction, the salient and non-salient template construction, and regularizing the low-rank representation through the Laplacian and mean discrepancy terms are given in the Section III. The evaluation metric, evaluation of our method, and its comparison with the state-of-the-art methods are given in Section IV. The conclusion and the future work are discussed in the Section V.The remainder Sections of this paper are ordered in the following way: the existing schemes related to the SRD are discussed in the Section Several computational methods are proposed for SRD. The majority of the preceding schemes are appearance-based models, these models mainly depend upon the global or local contrast for their saliency map computation.The dictionary-based approaches , 11–14 fLow-rank representation finds the lowest-rank from all the candidates that are available as a dictionary base. Low-rank representation works as an effective tool and computes all the global features information. On the other hand, the sparse representation seeks the sparsest representation of available data vector and computes all of the local features related to the salient objects. In , the autThe image boundary is always standing out as a part of the background. So, it can be very helpful in constructing the background template set , 15. TheThe previously designed SRD techniques are broadly divided into two categories local and global methods. The local methods compute the saliency by the rarity of neighbors or surrounded regions. While the global methods extract saliency using the uniqueness of features over the entire scene. In , the autYCbCr color space for their result calculation. The authors consider the near-infrared region with the RGB color model for SRD  over D =  and error S. Then, the salient region maps are computed using the background and foreground templates as:SST and SNT represents the reconstruction errors due to the salient and non-salient parts, respectively. We will get the final saliency results after merging the SST and SNT maps using the following expression:After the extraction of discriminative saliency, we divided the computed feature matrix into parts with the proposed model as the coefficients are We initialize the low-rank representation method , after tWe analyzed and investigate our model on four largest benchmark datasets against existing methods. For performance assessment, four evaluation criteria are selected to completely analyze the proposed algorithm against the preceding schemes. In the next section, we discuss the details of selected benchmark datasets for performance evaluations.To analyze the computed saliency results, many datasets available that differ from each other in size, objects in the scene, image background like simple or complicated and the GT. We employ different datasets to assess the performance of our proposed algorithm. We assess our SRD model on four different standard datasets that are: 1) ASD , 2) ECSSOur SRD model is compared against fourteen state-of-the-art models. We first visually and then graphically compare to check and validate our framework. The schemes, we compare with our method are chosen due to these reasons: 1)citations, 2)recency, 3) computation complexity, and 4) variety. These schemes are: AC , FT 35]35], GB , HC 10]10], HS , MC 40]40], UFO S, that is calculated as:w and h represent the height and width of saliency map, respectively.Numerous techniques are applied to evaluate the concurrence between the obtained results and the GT. Before computing the evaluation metrics, the produced salient region maps should be changed in binary form to estimate the generated map. We also apply the adaptive threshold as discussed in , the thrThe saliency map S is converted to the binary-mask M using the given ground truth T. The PR-curve is computed using this expression:F-score is calculated using the Precision-Recall, the evaluation of the SRD is not complete without F-score. The F-score is computed using the following expression:ν = 0.3. So, we have take the value of ν = 0.3 for a fair comparison.All of the compared method take the value of The ROC-curve is obtained using the binary mask M with a fixed threshold as:To check the worth of SRD maps might have high significance as compared to binary mask. We also applied the MAE between the continuous SRD map S and the ground truth T, both are normalized in the range . The MAE value is defined as:In this section, we discuss the impact of different parameter on the performance of our proposed model. These parameters play a key role in obtaining the optimal results. These parameters are GR parameter, MMR parameter, and the sparsity parameter. We perform a series of experiments to adjust the values of these parameters. The details are discussed in the next subsection.α1 = 1, α2 = 0.1, α3 = 105, and N = 300.The performance of proposed model is severely affected by the GR parameter, MMR parameter, the sparsity parameter and the number of superpixels. Therefore, for the optimal results the values of these key parameters are adjusted as Normally, this parameter is theoretically utilized to penalize the discontinuities and geometrical features in representation coefficients. We also perform a series of simulation with varying the values Laplacian parameter. The results revealed that with the higher values the geometrical structures of the salient objects are lost. While, with the lower values of the Laplacian parameter, structural and geometrical features are preserved as revealed in α2 or as we move toward the zero a few numbers of iteration are required. In the contrast, if the value of sparsity parameter moves in the direction of infinity the computed saliency maps not remain significant due to the supremacy of the sparsity as demonstrated in To check the impact of the sparsity parameter, we use different values to run the simulation for the sparsity parameter in the range starting from {0.1} to {0.5}. Generally, the sparsity parameter is engaged to keep away from deterioration and over-fitting. From the simulation, we note that with the smaller value of 3, 107}. From the experimental results shown in α3 the distribution connection between the regions is assured while the sparsity is badly affected. In the contrast, if we choose the smaller value for the MMR parameter the geometrical structure of the salient object and the sparsity is secure while the distribution connection between the regions is lost. In both of the cases, the sparsity of the approach is also not robust. For the ease and efficiency, we select the value of α3 = 105.To check the impact of MMR parameter, we perform the simulation with varying its values in the range between {10N = 300 and at this value, the proposed model performs well.We also check the impact of the number of superpixels on the performance of the proposed model as shown in We perform simulations and used PR curve to check the effect of features on the efficiency of our model. The comparisons between the different features of our model are shown in We also analyze the impact of regularization terms like Laplacian regularization and mean discrepancy term on the final saliency map. By regularizing the low-rank representation through these terms can save the similarity as well as the locality of the regions. Here, we exploited the Laplacian term for representation coefficients and a MMRT for representation errors. By using, these terms the representation coefficients and representation errors of similar regions contain similar saliency values when sparsely encoded with the discriminative dictionary. We can note from the We analyze our model visually with the current schemes like: AC , CH 44]44], FT , GB 9],,9], HC , HS 36]36], MC , GM 5],,5], RB , RC 10]10], SR , UF 41]41], DS  on the MWe assess the performance of our SRD scheme with state-of-the-art methods on the ASD database  as reveaWe also use DUT-OMRON dataset  to analyMoreover, we as well engaged ECSSD dataset  to assesAdditionally, we employed SED2 dataset  to evaluOur designed model performs against the current salient regions detection methods. On the other hand, in some typical images, our model computed results are poor as demonstrated in the In the current literature, different methods employ the shape prior to segment the salient object, which is later utilized to produce the saliency cut. Some others used a rectangular locale to capture the salient object and then this captured region is converted to a fuzzy region. Both of the above-mentioned methods use the different strategies to compute the saliency cut. The proposed method also segments the image first and then produces a precise saliency cut, we use different datasets and found that the proposed model produces quite promising saliency cut results as demonstrated in In this paper, we propose a novel SRD method through salient and non-salient dictionaries. Initially, a new feature space is constructed by concatenating four feature spaces like CIELab, RGB, HOG, and LBP. Then, we combine a boundary metric, candidate objectness metric and a candidate distance metric to compute a low-level saliency map. After that, we extract a salient template and a non-salient dictionary from that low-level saliency. We regularize the low-rank representation through GRT that saves the structural and geometrical features and using a MMRT that reduces the distribution divergence and connections among similar regions. The proposed model is tested against over a dozen latest SRD method using four evaluation metrics. The proposed model remains persistent in all the tests and outperformed against the selected models with higher precision.


In this work, we propose a Spiking Neural Network (SNN) consisting of input neurons sparsely connected by plastic synapses to a randomly interlinked liquid, referred to as Liquid-SNN, for unsupervised speech and image recognition. We adapt the strength of the synapses interconnecting the input and liquid using Spike Timing Dependent Plasticity (STDP), which enables the neurons to self-learn a general representation of unique classes of input patterns. The presented unsupervised learning methodology makes it possible to infer the class of a test input directly using the liquid neuronal spiking activity. This is in contrast to standard Liquid State Machines (LSMs) that have fixed synaptic connections between the input and liquid followed by a readout layer (trained in a supervised manner) to extract the liquid states and infer the class of the input patterns. Moreover, the utility of LSMs has primarily been demonstrated for speech recognition. We find that training such LSMs is challenging for complex pattern recognition tasks because of the information loss incurred by using fixed input to liquid synaptic connections. We show that our Liquid-SNN is capable of efficiently recognizing both speech and image patterns by learning the rich temporal information contained in the respective input patterns. However, the need to enlarge the liquid for improving the accuracy introduces scalability challenges and training inefficiencies. We propose SpiLinC that is composed of an ensemble of multiple liquids operating in parallel. We use a “divide and learn” strategy for SpiLinC, where each liquid is trained on a unique segment of the input patterns that causes the neurons to self-learn distinctive input features. SpiLinC effectively recognizes a test pattern by combining the spiking activity of the constituent liquids, each of which identifies characteristic input features. As a result, SpiLinC offers competitive classification accuracy compared to the Liquid-SNN with added sparsity in synaptic connectivity and faster training convergence, both of which lead to improved energy efficiency in neuromorphic hardware implementations. We validate the efficacy of the proposed Liquid-SNN and SpiLinC on the entire digit subset of the TI46 speech corpus and handwritten digits from the MNIST dataset. SNNs are a class of bio-inspired neuromorphic computing paradigm that closely emulate the organization and computational efficiency of the human brain for complex classification and recognition tasks. Several SNN architectures have been independently proposed for learning visual and auditory signal modalities. Two-layered fully-connected SNN , a well established liquid computing model, consists of an input layer sparsely connected via synapses whose weights are fixed In an effort to enhance the liquid projections for different input modalities including the speech and image patterns, we train the input to liquid-excitatory synaptic weights in our Liquid-SNN using STDP that enables unsupervised inference without a readout layer. When a time-varying input, for instance, a speech signal is fed to the liquid, the recurrent connectivity causes a group of excitatory neurons to fire. Performing STDP on the interconnecting synaptic weights causes these neurons to learn the temporal dynamics of the presented input, which effectively renders them sensitive to similar inputs. Hence, the excitatory neuronal spiking activity can directly be used for inference. We show that the presented Liquid-SNN, trained in an unsupervised manner on a subset of spoken digits from the TI46 speech corpus  and SpiLinC (liquid-ensemble architecture), where each liquid is trained in parallel on a unique input segment, for unsupervised speech and image recognition.We demonstrate that input subdivision enables the individual liquids in SpiLinC to self-learn diverse input features, which can effectively be combined to recognize an input pattern.We propose optimal strategy to determine the ensemble size and input partition per liquid in the ensemble, and sparsity constraints in the input to liquid and recurrent-liquid synaptic connectivity required to achieve efficient unsupervised learning in SpiLinC.We validate and benchmark the proposed models on the MNIST dataset for image recognition and the entire digit subset of the TI46 speech corpus for speech recognition.The fundamental computing unit of an SNN is a spiking (post) neuron, which is driven by a group of input (pre) neurons via weighted synapses as shown in Figure Spike Timing Dependent Plasticity (STDP) postulates that the strength (or weight) of a synapse depends on the degree of timing correlation between the corresponding pre- and post-neuronal spikes. We use the power-law weight-dependent STDP model  if a pre-neuronal spike causes the post-neuron to fire within a definite period of time determined by the STDPoffset. On the contrary, the synaptic weight is decreased (depressed) for larger spike time differences. Additionally, the synaptic weight change has an exponential dependence (controlled by μ) on the current weight, which ensures a steady rise (decline) of the synaptic weight toward its upper (lower) bound that is desirable for efficient learning.where ΔThe Liquid-SNN  to produce states correlated with the input. The proposed methodology effectively tunes the spiking activity of individual neurons to correspond to specific input classes, which enables unsupervised inference without a readout layer as explained below for a speech input. In the beginning of the training phase, a time-varying speech input triggers a random group of excitatory neurons to fire. Carrying out STDP-based updates on the input to liquid synaptic weights allows the neurons to learn temporal correlations underlying the presented input. This increases the likelihood of the corresponding neurons to thereafter spike for similar inputs. The recurrent inhibition facilitates the activation of distinct excitatory neurons for various input classes. STDP imparts temporal learning and sensitizes each neuron to the appropriate input class, which enables the Liquid-SNN to recognize a test input straightaway by using the excitatory neuronal spiking activity as explained below. Once the input to liquid synapses are trained, we tag the excitatory neurons in the liquid with the classes of input patterns for which they spiked at a higher rate during the training phase. During the inference phase, for a given test pattern, we first estimate the average spike count of every group of neurons tagged as having learnt features pertaining to a specific class of input patterns that is specified byavg_spike_count(j) is the average spike count of the group of neurons carrying the tag “j,” nj is the total number of neurons with the tag “j,” spike_count is the spike count of the kth neuron with the tag “j” over the time interval for which the test pattern is presented , and nclasses is the number of classes for a given pattern recognition task. We then predict the unknown test pattern to belong to the class (or tag) represented by the neuronal group with the highest average spike count as described bywhere predicted test class is the class of the test pattern predicted by the Liquid-SNN using the average spike count of the excitatory neurons and the corresponding tags.where In order to illustrate the generality of the proposed model, we describe how the non-linear liquid dynamics together with STDP can be used for the unsupervised learning of image patterns. STDP, when applied to synapses connecting the input to a specific excitatory neuron, causes the synapses to encode a representation of the image pattern in the corresponding weights. This triggers the neuron to fire consistently for matching image patterns while STDP reinforces the synapses. Besides, the positive recurrent connection between pairs of excitatory neurons facilitates them to learn varying representations of a particular class of patterns. On the contrary, the inhibitory connections help separate the receptive field of different excitatory neurons. The proposed topology, in addition to being capable of learning different signal modalities, offers enhanced synaptic sparsity in comparison with two-layered fully-connected SNN  is the average spike count of the group of neurons in the ith liquid carrying the tag “j,” nij is the total number of neurons in the ith liquid with the tag “j,” spike_counti is the spike count of the kth neuron in the ith liquid with the tag “j” over the simulation interval, nliquids is the number of liquids in SpiLinC, and nclasses is the number of classes for a given pattern recognition task. Next, we combine the average spike count of the individual liquids to compute the resultant average spike count of the n-liquid SpiLinC for every possible input class (or tag) that is specified bywhere res_avg_spike_count(j) is the resultant average spike count of the groups of neurons in the n-liquid SpiLinC with the tag “j.” Finally, we predict the unknown test pattern to belong to the class (or tag) represented by the neuronal group with the highest resultant average spike count as described bywhere predicted test class is the class of the test pattern predicted by SpiLinC. Thus, we combine the spiking activity of each liquid that separately learnt characteristic input features to make a strong prediction regarding the class of a test pattern. SpiLinC effectively incorporates STDP-driven ensemble learning mechanism to provide a universal and scalable liquid computing model capable of unsupervised speech and image recognition.where We now provide insights on the sparsity and training efficiency of SpiLinC against an equivalently sized Liquid-SNN. Given a fixed number of training examples, SpiLinC is inherently capable of encoding more input information than the Liquid-SNN. In other words, the Liquid-SNN requires greater number of training examples to attain similar efficiency in learning. This is corroborated by our analysis, which reveals that SpiLinC with adequate number of neurons per liquid performs on par with the Liquid-SNN using significantly lower number of training examples. SpiLinC, by construction, offers enhanced sparsity than the Liquid-SNN. The favorable trade-offs render SpiLinC more amenable for realizing intelligent devices capable of adapting real-time to varied input data.We evaluated the proposed liquid computing models using BRIAN  at every simulation time-step that is specified byWe first describe the process of converting the input image patterns to Poisson-distributed spike trains, and then detail the corresponding mechanism for a speech input. For the handwritten MNIST digits, we converted the input pixel intensities to average Poisson firing rates constrained between 0 and 63.75 spikes per second. The average Poisson firing rate of an input neuron is then used to estimate its firing probability (tstep is the simulation time-step (0.5 ms used in this work). For instance, if the intensity of an image pixel is 255, the average Poisson firing rate for the corresponding input neuron is 63.75 spikes per second and its firing probability at every time-step is 0.031875. We then generate a random number between 0 and 1 at every simulation time-step and emit a spike if the random number is less than the probability of firing. On the other hand, for the TI46 speech dataset, we pre-processed the audio samples available in wave format (.wav extension) based on Lyon's Passive Ear model  synaptic connections  of the excitatory neurons are used to predict the class of a test pattern as described in sections 2.3.1 (for the Liquid-SNN) and 2.3.2 (for SpiLinC).#synapses) is calculated asWe use the classification accuracy (on the testing dataset), the number of input to liquid and recurrent-liquid synapses, and the number of training examples needed for convergence as the evaluation metrics for comparing the proposed Liquid-SNN and SpiLinC topologies. We report the classification accuracy of the networks across five different training and testing runs to account for the randomness in the input to liquid connectivity, the recurrent-liquid connectivity, and the input spike generation scheme based on Poisson process. The evaluation metrics, namely, the number of synapses and training examples, determine the degree of energy efficiency offered by the networks in neuromorphic hardware implementations. The total number of synapses in a network of given size determines the area footprint and read/write energy of the synaptic memory in neuromorphic hardware implementations. The number of synapses in the Liquid-SNN ] with the total number of liquids. We define the sparsity in synaptic connectivity offered by SpiLinC over a similarly sized Liquid-SNN as the ratio of the number of synapses in the Liquid-SNN to the number of synapses in SpiLinC. The final metric, i.e., the number of training examples needed for convergence, which depends on the liquid size (or the number of neurons per liquid), is determined experimentally by increasing the number of training examples until the classification accuracy (on the testing set) attains a maximum and saturates/deteriorates for any further increase in the training examples. For every two-fold increase in the liquid size, we find that the required number of training examples scales by a factor of 2 × to 4 × for the liquid sizes presented in this work across different pattern recognition tasks. The number of training examples dictates the total energy expended during the training phase including the neuronal computational energy, the read energy for loading the input to liquid and the recurrent-liquid synaptic weights every time-step, and the total number of weight updates carried out on the plastic input to liquid synapses that impacts the synaptic write energy. We use the same learning rate for both the networks to perform a fair comparison with respect to the number of training examples needed for convergence.where We employ the entire digit subset of the TI46 speech corpus  including the number of training examples listed in Table The proposed liquid computing models necessitate sparsity in both the input to liquid and recurrent-liquid synaptic connectivity to achieve efficient unsupervised learning. In our experiments using Liquid-SNN, we enforced 25% and 30% synaptic connections between the input and liquid-excitatory neurons, respectively for the speech and image recognition applications. Note that SpiLinC could have a higher percentage of input to liquid synaptic connections as shown in Table ee connections) or inhibitory neurons (ii connections). We find that excessive ee and ii synaptic connections lead to disproportionate increase in the spiking activity of the excitatory neurons while lowering the efficacy of the inhibitory neurons. This results in chaotic spiking activity within the liquid that is detrimental to learning. In this work, we used 0.5–1% ee and ii synaptic connections. However, we introduced a relatively higher percentage of synaptic connections (5%) between the excitatory and inhibitory neurons (ei connections) to sufficiently excite the inhibitory neurons. These neurons inhibit and help differentiate the receptive filed of various excitatory neurons. Hence, the inhibitory to excitatory (ie) synaptic connections ought to be substantial (20–30% used in this work) for enabling the liquid(s) to self-learn varied inputs in a multi-class pattern recognition task. It is important to note that the relative sparsity among the input to liquid and recurrent-liquid  connectivity controls the spiking activity within the liquid, which is critical for improved representation/feature learning and achieving competitive classification accuracy. The sparsity in the recurrent-liquid synaptic connectivity remains the same during training since the connectivity matrices are fixed a priori.Next, we discuss the sparsity requirement in the recurrent connectivity within the liquid  constituting Liquid-SNN (SpiLinC). Note that SpiLinC, by construction, offers enhanced sparsity in recurrent connectivity by breaking up a single large Liquid-SNN into an ensemble of multiple liquids operating in parallel. Our analysis on both Liquid-SNN and SpiLinC indicate that it is desirable to have limited synaptic connectivity between pairs of excitatory  instead of a single large Liquid-SNN (with 2x neurons). The individual liquids can effectively be trained in parallel using y (>y in practice) training examples. We showed that training the constituent liquids on disparate input segments causes them to self-learn characteristic input features. SpiLinC subsequently infers the class of a test input by identifying the underlying low-level features. This enables the two-liquid SpiLinC to perform on par or even outperform the Liquid-SNN  using 2 × (< 2 × in practice) fewer training examples. Higher the number of liquids forming SpiLinC, greater is the reduction in the number of training examples and faster is the convergence compared to Liquid-SNN. This is corroborated by our experimental results, which indicate that two-liquid SpiLinC, on average, learns with 1.95 × fewer training examples while four-liquid SpiLinC offers up to 3.3 × reduction in the number of training examples.We comprehensively validated the capability of SpiLinC to achieve faster training convergence (using fewer examples) compared to Liquid-SNN across speech Table  and imagThe scalability of SpiLinC with respect to the number of liquids (ensemble size) determines the trade-off between classification accuracy on one hand and sparsity together with training convergence on the other for a given task. Scaling up the number of liquids, in general, leads to improved sparsity and faster training convergence (as discussed in sections 4.1 and 4.2). However, the classification accuracy is comparable to (better than) that provided by Liquid-SNN only up to a certain ensemble size. The accuracy deteriorates rapidly for larger ensembles due to insufficient inputs received by the liquids. The number of inputs received by each liquid in SpiLinC is specified bypinp−e specifies the %synaptic connectivity between the input and liquid, inputdimension is the overall dimension of the input data, and noverlap is the number of overlapping inputs across ensembles. It is evident from (9) that the effective number of inputs per liquid decreases for larger ensembles (nliquids). This has a detrimental impact on the feature learning efficacy of the individual liquids, leading to degradation in the classification accuracy. Hence, the optimal ensemble size for a given task, which depends on the input dimension as shown in (9), needs to be selected so as to guarantee adequate number of inputs to each liquid. For the speech recognition task with a total of 39 frequency channels representing each speech sample, two-liquid SpiLinC  receiving inputs from 30 disparate frequency channels was determined to be the optimal configuration. On the other hand, we demonstrated up to four-liquid SpiLinC  for efficiently recognizing MNIST digit patterns, 28 × 28 in dimension.where ninp = 784, ne = ni = 6,400, pinp−e = 1, pee = pii = 0, pei = 1/ne, and pie = 1−pei. Further, the four-liquid SpiLinC with 3,200 neurons per liquid  would converge faster then the two-layered SNN with 6,400 excitatory neurons if both the networks were trained with the same learning rate. We find that the classification accuracy of the proposed models is lower than that achieved by the two-layered SNN because of the sparse recurrent inhibitory connections inside the liquid as explained below. When a test pattern is presented to the liquid, the neurons that learnt the corresponding pattern during training fire and only sparsely inhibit the remaining liquid neurons. This could potentially cause neurons that learnt different input classes but share common features with the presented test pattern to fire, leading to degradation in the classification accuracy. In order to precisely recognize a test pattern, it is important to attribute higher weight to the spike count of the correctly firing neurons and lower weight to the spike count of the incorrectly firing neurons. This can be accomplished by adding a readout layer and suitably adjusting the liquid to readout synaptic weights. We refer the readers to the Supplementary Material for performance characterization of SpiLinC with readout layer. Our results show that SpiLinC augmented with readout layer provides classification accuracy of 97.49% on the MNIST dataset and 97.29% on the TI46 digit subset.We compare the proposed Liquid-SNN and SpiLinC models with the LSM presented in Verstraeten et al.  that useFinally, we note that the deep learning networks (Wan et al., In this work, we proposed Liquid-SNN consisting of input neurons sparsely connected to a randomly interlinked liquid for unsupervised speech and image recognition. We showed that adapting the input to liquid synaptic weights enables the neurons in a non-linear liquid to self-learn time-varying speech and static image patterns. We demonstrated the generality of our Liquid-SNN by training it to infer digit utterances from the TI46 speech corpus and handwritten MNIST digits. Such a general computing model, capable of processing different signal modalities using a uniform self-learning methodology, is highly desirable for neuromorphic hardware implementations. However, the Liquid-SNN suffers from scalability issues because of the need to grow the liquid for enhancing the classification accuracy. We proposed SpiLinC that is composed of an ensemble of smaller liquids trained in parallel to learn distinctive input features. The ability of SpiLinC to recognize an input by identifying low-level features improves the classification accuracy. SpiLinC, with adequately sized liquids for efficient feature learning, provided competitive accuracy with added sparsity in synaptic connectivity and faster training convergence across different application domains. Therefore, SpiLinC is a universal and scalable computing model that can be used to realize intelligent devices capable of real-time online learning.GS wrote the paper and performed the simulations. All authors helped with developing the concepts, conceiving the experiments, and writing the paper.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.