Extraction of teeth is an important treatment decision in orthodontic practice. An expert system that is able to arrive at suitable treatment decisions can be valuable to clinicians for verifying treatment plans, minimizing human error, training orthodontists, and improving reliability. In this work, we train a number of machine learning models for this prediction task using data for 287 patients, evaluated independently by five different orthodontists. We demonstrate why ensemble methods are particularly suited for this task. We evaluate the performance of the machine learning models and interpret the training behavior. We show that the results for our model are close to the level of agreement between different orthodontists. Extraction of teeth is one of the most critical and controversial decisions in orthodontic treatment, largely because extractions are irreversible ,2. TheseIn this study, we aim to create an artificial intelligence decision-making model for the diagnosis of extractions using neural network machine learning. The primary objectives of the study were (1) to develop a decision-making model that simulates experts’ decisions of whether a tooth needs to be extracted or not based on standardized orthodontic pretreatment records (patient photographs and X-rays), and (2) to determine the knowledge elements required in formulating orthodontic extraction/non-extraction treatment decisions. It was expected that the diagnostic model created would match an expert’s diagnosis, both in binary decision-making (extraction vs. non-extraction outcomes), and in the more resolved decision-making process of which a specific extraction outcome would be followed (out of the 13 possible outcomes). This method would not only limit variability in decision-making in orthodontics but also limit the adverse effects of wrongly prescribed tooth extraction protocols. Additionally, this could also serve as a testing tool to train dentists and orthodontic students. Orthodontic pretreatment records in the form of extraoral photos, intra-oral photos, and cephalometric X-rays were collected. A panel of experienced orthodontists  evaluated the records individually and predicted the final outcome of extraction/non-extraction.The data consisted of 300 pretreatment patient records obtained from a private practice in Norwalk, Ohio, USA (orthodontist: C.A). Medical charts and conventional diagnostic records such as lateral head films , panoramic radiographs, facial photographs, and intraoral photographs were employed for each subject and screened by C.A for completeness. All subjects had full permanent dentitions except for the third molar, no abnormalities of the craniofacial forms or skeletal deformities, and no history of orthodontic treatment. Nineteen feature variables or elements that characterize orthodontic problems and are assumed to be important in deciding whether or not teeth need to be extracted were selected. This selection was based on the existing orthodontic literature. For all subjects, 5 experts , with an average experience of approximately 9 years among them, examined the records of each patient based on the pre-selected feature variables. Each expert also recorded his/her two most likely diagnostic outcomes (out of 14 available options) and categorized them as primary treatment and alternate treatment.The data were compiled and evaluated for potential errors by one of the authors (U.M). Data sets for thirteen patients were eliminated due to incomplete records and errors in data recording.Expert-provided features and decision data were analyzed using the R  platformWe collected data for 287 patients from five different experts. Each expert assigned values to 19 pre-selected diagnostic features based on cephalometric images and patient photographs in addition to selecting a primary and alternate treatment option. Experts were allowed to decide between one of the two binary outcomes: non-extraction, or extraction. Within the extraction plan, depending upon which tooth/teeth required extraction, the experts had to select one (specific) outcome out of the 13 different options (2–14) provided . CrucialWe used patient data from 287 patients. The age and gender distributions are shown in First, we wanted to establish the degree of agreement between the experts who evaluated the patients included in this study. If the multiple treatment plans selected by the different experts are considered as the gold standard for a machine learning method, the inter-expert agreement should provide us a practical higher limit on the accuracy to achieve. The agreement on the primary outcome of treatment between the different experts varied from 65% to 71% , and agrA number of different methods can be used to build a classifier for the prediction of orthodontic extractions. We considered twin problems of predicting whether to extract teeth or not, and the specific extraction treatment plan. As a classification problem, we had a discrete prediction, and used a neural network to learn the multinomial regression. Each output neuron learns to predict a specific extraction, taking inputs from the raw data. No hidden units were used. In addition, logistic regression was used for predicting the binary decision of extraction/non-extraction. The next step towards increasing the performance of the model was the use of two-way interactions in the logistic regression. Every pair of features was multiplied and used as additional features, generating a larger number of parameters. Although this was helpful in decreasing the error rates of the training sample, it increased the error of the test set, indicating that increasing the complexity of the model led to overfitting. Two-way interactions are more prone to overfitting due to the higher number of tunable parameters, and also increase the training time, as seen in In order to explore the benefit of including higher order interactions against overfitting with too many parameters, we implemented weight regularization for the two-layer dense neural network. We added a weight regularization penalty for the L1 norm, L2 norm, and an equally weighted L1 and L2 norm as an elastic net. While the training errors  monotonoSince the addition of additional parameters in the classifier (as seen in the logistic regression with multiplicative terms) leads to overfitting, we used an ensemble of classifiers to improve the performance. Ensemble methods are known to be resistant to overfitting. We trained random forest models using the standard algorithm, and varied the main hyperparameters to gain insight into the limitations for the performance.Each decision tree in the random forest was constructed using a dataset sampled with replacement from the training set. This process of bagging is one of the ways in which each decision tree attempts to capture a different aspect of the data. During the construction of each decision tree, a small number of features were randomly selected at each level and the one that was the most discriminating among the classes was used. The process continued until each node contained no more than a specific minimal number of samples.We varied these hyperparameters during the training of the random forest model. Further, even the relatively weaker hyperparameters  are strong enough to saturate the test set performance, while the training set performance continues to decrease with more complex models. Similar behavior is seen when looking at the prediction of the specific extraction  and the Since the random forest algorithm has an out of bag data sample for the construction of each decision tree, this out of bag error rate can be used to study the effect of adding each additional tree. The out of bag accuracy, which is a proxy for the accuracy on the test set, is visualized in When comparing all classifiers , it is cWe wanted to understand if certain predictors are especially influential in the treatment decision compared with others. Using a smaller number of predictors would help in minimizing the work required in the clinical setting in order to use the system. However, with a smaller number of predictors, any errors in the measurement of specific predictors may result in a larger error in the final diagnosis. Towards this end, we ran a predictor ablation study on the multinomial regression/neural network model. We measured the effect of deleting each predictor individually on the error rate evaluated over the test set, with a 5-fold cross-validation. Previous studies have approached this problem by utilizing machine learning using a neural network ,10. HoweOur research not only focusses on this binary decision but also on the thirteen other possible outcomes which highlight the specific tooth/teeth requiring extraction, creating a new artificial intelligence-based method to predict a plan from among a large number of possible extraction plans  based onSecond, after conducting a thorough review of the existing literature, we limited the diagnostic features to 19 most relevant predictors. The feature vector-elements adopted can be broadly classified into five major categories, i.e., sagittal dentoskeletal, vertical dentoskeletal relationship, transverse dental relationship, soft tissue relationship, and intra-arch conditions. Similar studies ,11 have We were able to exploit the diverse features using multiple machine learning models. We limited the number of features for ease of data collection and implementation of the method in the clinic. We were able to extract additional information from the interactions of these features and construct nonlinear predictors for regression using second-degree interactions. Although it resulted in overfitting, regularization was able to generalize the method and counter overfitting. Overall, we observed that ensemble learning and weight regression are both useful in such applications.One of the limitations of this study was that the treatment outcomes were confined to non-surgical orthodontic procedures only. Further, atypical extraction patterns like lower incisor extraction, second premolar extractions, and extractions dues to pathological reasons, among others, were excluded. In the current optimized model however, the elements that represented such features were not adopted. This is because the current study primarily focused on optimizing routine orthodontic diagnostic protocols.Finally, though the current model may not yet suffice to achieve complete agreement with human judgments, it should be noted that it has an advantage in that the system can independently improve its prediction accuracy by adding new patient records as templates just as orthodontists might increase their clinical knowledge and experience. This means the model will become more robust clinically for making decisions for individual patient treatment.We have shown that our limited feature set and machine learning algorithm is able to predict the extraction procedure to an accuracy that is approximately equal to that obtained from different experts. The use of an ensemble classifier (random forest ) model aA random forest ensemble classifier that simulates orthodontic tooth extraction/non-extraction decision-making was developed and confirmed to show a high performance, within the range of the inter-expert agreement.The features used for arriving at the extraction or treatment plan in this study can be ascertained by any dentist using this system in the clinic by evaluating the patient. Our future goal is to automatically arrive at these features from X-ray and photographic images. We plan to implement object recognition and landmark detection on images commonly used in the clinical practice. There is some existing work in the field of cephalometric X-ray landmark detection ,17, and


The detection and location of pipeline leakage can be deduced from the time difference between the arrival leak signals measured by sensors placed at the pipe access points on either side of a suspected leak. Progress has been made in this area to offer a potential improvement over the conventional cross-correlation method for time delay estimation. This paper is concerned with identifying suitable sensors that can be easily deployed to monitor the pipe vibration due to the propagation of leak noise along the pipeline. In response to this, based on the low-frequency propagation characteristics of leak noise in our previous study, polyvinylidene fluoride (PVDF) wire sensors are proposed as a potential solution to detect the pipeline leak signals. Experimental investigations were carried out at a leak detection pipe rig built in the Chinese Academy of Sciences. Their performances for leak detection were shown in comparison with hydrophones. It is suggested that with special considerations given to aspects pertaining to non-intrusive deployment and low cost, the PVDF wire sensors are of particular interest and may lead to a promising replacement for commercial leak noise transducers. High quality municipal utilities are fundamental to sustaining modern life and economic growth. Recently, a great challenge involving water loss in the stock of water distribution network, has confronts China’s buried infrastructure . AccordiLeak localization is the identification of the position of a suspected leak prior to excavation and repair, which is normally coordinated in parallel to leakage management, i.e., monitoring and control. Since the second half of the nineteenth century, many leak detection techniques have been developed, primarily including acoustic, chemical tracers, infrared thermography, ground-penetrating radars, and in-line leak detectors ,6,7,8,9.It is now common practice that the acoustic methods based on cross-correlation are relatively effective for pinpointing leaks in metallic pipes; however, difficulties are often encountered when applied in plastic pipes, even in an acoustically quiet environment, since leak noise in plastic pipes attenuates more heavily at lower frequencies. Substantial research has been carried out by the present authors for leakage localization in buried plastic pipes, with attention paid to improvement over the conventional cross-correlation method for time delay estimation (TDE) . MoreoveContinuous  pipe network monitoring systems have commenced more recently to take the leak detection and location methods into a whole new sphere, which requires the deployment of a wireless sensor network to work steadily in extreme environments of high humidity and high corrosion ,21,22. CIn this paper, PVDF wire sensors are developed and their performance for pipeline leakage detection and location are assessed. It is organized as follows. Acoustic leak detection equipment operates at low frequencies, where the axisymmetric (n = 0) fluid-borne (s = 1) wave is responsible for propagation of leak noise . Gao et As suggested in , the traα and For an elastic surrounding medium, the coupled equation of motion dominated by the s = 1 wave can be obtained by .(3)[Ω2−fined in . Settingve speed . FurtherBy substituting Equation (4) into Equation (2), the relationship between the internal acoustic pressure and the pipe radial displacements is obtained byg31 is much higher than other piezoelectric materials. The piezoelectric property of the PVDF has been studied in previous research ,31,32,33Q generated by the PVDF wire sensor is given by [S is the conductive electrode area. The performance of the PVDF wire sensor can be analyzed with a one-dimensional, lumped parameter approach, since the PVDF piezo film is very thin and the deformation in the direction of length is much larger than those in the directions of width and thickness. Thus, it is termed a wire sensor. The piezoelectric equation can be obtained bygiven by (9)Q=d31Since the PVDF wire sensor was wrapped around the pipe, the circumferential strain in the pipe wall is given bySubstituting Equations (7) and (10) into Equation (9), the pressure sensitivity of the wire sensor  is given byAs can be seen from Equation (11), the pressure sensitivity was proportional to the winding number of the pipe. It was also governed by the pipe material. For typical in-air metal and plastic pipes, as suggested in , the proWhen a pipe leak occurs, the leak noise will travel in both directions along the pipeline through the pipe wall and the water, which can be captured by leak noise transducers installed on the pipe wall and inside the pipe bracketing the leak, as shown in D is the distance between the two sensors; c is the propagation speed of leak noise given by Equation (6); and D and the propagation speed c, the problem of leakage localization was now transformed into the estimation of time delay k is the wavenumber given by Equation (4), which can be re-expressed as c and Referring to ained by (14)S12, and no pipe bulging occurs, the pressure sensitivity can be considerable for plastic pipes. This opens up new possibilities for the detection of water leaks in plastic pipes. However, in practical applications, a limitation for such sensors is that both aforementioned scenarios involving slippage and pipe bulging are likely to occur, particularly when the adhesives degrade over time, thus potentially reducing the pressure sensitivity dramatically. To the authors’ knowledge, their effects on the pressure sensitivity, to date, cannot be quantifiable. Further research is required to evaluate the performance for sensing the leak noise signal in real pipeline network. In this paper, the use of the PVDF wire sensors has been investigated for pipeline leakage localization. The performance of the wire sensor has been studied with respect to pressure sensitivity for monitoring the pipe vibration due to leakage. Leak noise measurements have been performed in a PE water pipe in the laboratory under controlled conditions. Theoretical analysis with experimental validations has confirmed the effectiveness of the wire sensors for the detection and location of pipeline leakage by comparing the performance of the PVDF wire sensors and commercial hydrophones. Two sets of measurements have been made, associated with pressure sensitivity and leak localization on the in-air and in-water pipe rig. From the measurements of pressure sensitivity on the in-air pipe, it has been demonstrated that good pressure sensitivity can be achieved by the selected PVDF wire sensors with possible improvement over the sensitivity by adopting higher winding numbers. To overcome the undesirable effects of the supports on the wave behavior, leak localization measurements have been done on the in-water pipe. Frequency analysis has clearly shown the effectiveness of the wire sensor for accurate location of the pipe leakage. Construction of a buried PE water pipe is under way, and more measurements in practical situations will be reported at a future date. This work may offer an alternative to leak noise transducers due to it easy employments and low cost.


The mine hoist sheave bearing is a large heavy-duty bearing, located in a derrick of tens of meters. Aiming at the difficulty of sheave bearing fault diagnosis, a combined fault-diagnosis method based on the improved complete ensemble EMD (ICEEMDAN) energy entropy and support vector machine (SVM) optimized by artificial fish swarm algorithm (AFSA) was proposed. Different location of the bearing defect will result in different frequency components and different amplitude energy of the frequency. According to this feature, the position of the bearing defect can be determined by calculating the ICEEMDAN energy entropy of different vibration signals. In view of the difficulty in selecting the penalty factor and radial basis kernel parameter in the SVM model, the AFSA was used to optimize them. The experimental results show that the accuracy rate of the optimized fault-diagnosis model is improved by 10% and the diagnostic accuracy rate is 97.5%. The head sheave is the key part in the multi-rope friction winder. The bearings of the head sheave are subjected to the heavy and complex loads. In case of failure, it will bring huge security risks and economic losses to the coal mine. Therefore, it is pressing to study the fault-diagnosis methods of head sheave bearings. Since the sheave bearings of mine hoist are characterized by low speed (40 rpm to 60 rpm) and heavy load , the impacts caused by the sheave bearing faults are weak. So, it is difficult to achieve ideal results with traditional fault-diagnosis methods. At present, the fault diagnosis of head sheave bearings mainly relies on workers to climb to the dozens of meters high platform, which is very dangerous, especially in extreme weather. Besides, monitoring the temperature of bearings is also very common in coal mine to get fault information, which indicates the bearing may have faults according to the excessive high temperature, but it cannot predict the exact position of the fault. The vibration-based fault diagnosis can avoid the shortcomings of monitoring temperature, and have been researched by many scholars by signal processing ,2,3,4. IThe fault-diagnosis method based on signal processing to extract fault characteristic information is defective as it needs to be executed and completed by professionals with professional knowledge and technology, and cannot provide the diagnosis results in time. It may not only delay the time to repair the fault, but may even cause serious accidents and casualties. Therefore, providing an intelligent fault-diagnosis method is of great significance for ensuring the safety of coal mine production. As one of the most intelligent and cutting-edge fields in the field of artificial intelligence, the application of SVM received increasing attention ,13,14, rFrom the above literatures, it can be seen that the fault-diagnosis research on sheave bearings are relatively few and lack the intelligent fault-diagnosis research on sheave bearings. Our aim is providing an intelligent fault-diagnosis method with high accuracy. The main factors affecting the accuracy of the diagnostic system are the extraction of feature information and the setting of model parameters. This paper used the energy entropy of the intrinsic mode functions (IMFs) containing the main fault information to establish feature vector, according to the fact that different bearing faults have different ICEEMDAN energy entropy, and uses the artificial fish swarm algorithm to optimize the penalty factor and radial basis kernel parameter in the support vector machine model to obtain the optimal diagnostic model. The experimental results show that the accuracy rate of the optimized fault-diagnosis model is improved by 10% and the diagnostic accuracy rate reached 97.5%.During the establishment of the diagnostic model, the feature information is related to the accuracy and reliability of the diagnosis results, and the extracted feature information must be able to reflect the nature of the corresponding state. The improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN), as an improvement on empirical mode decomposition (EMD), is an adaptive processing method for signals. Its essence is to decompose a complex signal into a finite number of intrinsic mode functions with transient frequencies. These IMFs contain the detailed characteristics of the signal and can essentially reflect bearing’s corresponding state.For rolling bearings, the faulty part may be the bearing inner race, outer race, ball, and cage. The different faulty parts will generate different vibration signals with different frequencies, which will result in different IMFs after ICEEMDAN decomposition. Therefore, it is an ideal choice to use the IMFs obtained by ICEEMDAN decomposition of the signal to construct feature information.The ICEEMDAN method proposed by Colominas in 2014 is the improvement on the ensemble EMD and complete ensemble EMD . The flokth IMF by EMD, In n IMFs and the residual after the ICEEMDAN decomposition, that is:The vibration signal of the rolling bearing can be expressed as the sum of the E1, E2, …, En of the IMFs is calculated respectively, that is:N is the number of sampling points of the jth IMF. Assuming that the energy carried by r (t) can be ignored, the total energy of the original signal is expressed as:The amplitude energy In order to make the data in the same magnitude, the amplitude energy of each IMF is normalized to facilitate subsequent calculations and reduce the impact of singular data, that is:Finally, the corresponding ICEEMDAN energy entropy is obtained:According to this method, the ICEEMDAN energy entropy of the vibration signals under the condition of normal bearing, ball fault, inner and outer race fault are calculated respectively. The results are shown in From γ and the radial basis kernel parameter σ2 need to be determined. Different parameter values have a great impact on the diagnosis results. Therefore, the artificial fish swarm algorithm (AFSA) . The average fitness and optimal fitness of the algorithm diagnosis results are shown in The parameters of the AFSA optimization algorithm are set as follows: the maximum number of iterations is 50, the size of the population is 10, the maximum number of foraging trials is 10, the congestion factor γ = 5.0702 and σ2 = 10.5966.It can be seen from γ and radial basis kernel parameter σ2 of the SVM model. Total of 240 sets of bearing feature vectors are input into the optimized AFSA-SVM model. About 80 sets of bearing feature vectors are used to test the generalization performance as the data to be tested. The classification result is shown in The optimal classification accuracy of SVM diagnosis model is obtained by using AFSA to optimize the two parameters penalty factor It can be seen from The diagnostic results of the SVM model without optimization and the optimized SVM model are compared, as shown in It can be seen from In order to verify the generalization ability of the AFSA-SVM algorithm, different numbers of train samples and test set were constructed. It can be seen from In order to verify the effectiveness of the proposed method, the least squares support vector machine (LSSVM) optimized by particle swarm optimization (PSO), combined with feature vectors constructed by ICEEMDAN energy entropy, was used to diagnose the same bearing faults. Besides, the feature vectors constructed by EMD energy entropy and EEMD energy entropy respectively, combined with AFSA-SVM model, are used to verify the effectiveness of ICEEMDAN energy entropy. The diagnostic process is the same as above. The classification results are shown in From An intelligent fault-diagnosis method for the hoist sheave bearing is presented in this paper. Because of the fact that the coal mine multi-rope friction winder is a large-scale revolving equipment and has strict safety assurance requirements, it is hard to carry out the field tests on the coal mine hoist bearings. The experiments were carried out on the hoist testing setup whose working principles and load characteristics are similar to the real coal mine hoist. Four states of the bearing were analyzed: the normal condition, the inner race defect, outer race defect, and ball defect of the bearing.The feature vectors were constructed by the energy entropy of the IMFs derived from ICEEMDAN decomposition based on the fact that different bearing faults have different ICEEMDAN energy entropy.γ and radial basis kernel parameter σ2 of the SVM were optimized by the artificial fish swarm algorithm in this paper. After optimization, the accuracy of the fault-diagnosis model can reach 97.5%, increasing by 10% compared with the accuracy without AFSA optimization.The SVM was used to diagnose bearings’ faults. In order to improve the diagnosis accuracy, the two parameters penalty factor This paper provides a method for the intelligent fault diagnosis of large-scale revolving machinery with low speed and heavy load. The disadvantage of this scheme is that the vibration data cannot be collected in some difficult-to-access places , while the advantage is that the incipient fault can be detected from the vibration data which are more sensitive to faults.In the future analysis, the accuracy of the intelligent fault-diagnosis model can be improved by more optimization algorithm and new feature-extraction methods.


The reconstruction of positron emission tomography data is a difficult task, particularly at low count rates because Poisson noise has a significant influence on the statistical uncertainty of positron emission tomography (PET) measurements. Prior information is frequently used to improve image quality. In this paper, we propose the use of a field of experts to model a priori structure and capture anatomical spatial dependencies of the PET images to address the problems of noise and low count data, which make the reconstruction of the image difficult. We reconstruct PET images by using a modified MXE algorithm, which minimizes a objective function with the cross-entropy as a fidelity term, while the field of expert model is incorporated as a regularizing term. Comparisons with the expectation maximization algorithm and a iterative method with a prior penalizing relative differences showed that the proposed method can lead to accurate estimation of the image, especially with acquisitions at low count rate. Positron emission tomography is an imaging technology that provides quantitative studies to detect, diagnose, and monitor treatment of different diseases such as hypernated myocardium, cancer, and many others ,2.The scan process begins by administering a radioactive substance, the radiotracer, to the patient. The substance is absorbed mainly by the target organs or tissues. The positron emission tomography (PET) data are then obtained by detecting the radiotracer distribution within the body. These acquired data are then processed by reconstruction algorithms to obtain the final image, which is presented to the medical research personnel ,3.Quality of the PET image depends of several factors: physical factors such as positron range, non collinearity and spurious events; hardware related factors such as crystal type and size and response time of the electronics; and the software or reconstruction algorithm used to estimate the final image .In this paper, we are interested in improving the quality of the PET images by using algorithms to reconstruct the image. Reconstruction algorithms can be broadly classified into analytic and iterative methods. Iterative methods are popular in PET due to their robustness and ability to incorporate prior data and noise statistics. Iterative methods are mainly based on the maximum likelihood expectation maximization estimator or the least squares model. However, when excessive noise is present in the acquired data, such as in low count acquisitions, most iterative methods have difficulties in obtaining an accurate estimation of the data and regularization techniques are required to stabilize the solution .In this paper a novel approach to reconstruct PET images is presented. Our approach is based on a regularized expectation maximization (EM) algorithm. Here, we propose to regularize teh problem by using a cross-entropy fidelity term and field of experts (FoE) priors , which aThe rest of the paper is organized as follows. In The Maximum Likelihood Expectation Maximization (ML-EM) algorithm takes into account the Poisson-based likelihood distribution of the data.I detectors of the scanner by the vector y is modeled as independent random variables and Poisson distributed with expectation J pixels. Here, the image will be ordered as a vector A. The probability The PET scanner detectors count pairs of events in coincidence throughout the entire ring. These counts are subsequently processed to form the final image. Here, we represent the counts registered by the The log-likelihood is obtained by combining Equations  and 2)  2) as=Duation (1∂JFoE(x)∂Then, we can generalize the idea of (MXE1 iterative scheme) to the proposed regularization of the cross-entropy (Equation )(16)xj software  . We usedIn this experiment, the efficacy of the reconstruction under low count conditions was evaluated. We used the reconstructed images from each method to generate selected surfaces to show graphically how close the reconstructed image was to the ground truth.To this end, we designed a cylindrical software phantom of polymethyl methacrylate with five rows of holes  of 2, 3, 4, and 5 mm in diameter. Each hole was filled with activity of 1:8 with respect to background. We were interested in low count reconstructions, since this means less radiation exposure for the patient. Similar to higher counts data, all methods evaluated had practically equal quality in the reconstruction. Thus, we ran a PET scan simulation at 5 M count using the Simset software. In our next experiment, we used the Digimouse anatomical atlas dataset described in  to desigWe simulated two realizations with different number of counts: 30 M and 5 M counts. The 30 M counts realization was taken as a ground truth. To quantitatively evaluate image quality in the Digimouse, a channelized Hoteling observer ,24 was uhttp:\\web.eecs.umich.edu\~fessler\, which is from a subject who was scanned on a CTI ECAT PET scanner. The raw sinograms were acquired at 160 radial samples and 192 angular samples; data were pre-corrected for delayed coincidences [In the next experiment, we evaluated the performance with measured data. We used data from cidences . Figure In this paper, a novel reconstruction method for PET images is presented based on a cross-entropy fidelity term. We propose to regularize the ill-posed problem by using field of experts priors. In this way, we can incorporate into the reconstruction process prior distributions from adapted filters to structure PET images and anatomical information.The experimental results show that the proposed method led to a better reconstruction performance than EM and CP. In an experiment with a phantom with rods of different sizes, we found an improvement in the recovering of the pixels of the smallest rods, showing that the proposed method could perform better in the reconstruction of small structures such as lesions. This was also observed in a second experiment with a phantom with lesions, where a ROC analysis of a observer detecting lesions was simulated and the proposed method outperformed EM and CP methods. In the experiment with measured data, our method obtained more contrast resolution, and visually the image of the proposed method had more defined edges than EM and CP. As future work, we plan to implement the algorithm with listmode data and take into account the additional information of time of flight, as well as construct a database of more images including PET/CT and PET/MRI.


We have recently developed the Unity Simulator for Epidural Insertion Training (USEIT) system that provides an innovative and relatively inexpensive virtual simulation approach for epidural training. This report describes the design and development process to produce the USEIT system. Epidural anesthesia is a regional anesthetic that blocks pain in a large area of the body and is performed by guiding a special needle into the epidural space in order to insert a catheter for the continuous infusion of local anesthetic . It is tThis technical report describes the design and development of a virtual simulation, known as the Unity Simulator for Epidural Insertion Training (USEIT), for epidural training. The USEIT was developed leveraging visual and haptic feedback employing consumer-level (low-cost) technologies including the Novint Falcon haptic controller , and 3D printing in conjunction with open electronics. With respect to prior work, the USEIT simulation system is similar to the virtual reality simulator developed by Brazil et al. . More spUSEIT developmentThe development process encompassed software and hardware components aimed at reproducing the epidural process. To determine the characteristics of USEIT, first, we defined the system’s inputs and outputs to perform the procedure. Inputs include: (i) capturing the position and orientation of the needle between the upper and lower lumbar spinous process, (ii) pushing the plunger flange to apply the solution in the syringe until loss of resistance (LOR), signifying entering the epidural space, is achieved, and (iii) menu inputs through the graphical user interface (GUI). Outputs include: (i) haptic feedback provided when inserting the needle and reaching the LOR, (ii) visual feedback providing information about the needle position and orientation towards the LOR, and (iii) user metrics for performance assessment after the simulation.Software DesignThe software development process included the definition of uses cases for the trainee to perform as presented in Figure Virtual EnvironmentThe virtual environment was developed to resemble an operating room prepared for the epidural procedure. Additionally, a virtual patient in a C-shaped position ready for the procedure is shown in Figure Haptics SimulationThe Novint Falcon haptic gaming controller due to its affordability and technical properties summarized in Table Gripper CustomizationThe Novint Falcon allows its end-effector (“gripper”) to be changed Figure . InterchOrientationA 6 DOF MPU6050 Inertial Measurement Unit (IMU) is attached to the custom-made gripper and controlled by a separate Arduino. Figure Needle Force-Feedback ModelThe force feedback model was separated into the following two components to better simulate haptic interactions while inserting the needle to reach the LOR: (i) parallel to the needle direction and (ii) perpendicular to the needle direction. Each component is calculated separately and then combined into a 3D output force command for the haptic device. The haptic feedback is then used to alter the virtual environment by additionally providing visual feedback allowing the trainee to guide the needle towards the LOR. A closed-loop system, receiving position and velocity data from encoders in the haptic device, and orientation data from an additional orientation sensor, is employed to create the interactive loop for the trainee to practice the procedure while getting feedback as presented in Figure Parallel Force ComponentThe geometry of each haptic-enabled object in the simulation is sent from Unity to Chai3D as either solid  or penetrable . During the procedure, a ray cast  determines the type of haptic collider object in the needle’s path. This translates into a one-dimensional array of tissue layers sorted by depth. In this model, two points are tracked: i) the contact point between the needle and the layer, and ii) the resting position of the layer under zero stress. For each layer, tension and friction effects are simulated employing a mass-spring-friction model to provide the sensation of needle insertion. Figure Each tissue layer includes fluid resistance, penetration force, and friction properties to increase realism. The spring stiffness value (k) was configured to mimic the force required to penetrate, the contact point between the needle and the layer . The sprPerpendicular Force ComponentThe perpendicular force model is applied when the needle enters a penetrable object such as skin. Since the needle orientation takes place within the 3D virtual space, we employed a 2D plane perpendicular to the needle direction to output a controlled force focused on the needle entry point. The position of the needle during the procedure in conjunction with that of the entry point are both projected onto the 2D plane to control the amount of force generated. The force response is controlled by a Proportional Integral Derivative (PID) controller, which takes this position as input and outputs a controlled force in 2D toward the needle entry point. A PID controller is a control loop mechanism employing feedback, which is used to compare and modulate outputs .Plunger and Fluid Resistance Part of the epidural procedure is the anesthetic administration, and to simulate this, a hand valve is fitted into a stepper motor that is connected to the syringe coupling. More specifically, a NEMA 17 size stepper motor is connected through a 1:4 gearbox with 2.5 mm bolts. The gearbox exists solely to make the rotation of the valve smoother and more reliable. Next, the stepper motor is connected (via a 6-pin connector) to an A9488 stepper motor driver with a 12 V power supply. The motor driver’s current limiting potentiometer is trimmed according to the specification of the motor. Finally, the stepper motor’s step-per-revolution is configured in the Arduino driver code. A 200 step-per-revolution motor is treated as 800 step-per-revolution due to the gear ratio. Using a hand valve also allows for partial opening and closing for varied levels of fluid resistance. In Unity, the fluid resistance control software queries the haptic simulation for data on the layer and the tip of the needle within, then uses its fluid resistance property to adjust the resistance level, normalized between zero (closed) and one (fully open). An overview of this system is provided in Figure Here, we discuss the areas of improvement for USEIT. Although formal testing has yet to conduct, we have presented USEIT to many content experts and interested stakeholders at numerous medical-based conferences, workshops, and meetings. including Simulation Canada’s Sim Expo 2019  and 2019 IEEE International Conference on Consumer Electronics (ICCE) [FidelityHere, the attention focused on two main areas, technology limitations, and the current implementation. With respect to technology limitations, occasionally the USEIT system physically oscillates the needle, distracting the user and breaking immersion. This occurs when the force applied to the system exceeds a constraint within the virtual environment, or when the device loses calibration. If immersion is broken, the realistic perception lowers as the system behaves in an unrealistic manner. This problem lies within the Novint Falcon libraries and how they handle large forces. Unfortunately, we have not found a solution for this as the same situation occurs with other haptic plugins. Calibration is performed every time the system is used to reduce this jittering. Regarding the current implementation, the use of the custom-made gripper and the hand valve was found innovative, reproducible, and meaningful with respect to increasing realism within the procedure. However, palpation of the ‘patient’s back’, which is a key step to determine the point of insertion was found missing during our demo sessions. Palpations were omitted from our simulation since adding another custom-made gripper for adequate palpation would have increased the system complexity as trainees would have to engage in assembling and disassembling the system, which can lead to malfunctions from wear and tear.DesignHere, the attention focused on the hardware and virtual environment design. On the hardware side, the USEIT system is a mosaic of different components connected together. At this point, all interactive components have been designed to mimic the epidural process. However, we are currently investigating alternative components, while troubleshooting the existing model to improve the 3D printing process to secure all electronic components and ease assembly and use. Figure  ErgonomicsDue to the quantity of space required by the gripper, USEIT distances the user from the screen, thus creating a gap that is unrealistic when compared to the real procedure, where the back of the patient is closer to the syringe Figure . The disHere, we have presented USEIT, a virtual epidural training system that employs affordable consumer-level technologies, and we have focused on the development process employing consumer-level technologies. Approaching USEIT prototyping from an accessibility and affordability point of view, allowed us to implement and explore shortcomings and opportunities of a haptics game controller, 3D printing, and open electronics. While all the components of USEIT are important, the Novint Falcon is the foundation of the project as at the time of developing the system it provided us with a set of technical features adequate for simulating the epidural process. While the Novint Falcon provides a suitable haptic user interface, we had to customize the gripper to add three additional degrees of freedom to allow us to track the orientation of the needle, in conjunction with the hand valve to enable pushing fluid through the syringe once attached to the gripper. On the software side, CHAI3D provided a reliable simulation model for replicating the needle insertion, which we accomplished by setting a set of layers and combined forces towards achieving LOR when reaching the epidural space. Furthermore, Unity provides a suitable game engine to create an immersive environment that represents appropriate visuals for recreating the procedure digitally. Finally, this process has shown that it is possible to develop such consumer-level systems in a manner that presents potential solutions to lower simulation costs that can help practice medical skills.Future directionsThe next phase of development will include face, content, and construct validity testing. In addition, expert opinions gathered during our informal testing sessions will be integrated to create an iteration of USEIT that will include both palpation and landmarking. The representation of what is happening in real-time will still be included on a computer monitor, for pedagogical purposes. Furthermore, we will explore virtual and mixed reality to increase immersion in addition to gamification to increase engagement and motivation.


Fluorescence-guided surgery with five-aminolevulinic acid (5-ALA) is the state-of-the-art treatment of high-grade gliomas. However, intraoperative visualization of 5-ALA under blue light remains challenging, especially when blood covers the surgical field and thereby fluorescence. To overcome this problem and combine the brightness of visible light with the information delivered with fluorescence, we implemented multispectral fluorescence (MFL) in a surgical microscope, a technique that is able to project both information in real-time. We prospectively examined 25 patients with brain tumors. One patient was operated on two different lesions in the same setting. The tumors comprised: six glioblastomas, four anaplastic astrocytomas, one anaplastic oligodendroglioma, two meningiomas, 11 metastatic tumors, one acoustic neuroma, and one ependymoma. The MFL technique with a real-time overlay of fluorescence and white light was compared intraoperatively to the classic blue filter. All lesions were clearly visible and highlighted from the surrounding tissue. The pseudocolor we chose was green, representing fluorescence, with the surrounding brain tissue remaining in its original color. When blood was covering the surgical field, orientation was easy to maintain. The MFL technique opens the way for precise and clear visualization of fluorescence in real-time under white light. It can be easily used for the resection of all tumors accumulating 5-ALA. Drawbacks of classic PpIX fluorescence such as hidden fluorescence, intraoperative changes could be overcome with the presence of additional white light in MFL technique. Malignant brain tumors and especially high-grade gliomas (HGG) are characterized by their aggressiveness and incurability. An influential phase III trial was able to show that mean survival of patients diagnosed with glioblastoma (GBM) is only 14.6 months when treated with radiotherapy and temozolomide . Apart fused for the visualization of other entities like meningiomas and/or metastases, although more appealing and strong evidence for its use in these lesions is needed  fluorescence and is strongly absorbed by blood cells, as it has a strong attenuation to hemoglobin , 13. EveTo do so, the multispectral fluorescence microscopy (MFL) was incorporated into a commercially available microscope . We have already investigated its use for the combination of indocyanine green (ICG) fluorescence and white light images in the therapy of vascular anomalies of the brain in both an animal model, and on a human cadaver model as well as in clinical use for vascular and tumor applications –17. SuchTo the best of our knowledge, no other observational study has used MFL for the intraoperative visualization of brain tumors accumulated with PpIX fluorescence. As the MFL mode uses the information obtained from the blue light image, we hypothesized that the intraoperative assessment of the fluorescence quality and contrasts in the MFL mode would match the fluorescence under blue light. Thus, outcome parameters of this study were the matching information of both fluorescent pictures and intraoperative usability of the MFL mode when assessing and resecting a brain tumor. With this study, we want to get first insights into the use of MFL as a possible tool that could diminish the technological and ergonomical disadvantages of the visualization of PpIX fluorescence under blue light for fluorescence-guided surgery of brain tumors.The present study was approved by the ethics committee of the University of Witten/Herdecke, Germany (Nr: 35/2017). All patients participating in this prospective observational study gave written informed consent and were recruited in the Department of Neurosurgery of the Cologne Medical Center, Cologne, Germany. The follow-up was performed in the usual manner for every patient that is treated because of a brain tumor in our clinic.Inclusion criteria for this study were:- Primary or secondary brain tumor on an MRI, regardless of WHO grading- no contraindication for the administration of 5-ALA- histopathological confirmation of a brain tumor (primary and secondary)- Written informed consent- Age 18–75 years although we included also elderly patients if the Karnofsky index higher than 80%.Exclusion criteria were:- Severe comorbid diseases that could negatively influence the participation in this study- Therapeutically uncontrollable arterial hypertension - Lung diseases that are related to complications during narcosis and anesthesia - Patient that received an attenuated vaccine 14 days or the influenza vaccination 7 days prior to their participation in this study- All comorbid diseases that could influence the participation in this study.n = 6), anaplastic astrocytoma (n = 4), and anaplastic oligodendroglioma (n = 1). Apart from that, we investigated the off-label use of 5-ALA in metastases from bronchial carcinoma (n = 5), metastasis from colon carcinoma (n = 3), metastases from breast carcinoma (n = 1), meningioma (n = 2), ependymoma , acoustic neurinoma (n = 1), metastasis from a myxoid liposarcoma (n = 1), and metastasis from tongue carcinoma (n = 1)  that were operated over the course of 22 months  between  (n = 1) .All patients received an oral dosage of 20 mg 5-ALA per kilogram body weight at least 3 h before the surgical procedure. Post-operatively, all patients were treated on the operative intensive care unit of our clinic before being moved to the general ward and kept in an area with low light exposition due to the phototoxic properties of 5-ALA.The MFL system, mounted on a conventional microscope , consists out of a fluorescence excitation filter covering the absorption spectrum of PpIX in broad range around 405 nm, but blocking the PpIX emission peak at 635 nm to acquire the red fluorescence signal with a separate fluorescence camera. The PpIX fluorescence signal is processed into a pseudo color image with high contrast to the operative field  and transparently overlaid to the stereoscopic, optical view in the eyepieces (Leica Captiview). In addition, the pseudo color fluorescence image can be recorded and observed as embedded information in the white light image on the monitor. The microscope control allowed switching directly from white to blue light or MFL mode or from blue light to MFL mode and vice versa for surgical workflow support and comparison purposes.Following the craniotomy, we continued with a sharp surgical incision of the dura mater. Once the brain parenchyma was visualized macroscopically, we started to continue the procedure under the operation microscope . If according to the MRI and intraoperative neuronavigation  the lesion was on the surface of the brain we continued under blue light in order to assess PpIX fluorescence and coherently visualize and identify the tumor. If the tumor was exposed, we changed to MFL-mode and observed the operation field and evaluated if the fused MFL image matched the different properties of the white light and blue light images. In tumors lying more deeply in the brain parenchyma we first operated under white light until the lesion was exposed. Then, we continued in the same fashion as described before.When we suspected slight bleeding inside the surgical field during resection under blue light we changed to white light and assessed bleeding. After that, we turned to the MFL-mode in order to evaluate both the fluorescent properties of the tumor tissue and the bleeding. To achieve hemostasis, we switched back to white light. We evaluated the surgical field in the MFL-mode only in the case of slight bleedings that masked the fluorescence under blue light in order to not jeopardize the patient. Larger bleedings were immediately treated under white light. We repeated this procedure in the cases where small bleedings occurred while meticulously resecting the lesion under blue light.ex vivo under blue light and then changed to MFL-mode for the assessment of its fluorescent properties and if the characteristics of both images match. After each operation, the operating surgeon (CC) assessed the raw data , which is calculated based upon the blue light fluorescence data. This was then compared to the MFL picture regarding matching fluorescence areas and shades.During final inspection of the cavity under blue light we changed to white light to do a final assessment of coagulated blood or small bleeding that both could conceal tumor tissue. Then we switched to the MFL-mode to assess if the bleeding sites were matching the ones identified under white light. After that, we turned to blue light again. If any previously masked lesions were then identified we resected them accordingly until no fluorescent tissue was visible. Following the final inspection, we continued with sufficient hemostasis and closure of the surgical wound under white light. At the end of the operation, we assessed the resected tumor As the software was installed in a surgical microscope the operative use of the MFL technique was easy to implement in the clinical setting. To correlate the overlay of fluorescence between the traditional blue light mode and the MFL mode the surgeon did an intraoperative assessment. Before switching to the MFL mode, the PpIX fluorescence under blue light was assessed for correspondence to the MFL picture.In all 26 surgeries, we were able to correlate the pink fluorescence under blue light with the “green” fluorescence in the MFL-mode. The intensity of the green indicated fluorescence was always similar to the intensity of pink seen under blue light. In contrast to the conventional blue mode, under the MFL mode bleeding did not cause any problem as the sources of bleeding were immediately identifiable and were treated accordingly. Thus, periodic switching back to the white light was not necessary during the resection under the MFL fluorescence mode, which facilitated resection and surgical convenience especially in the situations that were ergonomically or surgically challenging .Postoperatively, we reviewed the raw fluorescence signal video recordings and MFL video. Again, the perceived intensities and locations of pseudo colored green fluorescence overlay in the MFL mode corresponded perfectly with the single channel fluorescence recordings in all 26 surgeries. There was no instance when the natural color of the surgical field of view, reflections from the brain tissue or cerebrospinal fluid would impede fluorescence overlay visualization or create a false positive pseudocolored green fluorescence overlay.In three cases, PpIX fluorescence using the MFL technique was intensive but inhomogeneous see . In one In 3/25 cases a new neurological deficit was found after the procedure. One patient was operated during awake surgery. In one patient, the intraoperative monitoring was positive and thus the surgeon decided to leave a minimal tumor residual in the precentral region. This patient then presented with a new hemiparesis of the left arm (muscle strength 4/5).Regarding the degree of resection, postoperative MRI confirmed complete resection in 20/26 lesions. The extent of resection was determined using postoperative T1 weighted MRI scans with contrast. Those were compared with the preoperative MRI scans for residual contrast enhancement. In five lesions, a minimal residual was still present on the MRI. For one patient no postoperative imaging study was available.A 53-year-old male reported to our clinic with persisting headaches, amnesia, concentration deficits, confusion, and coordination deficits on the left side. This was accompanied by a tendency to fall to the left and insecure gait. The initial CT scan obtained showed a lesion in the right temporal-parietal region with significant surrounding edema. The cranial MRI scan revealed a lesion highly suspicious for GBM. We, therefore, scheduled the patient for the resection of the lesion. Prior to the intervention the patient received 20 mg per kilogram bodyweight of 5-ALA. Microsurgical resection of the tumor was performed with the modified microscope and concurring sonographic navigation and neuronavigation. The use of the MFL mode showed a coherent fusion of both white light and blue light images of the highly fluorescent tumor tissue and allowed for easy assessment of any bleedings occurring throughout resection. A tissue sample of the resected tumor was sent to the affiliated department of neuropathology and revealed a GBM. After the operation at the final neurological examination the patient presented himself with no neurological deficits and was totally mobile. Postoperative MRI showed a complete resection of the tumor.A 65-year-old male initially presented himself with temporary hemiparesis of the right side accompanied by nausea. An initial MRI showed a hyperintense lesion on T2 in the left medial cingulate gyrus. We scheduled the patient for a biopsy of the lesion, which revealed an anaplastic astrocytoma . After being discharged to his home prior to the elective surgery the patient presented himself through our emergency room with focal epileptic seizures of the right leg, namely the thigh region, causing local pain and cramps. We then decided to resect the lesion as soon as possible. Prior to the intervention the patient received 20 mg per kilogram bodyweight of 5-ALA. We planned neurophysiological monitoring of the right arm and leg during the operation to secure neurological function of the nearby supplementary motoric cortex. Microsurgical resection of the tumor was performed following by neurophysiological monitoring and neuronavigation. After the lesion was identified in the depth of the longitudinal fissure the use of the MFL mode showed a coherent fusion of both the white light and blue light images of the highly fluorescent tumor tissue and was able to assess bleedings occurring throughout resection. The resection was finished without any complications. After the operation, neurological examination revealed a light right leg hemiparesis, which was in remission on the day of discharge. Postoperative MRI showed a complete resection of the tumor.A 78-year-old female suffering from a known squamous carcinoma of the tongue came via the emergency room of our clinic because of a decline in her general condition accompanied by confusion and dizziness for the previous 3 weeks. The conducted MRI showed a necrotic and cystic lesion in the left temporal region with a midline shift. We, therefore, planned resection of the symptomatic temporal lesion. A CT scan obtained for disease staging showed several pulmonary lesions and potential lymph node metastases in the mediastinum as well as a lesion in the left gluteal region and inguinal lymph nodes, all highly suspicious of metastases. The patient received 5-ALA. The use of the MFL mode showed a coherent fusion of both the white light and blue light images of the highly fluorescent metastatic lesion tissue and was able to assess bleedings occurring throughout resection. The resection was finished without any complications and at the end no fluorescence was detected. The histological diagnosis revealed metastatic cells of the squamous carcinoma of the tongue. Postoperative MRI showed a complete resection of the tumor.Even though the use of 5-ALA in neurosurgical patients was reported for the first time almost 20 years ago, fluorescence-guided surgery is, on the one hand, standard of care for brain tumor surgery, while on the other hand, still a developing field, especially in the improvement of fluorescence visibility from the technical point of view. This is highlighted by the fact that 5-ALA has only recently been approved by the FDA  and thatTo the best of our knowledge, the present observational study is the first investigation that describes a promising new technique that could help to improve PpIX fluorescence-guided tumor surgery while being easy to implement in the operation room. As hypothesized, intraoperative blue light images matched the MFL images while the new visualization technique did not violate operation ergonomics. On the contrary, as we have shown in Furthermore, previously to this study we made the observation that this mode is able to sufficiently differentiate between the brain's physiological color and the artificial color chosen . This leIn a recent paper, it was shown that the intensity of the PpIX fluorescence is related to the light source used, which, according to the authors, could have implications for surgery . This isApart from the means used by the surgeon to cope with the problem of light intensity depending on the light source used, some technical points have to be mentioned and discussed. PpIX is excited with blue light of a narrow spectral range around 405 nm. Illuminated with such light the healthy tissue can be observed as a blue image. With increasing tumor cell density, the red fluorescence marking is added to the reflected blue light and colorizes the observed tissue detail from pinkish blue to strong pink (tissue with low up to strong PpIX fluorescence).This color shift is generally independent of the excitation intensity, allowing the interpretation for surgical resection as long as the color visibility is given, but with further decrease of excitation intensity the visibility of the observed color signal becomes too dark and recognition of the faint fluorescence in the resection area is more or less impossible.Comparing the PpIX MFL-white light fluorescence illumination used in the present study to the blue light fluorescence, the excitation of white light fluorescence is broad and comprising the wavelengths shorter than 400 nm up to 700 nm, excluding the PpIX emission peak around 635 nm for the detection of the PpIX fluorescence.Because the detected fluorescence intensity is related to the light source excitation level in the absorption range of PpIX and the optical parameters, the MFL system allows the control of the related effects as much as possible to provide correct fluorescence intensity and visibility for the combined digital anatomical white light and fluorescence image on the monitor and for the fluorescence image overlay in the eyepieces.In this observational study, we also report tumors other than HGG that showed PpIX fluorescence even though we applied 5-ALA in an “off-label” setting. Such lesions were different metastatic processes, meningiomas, acoustic neurinoma as well as ependymoma. Although in the literature, mainly HGGs are regarded as tumors that express the highly fluorescent PpIX and hence have fluorescent properties under blue light evidence exists that other brain lesions can also be visualized with 5-ALA, such as brain metastases . Of noteNonetheless, the present study bears a few limitations. Being an observational study, we were only able to get the first insight into the way the MFL-mode could be used in neurosurgery. All pictures obtained in the MFL-mode were matching the fluorescence pictures under blue light while, however, the fluorescence and its contrast to the surrounding brain tissue was subjectively perceived stronger in the MFL-mode as opposed to the blue light pictures. In other words, we did not include any objective data such as quantitative assessment of fluorescence intensities or histopathological analysis of the tumor zones and the areas that showed different fluorescence intensity. Hence, future studies would need to quantitatively compare blue light and white light pictures to the generated MFL-mode picture regarding factors such as fluorescence intensity, discrimination of different tissues in real-time, or/and simultaneously identification of anatomically relevant structures such as blood vessels, to name a few. Apart from that, it is on our next steps to compare the traditional way of operating with both blue and white light to the sole use of the MFL-mode toward clinical endpoints such as progression-free survival, neurological function, or degree of tumor resection.The MFL technique embedded on a classic surgical microscope opens the way for precise and clear visualization of brain tissue PpIX fluorescence in real-time under white light. It can be easily implemented in the resection of all brain lesions accumulating 5-ALA and producing PpIX. Drawbacks of classic PpIX fluorescence visualization techniques such as difficulty in identification of bleeding sources, operating in dark environment and the necessity to regularly switch off the fluorescence mode could be overcome with the presence of additional white light which allows for clear simultaneous visualization of the exposed brain in natural colors and PpIX fluorescence in green pseudocolor.All datasets generated for this study are included in the article/The studies involving human participants were reviewed and approved by University of Witten/Herdecke, Germany (Nr: 35/2017). The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.PP and PC drafted the manuscript. PC and AH designed the manuscript. All operations were carried out by PC and MN. All authors read and approved the final manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Several new programming languages and technologies have emerged in the past few decades in order to ease the task of modelling complex systems. Modelling the dynamics of complex systems requires various levels of abstractions and reductive measures in representing the underlying behaviour. This also often requires making a trade-off between how realistic a model should be in order to address the scientific questions of interest and the computational tractability of the model.temporal constrained objects, which facilitates a principled approach to modelling complex dynamical systems. Temporal constrained objects are an extension of constrained objects with a focus on the analysis and prediction of the dynamic behaviour of a system. The structural aspects of a neuronal system are represented using objects, as in object-oriented languages, while the dynamic behaviour of neurons and synapses are modelled using declarative temporal constraints. Computation in this paradigm is a process of constraint satisfaction within a time-based simulation.In this paper, we propose a novel programming paradigm, called temporal constrained objects. Simple neuronal networks were modelled by composing circuit components, implicitly satisfying the internal constraints of each component and interface constraints of the composition. Simulations show that temporal constrained objects provide significant conciseness in the formulation of these models. The underlying computational engine employed here automatically finds the solutions to the problems stated, reducing the code for modelling and simulation control. All examples reported in this paper have been programmed and successfully tested using the prototype language called TCOB. The code along with the programming environment are available at http://github.com/compneuro/TCOB_Neuron.We identified the feasibility and practicality in automatically mapping different kinds of neuron and synapse models to the constraints of Temporal constrained objects provide powerful capabilities for modelling the structural and dynamic aspects of neural systems. Capabilities of the constraint programming paradigm, such as declarative specification, the ability to express partial information and non-directionality, and capabilities of the object-oriented paradigm especially aggregation and inheritance, make this paradigm the right candidate for complex systems and computational modelling studies. With the advent of multi-core parallel computer architectures and techniques or parallel constraint-solving, the paradigm of temporal constrained objects lends itself to highly efficient execution which is necessary for modelling and simulation of large brain circuits. Modelling complex systems using computer languages has spanned a wide range of domains: from organs and organ systems to weather and atmospheric turbulence to economic systems and social networks. While it is the responsibility of the programmer to choose an appropriate paradigm for the problem at hand, conventional languages are limited in their ability to provide the right framework for a broad range of problems. Models for complex problems tend to be large and unwieldy, and hence it is critically important that the programming language used to program such models not exacerbate the problem with inadequate support. In this regard, imperative languages require more effort on the programmer, in providing the detailed data representation and algorithms, needed to solve a problem. This adds another layer of software complexity, especially when the problem to be modelled is a highly complex one.temporal constrained objects  and computational space (simulation control) can be represented with less implementation complexity. For modelling multiple levels as in enzyme or biochemical kinematics to neurons and neural circuits, a constraint-based solver that could handle several models of differential equation style mathematical abstractions was attempted in this study.Although the aforementioned simulators hide the implementation complexity from the modeller either through GUI or through modules and scripting, the software complexity of the simulator increases while satisfying these requirements. The computational architecture of the simulators handled the complexity and provided interfaces to the end user. Since temporal constrained objects is its amenability to parallelization. Modern programming paradigms are biased more towards many-core and multi-core programming. Current day simulation systems have become more reliant on high performance computing techniques. In computational neuroscience, a modelling study generally involves learning a domain specific language and then depending on the framework of this language to parallelize the programs. NEURON and GENESIS require modifications in the coding strategy to port the model to cluster computing environment while NEST almost transparently maps a model to multicore computers. Brian has little support for parallelization which limits its use for large scale systems  behaviour is the series variable, declared as:series type variable_name [initialization];One of the most useful features of v, the expression ‵v refers to the immediate previous value of v and v‵ to the next value. These operators can be juxtaposed  to refer to successive values of v in the past or future. A series variable may also be initialized by explicitly assigning values at specific time points.The series variable takes on an unbounded sequence of values over time, and temporal constraints are defined in terms of past and future values of the series variable. For every series variable V) and current (I), in the abstract component class holds the sequence of values at different instance of time. The source class generates the input voltage for the circuit, which varies sinusoidal with time. The classes resistor and capacitor inherit voltage and current from the component class. The constraints in each class define the laws of circuit elements: the resistor class incorporates Ohm’s law for resistance  circuit illustrates the basic constructs of the language . The sersistance ; and theacitance .capacitor class, i.e. the rate of change of voltage can be approximated by change in voltage in one unit of time. The parallel_comp and series_comp classes represent the two different ways of combining the components of the AC circuit. , the voltage-gated ion channels as electrical conductance (g) and the electrical gradient as reversal potentials. Then, basic voltage and current laws (Ohm’s and Kirchhoff’s laws) are employed to calculate the voltage across the membrane of the neuron. The cell membrane acts as a diffusion barrier and an insulator to the movement of ions. The lipid bi-layer of the membrane accumulates charges over its surface where the intracellular and extracellular solutions act as the conducting plates separated by the non-conducting membrane. The capacitive current cI, thus formed isC is the capacitance and V is voltage across the membrane. Ions flow into and out of the cell through ion channels. The flow of ions through these channels leads to resistive current flow into the neuron, which is represented using Ohm’s law:Gion is the conductance of ion across the membrane and Iion is the ionic current.A single neuron is often conceptualised using the biophysics of a neuronal membrane, represented by resistance-capacitance (RC) low-pass filtering properties wherein the lipid bi-layer of the neurons is modelled as capacitance  model and simpler phenomenological models like Izhikevich model and Adaptive exponential integrate and fire model were used in this study to explain the spiking mechanisms of neurons.Passive electrical properties persist in the neuron if the current is a sub threshold current or a hyperpolarizing current. Neurons are known to communicate to each other using stereotyped electrical signals called action potentials or spikes. The generation of action potential in the neuron can be explained by modelling the activation of ion channels. Detailed conductance-based models like temporal constrained objects to model the time-varying dynamics of a neural circuit as exhibited by the electrical activity of neurons and synapses. In modelling studies, elemental features of a neural system are abstracted into a conceptual model and are formalized into a mathematical form. A working embodiment of neural dynamics is created using computational models which is used to test and examine the modelled abstract. Chemical and electrical phenomena of neural systems were then simulated from mathematical representations  using voltage functions which are continuous over a time evolution ; and (2) using voltage reset functions which show piece wise state changes . ‘Modelling synaptic dynamics’ presents modelling the synaptic interactions between neurons using continuous functions of time.temporal constrained objects, the biophysical mechanisms  were represented as constraints of the class. Continuous state changes of variables were represented using simple constraints while voltage reset functions were represented using conditional constraints. Once the fundamental components of a neural network were modelled, the interaction between these components in a network  was incorporated. Biological fidelity of the models was tested to validate whether the model produces experimental outcomes.In modelling neurons and synapses with gNaMax and gKMax are the maximum conductance of Na+ and K+ ions, m and h are the probabilities of the fast and slow subunits of Na channel being open and closed states and n is the probability of the subunit of a K channel being open. Dynamical properties of these gating variables are also represented using differential Hodgkin–Huxley equations  model biV of the membrane. In the standard formulation of HH model, the rate constants α and β of the ionic gates are empirically determined as a function of membrane voltage V asHodgkin–Huxley model represents a system of four differential temporal constrained object implementation of the HH model, the properties of neurons, such as maximum channel conductance, reversal potential, or capacitance of the cell were represented as attributes of the class. Dynamically changing membrane voltage, the gating variables and the input current values were represented as series variables. The mathematical equations representing the relation between these variables were represented using quantified constraints and their initial values were represented using simple constraints  explain the behaviour of the neurons at a mechanistic level. Since such models have higher computational costs for complex circuits, we adopted two mathematical abstractions: Izhikevich model and Adaptive Exponential Integrate and Fire model .The Izhikevich model is expressed as:V is the membrane potential, u is the recovery variable, a represents the time scale of this recovery variable, b represents the sensitivity of u towards sub-threshold fluctuations in V, C represents the after spike reset value of V, d represents the after spike reset value of u. This implementation of a canonical model includes a nonlinear term 2V, and has been shown to reproduce multiple behaviours of neurons such as spiking, busting, adaptation, resonance and mixed mode behaviours.In the two-dimensional set of ordinary differential equations of the Izhikevich model , V is thThe Adaptive-Exponential Integrate and Fire (AdEx) model is expressed as:V representing the membrane voltage, C is the membrane capacitance, Lg is the leak conductance, LE is the resting potential, ΔT is the slope factor, tV is the threshold potential, rV is the reset potential, Isyn is the synaptic current, τw is the adaptation time constant, a is the level of sub-threshold adaptation and b represents spike triggered adaptation. The model implementation follows the dynamics of a RC circuit until V reaches tV. The neuron is presumed to fire on crossing this threshold voltage and the downswing of the action potential is replaced by a reset of membrane potential V to a lower value, rV.This model  represenv and the membrane recovery variable u, required a discontinuous change during constraint solving, the value being independent from the value of the variable in the previous time step. This change was controlled using the value of a series variable ‘flag,’ and using conditional constraints. It should be noted that this is not an imperative update of a state variable but rather the declarative specification of a value at a new point in time. Since the membrane potential Vm diverges towards infinity at a particular step and Vm need to be reset before this step, another series variable Varray was used to store a finite maximum upswing value for Vm.Since AdEx and Izhikevich models do not contain continuous-time equations, the membrane dynamics and its reset mechanisms in TCOB models were represented using conditional constraints. For example, AdEx type neurons were also modelled similarly (See gt)syn(. The changes were represented as continuous-time equations which represented the fluctuations in synaptic conductance changes attributed to current flow (gt)syn( = gmax · gt)synsyn( from 0 to gmax at time t0 (the time at which spike occurs) followed by an exponential term specifying the decay, with a time constant τ (The neurotransmitter release and intake mechanisms of the pre- and post-synaptic neurons were modelled using synaptic conductance ent flow . The synent flow .(13)\doulations . For exanstant τ .(14)\doThe exponential synapse model approximates synaptic dynamics with the rising phase shorter than the decay phase. A single-exponential function is a reliable approximation for several synapses where the channels instantaneously transit from closed to open states. Alpha synaptic models have been used for synapses with finite duration rising phases  while intemporal constrained object based implementation, synapses were also modelled using classes. Continuous conductance changes of the synapses were represented as constraints in each constituent class. Conditional constraints were used to represent the conductance changes before and after the stimulus onset. Synaptic currents were calculated by automatically evaluating the constraints in each class.In Behaviour of sub-cellular or cellular components in biological neural circuits is not entirely sufficient to understand network function, since the dynamics and complexity of the neural systems are known to be nonlinear   to a neuron was set as the sum of its synaptic currents using the constraint:N.Iin =Am.Iampa+Ga.Igaba+Nm.Inmda;The modelled neuron receives synaptic inputs through one This constraint automatically enforces the relation between change in membrane voltage of the neuron and the synaptic inputs it receives. A cluster of such neurons were simulated by creating an array of TCOB objects. Constraint solving and evaluation of these objects utilized the implicit parallelization of constraints from the constraint store .temporal constrained object based models for neural modelling, firing patterns of several types of known neurons of the cerebellum were reconstructed as in a previous study  library. The potential sources of performance inefficiency of TCOB are due to the layers of predicate calls arising from the systematic translation and also due to the repeated checking of consistency in the underlying constraint store. To alleviate these inefficiencies, we adapted and extended a partial evaluation technique (TCOB compiler translates the echnique  for optiechnique .Temporal constrained objects were able to successfully recreate substantive classes of neuroscience models related to time-varying behaviour using a declarative approach. With large-scale brain circuit models in mind, the compositional modelling requirement was tested by mathematically reconstructing the constituent components of neuronal network, i.e., neurons and synapses. Models that followed continuous-time functions, such as the HH type neuron model, as well as different synaptic models were implemented in temporal constrained objects easily. Special mention must be made of how temporal constrained objects were able to express voltage-reset models which exhibit piecewise continuous-time behaviour through a constant reset of the voltage. These models exhibit both discrete and continuous state changes, in contrast to the continuous-time behaviour of HH type neuron models. Such systems that exhibited discontinuous behaviour or choice, such as the Izhikevich- and AdEx-type neuron models, required injecting a discontinuous change during constraint-solving. Temporal constrained objects were able to elegantly support this capability using series variables and conditional constraints, i.e., whenever the condition was met for resetting the voltage, the value of the series variable at the applicable point in time is set appropriately. It should be noted that this is not an imperative update of a state variable but rather the declarative specification of a value at a new point in time of the series variable. Both continuous and voltage reset behaviour of the models implemented in temporal constrained objects, generated typical firing patterns of different neurons. Even though we have used manually defined synaptic connections in the example scripts, conditional constraints and dynamic typing in TCOB enables to use dynamic construction of objects based on the various conditions of the networks. Since temporal constrained objects also supports unbounded arrays, neurons can be programmed to receive dynamic stimuli generated from constraints. Since TCOB programming environment use SWI-Prolog built-in predicates to create random numbers, synaptic jitter and other forms of randomness in the network can also be modelled easily. Having been able to test that the tractability of employing temporal constrained objects as arbitrary neuron models, we perceive that temporal constrained object implementations allow continuous functions as in HH models or discrete state change of functions as in voltage reset models.Temporal constrained object-like frameworks are well-suited for modelling systems from which useful inferences are to be derived based on partial or incomplete information available at the current stage. In computational neuroscience of neurons and circuits, this feature enables the modeller to process information incrementally at each functional zone with in specialized brain circuits.Constraint-based systems accumulate and aggregate the partial information from the set of constraints. Moreover, the order of constraint specification does not depend on the final result, which permits the incremental addition of information to the constraint store at any point of execution without worrying about the computation state. temporal constrained object implementation, the modelled neural responses reproduced biologically validated and relevant outputs as observed in experiments. The identification of global constraints of the system are to be tested on a larger scale, i.e. at the population level. Debugging network models in such frameworks has been changed to a constraint satisfaction problem where the constraint-solving algorithms operate on the constraint representation of the network variables, their domain and a set of relations between these variables. Successively imposing constraints with the level of details expected makes the system automatically scalable in terms of its software representation.In our temporal constrained objects is not most efficient to compare with specialised simulators available for computational neuroscience, it is hoped that with novel re-implementation of the temporal constrained objects programming platform, we would be able to express models for large-scale reconstructions in the future. With new computing architecture and multiple core GPUs and CPUs, it is crucial to consider declarative modelling strategies that allow implicit parallelization. This re-implementation, however, pilots a general-purpose constrained object-based programming paradigm to declaratively express both the concept space and computational space of neuron models where the model evaluations are automatically handled by the computational engine.Although the current implementation of temporal constrained objects is a good candidate for modelling small- to medium-scale brain circuit structures. In order to build constrained-object-based simulators for large-scale networks, we propose studying the parallelization of temporal constrained objects. Temporal constrained objects with parallelization are a promising approach for representing the emergent properties of systems that otherwise is too complex to model at multiple translational levels.While declarative languages provide a much higher level of abstraction and are more scalable, the execution of declarative programs under early implementations faced performance bottlenecks. Since slight changes in the constraint formulation may lead to unpredictable changes in the performance of the system, stability of constraint formulation of a model has always been challenged. Similar limitations exist while applying cost optimization techniques . Over th10.7717/peerj-cs.159/supp-1Supplemental Information 1Code samples and descriptions of modeling implementation of neurons and synapses as temporal constrained objects.Click here for additional data file.