Drosophila melanogaster, has been used as a model organism for the molecular and genetic dissection of sleeping behaviors. However, most previous studies were based on qualitative or semi-quantitative characterizations. Here we quantified sleep in flies. We set up an assay to continuously track the activity of flies using infrared camera, which monitored the movement of tens of flies simultaneously with high spatial and temporal resolution. We obtained accurate statistics regarding the rest and sleep patterns of single flies. Analysis of our data has revealed a general pattern of rest and sleep: the rest statistics obeyed a power law distribution and the sleep statistics obeyed an exponential distribution. Thus, a resting fly would start to move again with a probability that decreased with the time it has rested, whereas a sleeping fly would wake up with a probability independent of how long it had slept. Resting transits to sleeping at time scales of minutes. Our method allows quantitative investigations of resting and sleeping behaviors and our results provide insights for mechanisms of falling into and waking up from sleep.The fruit fly,  Xu, et al. developed a quantitative method to characterize the sleep of flies by continuously tracking the fly activity. They find that the sleep state is distinguished from a short-rested state with an exponential distribution of sleep duration, which suggests a memoryless wakeup probability. Sleep disorder is associated with neurodegenerative diseases10 and an increased risk of cardiovascular disease12. As a simple yet powerful model system, the study of fruit fly sleep behavior has led to the discoveries of important genes and mechanisms, which are also conserved in mammals15. The flies’ sleep behavior is usually characterized as a long-term resting behavior16. The animals hardly respond to moderate stimuli in a prolonged resting period, and they perform sleep rebound after sleep deprivation17. Before flies fall asleep, there is a latent phase in which flies display the same rest posture as they sleep but reduce their consciousness gradually. Thus, it is hard to distinguish whether a fly is in the sleep latency phase or sleep phase with undisturbed observation. On the other hand, judging flies’ sleep state by disturbing flies’ sleep with stimuli is incompatible with measuring sleep time. To make some compromise, previous studies used 5 min as a threshold to extract long-term rest episodes as sleep bouts based on observing the wildtype flies’ response to stimuli17. However, a unique threshold for all flies is questionable, especially for studies including mutants that are supposed to change their sleep behavior either in the latency phase or sleep phase. For example, this standard will confound mutants with shorter latency and mutants with shorter sleep time18. This issue greatly hindered sleep behavior research in flies. The field is calling for accurate and quantitative methods to characterize the sleep behavior of flies.Sleep is a universal physiological state among species. It plays an irreplaceable role in many aspects of life, ranging from regulating the body’s metabolism20. A power law exponent is used to describe the individual fly’s short-term rest behavior and the exponent can be tuned by dopamine21. But for the long-term rest behavior associated with the sleep phase, the power law is not an accurate characterization of the temporal structure20. The long-term rest samples were usually ignored or simply regarded as a tail of the power law distribution, often due to the lack of statistics compared with that of the short rest samples.The short-term rest behavior of flies is fractal and embedded within active ones. Several studies suggest that the temporal structure of the behavior is scale invariantIn order to develop a systematic framework of quantitative characterization of flies’ sleep behavior including both sleep latency phase and sleep phase, we established a device to monitor fly behavior continuously with high spatiotemporal resolution. Our datasets enriched the long rest behavior of flies. Our work demonstrated that the temporal structure of rest behavior included different phases, clearly showing that the state of the fly in long-term rest was distinct from that of short-term rest. The long-term rest was memoryless of time, resulting in an exponential distribution of sleep duration. Including the power law distribution in the latent phase, we concluded a generic sleep pattern of the fruit flies. This pattern described an intrinsic law of the whole sleep process and gave parameters to characterize single fly’s sleep properties quantitatively. The parameters of the sleep pattern accurately captured the sleep architecture at different ages and in different environments, as demonstrated by the aging-associated and light-associated changes. Thus, the pattern provides a quantitative method for comparison between different scenarios, as well as insights into the mechanisms of sleep.Drosophila Activity Monitoring System, DAMS) to monitor the locomotive trajectories of flies. It improved the accuracy of determining the rest time and sleep behavior. It monitored the flies day and night continuously. Our experimental setup included infrared lighting, computer, and observing chambers  Fig. .Fig. 2Qut. Due to the power law decay when t was small, the probability function had a linear relationship on double logarithmic plot  the switching rate from rest to movement at instant moment t, we get:The meaning and implication of the power law distribution and that of the exponential distribution were completely different. If we denote as K, above which the fly entered another phase with an exponential distribution.This form of switch rate is universal for power law distributions, that is, after the fly entered the resting state, the rate of switching from rest to motion is inversely proportional to the time it has rested. In other words, the longer a fly stayed at rest, the more likely it would continue to rest, implying that the fly had a memory of the rest duration. Obviously, this state cannot be continued indefinitely, otherwise the fly would never move again. Consistent with this, the duration of rest obeys a power law distribution only below a threshold period For the exponential distribution, the switch rate takes the formNote that in previous studies of flies, a rest interval ≥ 5 min was used as a criterion for sleep phase. Here we found that there was indeed a state transition in resting behavior at around this time scale. The relatively rare long-term rest behavior corresponding to sleep displays a different statistical characteristic compared with the short-term rest behavior.β, the more probable the fly would move again in a resting state. We called β as the active index; (2) sleep phase: fly entered a long period of rest, characterized by an exponential distribution of rest duration. The parameter λ of the exponential distribution indicated how hard a fly would wake up spontaneously in the sleep phase. This parameter can be used to represent the average sleep duration in the sleep phase. From the transition time between the two distinct resting patterns, we got the parameter K as sleep latency. With the K threshold, we could count how many times the fly fell asleep (N) and estimate the total time  the fly spent in the sleep phase (λ × Ν).In summary, we established a general rule of sleep in individual flies, characterized by the probability of rest duration or equivalently by the switching rate from rest to motion Fig. . This moβ, λ, K, N, and Total characterizing the pattern could be different for each individual fly and may vary between day and night, and between male and female . Thus, sleep in the light phase is more fragmental. The male had comparable total sleep time for the light phase and dark phase, while the female mainly slept in dark time  of the sleep phase varied slightly from 1D to 2D chambers  in the dark phase. Notably, the active index in the light phase consistently decreased for both males and females in the 2D chambers  were compared with the young groups (5 days post eclosion).The aging process causes loss of sleep consolidation in elderly human and flies27. Correspondingly, the 5 parameters of sleep did not significantly change in older male flies  increased  and those in normal light-dark condition (LD). The flies transferred to the DD condition increased their activity dramatically , which integrated the changes in sleep bouts (N) and sleep duration (λ), decreased both in male and female flies in the light phase of the DD condition compared with that of LD condition   of the females decreased under the DD condition  Fig. , which lλ) Fig. . We alsoion Fig. . Thus, tTaken together, these results indicate that the sleep pattern is a widely applicable metric to quantify the individual fly’s sleep behavior. It can accurately capture the difference when the experimental scenario changes.Noting that the active index tended to change differently from the other sleep parameters, we investigated the correlation between the parameters. We used the rank correlation (Spearman’s correlation coefficients) to quantify the correlations. The dataset consisted of all the samples from different settings and conditions discussed before .λ, K, Total) were in one group, and the active index (β) and sleep bout (N) were in the other group. It indicates that the changes in Total mainly depended on the changes in sleep duration  into two groups Fig. . The sleion Fig. . The numThe active index was in general negatively correlated with all other parameters in the day. That is probably because both the active index and the other sleep phase parameters are regulated by light. On the other hand, the active index was uncorrelated with the sleep duration, the total sleeping time, and the latency at night  is often associated with a simple two state transition  with a memoryless transition rate. On the other hand, a power law distribution of time durations (as for the rest time in the latent phase) is usually more complicated and can be generated by multiple mechanisms. It is known that the human brain has a few different states during sleepL possible states during rest  while the parameter a related to the active index β  and bipolar II patients (β ≈ 0.7) compared with a normal value (β ≈ 0.9)36. In our experiments, the power law exponent for flies was around 0.7–1 (instead of 0.37 as measured previously19), which is quite close to the ranges in mice and humans. Furthermore, the sleep duration of mammalians was found to be exponentially distributed by using EEG measurement37. Our results in flies suggest that this pattern of rest and sleep can be more universal. Even though the sleep state of flies is short of accurate physiological definition as in mammals, it is still possible to define sleep state quantitatively at the behavior level through the sleep pattern we found. With the five parameters , one can obtain quantitative information about each fly’s rest and sleep behavior. Given the shared universal pattern with mammals, the fly could be used as a model organism to study rest and sleep behavior.In the past years, power law distributions of rest duration were also observed in mammalian species including mice and humans38. It considers the relationship between the precious sleep state and the next sleep state and captures the properties of stage transition of sleep behavior. It is also a promising model for sleep behavior analysis in flies, and as reported the probability of initiating activity (P(Wake)) and the probability of ceasing activity (P(Doze)) could give a relative measure of sleep39. But when applying the DHHM model, those studies hypothesized that the current state is only affected by the previous state in the last time window (short memory). This hypothesis has never been tested biologically and is contrary to the observation that the flies’ rest behavior has a power law decay. A power law decay implies that the current state is dependent on how long the animal has been resting, i.e., the state has a memory. From our analysis, we found that only the sleep behavior has an exponential decay, suggesting that the HHM model with a constant transition rate is probably applicable only in the sleep phase. The sleep pattern we found can also explain why P(Wake) changes along the day and especially showed two peaks around ZT0 and ZT12 in the previous study . Around those times, the flies move frequently and have mainly shorter rest time, the average level of P(Wake) should then be larger according to P(Wake) ≈ β/t. Comparing with the previous probabilistic analysis method, the sleep pattern studied here gives a more straightforward framework to quantify the whole sleep process of flies.The Discrete Hidden Markov Model (DHMM) has been used for sleep staging analysis to aid the understanding of sleep behavior in humansβ) of a fly was correlated with how many times the fly fell asleep (N) but not with its sleep duration (λ). It implied that the two phases of the sleep pattern are associated with two different aspects of sleep: sleep onset and sleep maintenance. It has been found that dopamine signaling regulates the active index21 and that a variety of genes and neurons are related to sleep behavior15. Which circuits can tune the sleep pattern parameters is still an open question.We found that the active index  on standard yeast media with 12 h:12 h light: dark LED light cycle. Virgin flies were collected since they were born and grouped in plastic tubes. For all experiments, every single fly was put into the observing channel separately under short COThe 1D chamber is 5 mm wide and 78 mm long, the food will occupy 15–20 mm of the length. The 2D chamber is an ellipse with 50 mm longer axes and 30 mm shorter axes. The depth of the Z-dimension of all the chambers is 2.5–3 mm. The fly walked on the ceiling sometimes, but most of the time it stayed or walked on the ground. Our observing chambers made of two parts, acrylic shaped base and glass plate above.We simplified the image processing problem by segmenting the full images into partial images with only one fly in each chamber. We use difference images from the subtraction of the reconstructed background image to detect the location of the fly.For each light phase or dark phase, we reconstructed the background picture by calculating the mean grey value of the first frames of each hour . We can extract the area where the fly moves around by performing morphological opening on the greyscale background picture since the designed chambers were darker than the surroundings.x(t), y(t)] of the locations, and the y-coordinate is referred to the longer axis of the 1D chamber or the 2D elliptic chamber.Only the difference of the pixels in the moving area was considered. Since the fly was the only moving object, we took the top 100 pixels that have the most difference. We then produced a binary image of the same size as the partial image with value 1 at the top 100 pixels. The largest connected component in this binary image was found as where the fly was. The centroid of this largest connected area was used as the fly’s location in the current frame. The trajectory of the fly is a time series [The rest behavior is extracted from the trajectory. At each location of the trajectory, we calculated when the fly reached the location and when the fly left it. If the fly stayed at the same location for more than 1 second (25 frames), the fly had been resting at that location in that time interval.To mimic the previous measurement method of circadian and sleep profile, DAMS, we used the midpoint of y-coordinate of the locations as an imaginary infrared beam. This midpoint divided the whole chamber into two parts. Every time the fly moved from one part to the other, it was regarded as crossing the infrared beam for once.The raw videos were first transformed to avi format then processed with the Image Processing Toolbox provided by MATLAB 2019a.The rest events of each fly were classified by the light phase and dark phase. That means, we got the distributions of the rest time in the light phase and dark phase, respectively. The samples for the light phase distribution consisted of all the rest events when the light is on in the 3 shooting days, while the dark phase distribution consisted of all the rest events when the light is off in the 3 shooting days. In the paper, the data of the light phase was labeled as “day”, and the data of the dark phase was labeled as “night”.41 to estimate the exponent of power law distribution. The lower bound of the power law behavior was set as 3 s. We found the cutoff of power law distribution and the start of the exponential decay with one-sample Kolmogorov-Smirnov test. We scanned the K value from 50 s. For each K, the rest samples longer than K were subtracted by K first and then tested with the null hypothesis that the data comes from an exponential distribution with the mean λ. The smallest K that made the test fails to reject the null hypothesis at the 10% significance level was the sleep latency parameter K for the individual fly, and the corresponding λ was the sleep duration parameter.We used the MATLAB packages written by Aaron ClausetN indicates how many times the fly slept (was in rest longer than K) in 3 days in the experiment, which is also the total sample size of the exponential decay. We calculated the total sleep time of a fly as N × λ. The total sleep time does not include the time when the fly was still in the sleep latency phase.The parameter 42.All the code for the data analysis and simulation is deposited and accessible on Zenodot test was used to determine whether the two data sets came from distributions with equal means. The samples assumed from populations with equal variances were analyzed by a two-tailed Student’s t test. The samples assumed with unequal variances were analyzed by two-tailed Welch’s t test. The test was performed with the embedded MATLAB function “t test2” with 5% significance level of the hypothesis. The p value of each test was indicated within the figures. Results were presented as mean ± standard error of mean (SEM). The total number (n) of individual flies for each experiment was indicated within the figures and in the legends. A minimum of three biologically independent repeats was used for each experiment. The flies were randomly sampled.The two-sample Further information on research design is available in the Peer Review FileSupplementary InformationDescription of Additional Supplementary FilesSupplementary Data 1Reporting Summary


In the emerging field of 3D bioprinting, cell damage due to large deformations is considered a main cause for cell death and loss of functionality inside the printed construct. Those deformations, in turn, strongly depend on the mechano-elastic response of the cell to the hydrodynamic stresses experienced during printing. In this work, we present a numerical model to simulate the deformation of biological cells in arbitrary three-dimensional flows. We consider cells as an elastic continuum according to the hyperelastic Mooney–Rivlin model. We then employ force calculations on a tetrahedralized volume mesh. To calibrate our model, we perform a series of FluidFMThe online version of this article (10.1007/s10237-020-01397-2) contains supplementary material, which is available to authorized users. The dynamic behavior of flowing cells is central to the functioning of organisms and forms the base for a variety of biomedical applications. Technological systems that make use of the elastic behavior of cells are, for example, cell sorting   flow, we investigate the shear stress dependency of single cell deformation, which we compare to the average cell deformation in suspensions with higher volume fractions and show that our results in the neo-Hookean limit are in accordance with earlier elastic cell models  system that combines the AFM with the FluidFMo et al. . All expAn example micrograph of the experiment before compression is shown in Fig. The experimental setup of the previous section is easily transferred and implemented for our cell model: The undeformed spherical cell rests on a fixed plate while a second plate approaches from above to compress the cell as depicted in Fig. We use meshes consisting of 2000 to 5000 vertices and about 10000 to 30000 tetrahedra to build up a spherical structure. More details of the mesh and its generation (Sect. S-2.4) as well as the algorithm (Sect. S-3) are provided in the SI. In our FluidFMWe use the compression simulation setup as detailed in Sect. In this section, we compare our simulations to axisymmetric calculations using the commercial software Abaqus and validate our cell model with further experimental data for bovine endothelial cells from  flow setup and compare the steady cell deformation to other numerical and analytical cell models of Gao et al. , Rosti ecription  to two fcument}D  of our ix-z-planes at x-direction, as depicted in Fig. H of the parallel planes and the constant shear rate x and z. A single cell is placed at the center of the simulation box corresponding to a volume fraction of The first simulation setup, a single cell in infinite shear flow, is realized by choosing a simulation box of the dimensions o et al.  for a thA possibly even more intuitive way to measure cell deformation is the net strain of the cell which we define asThe second simulation setup, implemented to investigate the multiple particle aspect of our model, consists of 4 (8) cells in a Figure We presented a simple but accurate numerical model for cells and other microscopic particles for the use in computational fluid-particle dynamics simulations.The elastic behavior of the cells is modeled by applying Mooney–Rivlin strain energy calculations on a uniformly tetrahedralized spherical mesh. We performed a series of FluidFMBy coupling our model to Lattice Boltzmann fluid calculations via the Immersed-Boundary method, the cell deformation in linear shear flow as function of the capillary number was found in good agreement with analytical calculations by Gao et al.  on isolaThe presented method together with the precise determination of model parameters by FluidFMSupplementary file1 (PDF 836 kb)Below is the link to the electronic supplementary material.


Rank orders have been studied in evolutionary biology for almost a hundred years. Constraints on the order in which mutations accumulate are known from cancer drug treatment, and order constraints for species invasions are important in ecology. However, current theory on rank orders in biology is somewhat fragmented. Here, we show how our previous work on inferring genetic interactions from comparative fitness data  is related to an influential approach to rank orders based on sign epistasis. Our approach depends on order perturbations that indicate interactions. We apply our results to malaria parasites and find that order perturbations beyond sign epistasis are prevalent in the antimalarial drug-resistance landscape. This finding agrees with the observation that reversed evolution back to the ancestral type is difficult. Another application concerns the adaptation of bacteria to a methanol environment. Rank orders of genotypes with respect to fitness are central in evolutionary biology. Key concepts, such as peaks and mutational trajectories, refer to an underlying rank order. The impact of suboptimal peaks has been discussed ever since fitness landscapes were introduced . A majorThe concept of sign epistasis introduced by The squares representing a two-locus subsystem . Differently expressed, in addition to single mutations it is sometimes useful to consider the effects of double mutations and any higher order mutations, as well as replacements sistance , whereassistance .As in We consider biallelic For two-locus systems, the expression For a systematic treatment of order perturbations, it is necessary to write the information in compact form. Notice that a rank order where wo peaks . In partIt is straightforward to a check that a two-locus system has sign epistasis exactly if at least one of the following expressions is negative, and reciprocal sign epistasis if both of them are negative .(1) is negative, then the system has sign epistasis, and if expression (4) is negative, the system has an order perturbation of size 2. The short blue arrows in Remark 1. The relation between signed circuits and sign epistasis can be summarized as follows for A rank order of the genotypes 00, 10, 01 and 11 with respect to fitness implies that From the information that the rank order implies The system has reciprocal sign epistasis if both expressions (1) and (2) are negative, and sign epistasis if at least one of them is negative.Remark 2. A similar observation holds for signed circuits and any Each case of sign epistasis is associated with a signed interaction for a circuit in More generally, each rectangular perturbation is associated with a signed circuit interaction for a circuit in Each circuit in In particular, in order to check a three-locus system for perturbations it is sufficient to check the signs of 24 expressions, including expression (3) and (4), associated with 12 circuits .Plasmodium vivax . However, an investigation of the three types of replacements described could be conclusive. In particular, similar proportions of unexpected results would constitute a strong argument against additive fitness.* is an arbitrary sequence. For an Rank orders and perturbations are important for problems beyond natural selection. Constraints on the orders in which mutation occur are known for development of cancer drug resistance . In brieThe problem of determining if rank orders are compatible with additive fitness is already interesting for For the fitness landscapeAs remarked in Theory on rank orders and additivity has been developed independent of biological application and is still an active research area . In prin=5 exist   provided one assumes the Strong Selection Weak Mutation regime . If the It would be interesting to determine the extent to which rectangular perturbations are helpful for relating local and global properties of fitness landscapes, similar to results on sign epistasis , and alsFor a biallelic The four vectors are linearly dependent sinceThe dependence relation corresponds exactly to the linear formminimal dependence relations, in the sense that each proper subset of the vectors (with non-zero coefficients) are linearly independent.Similarly, one can form vectors in There are in total 20 circuits for In this terminology, one can describe ractions .To count rectangular perturbations, one needs to find all rectangles with vertices in an Theorem 1. The total number of  rectangular perturbations for an Moreover, the number of rectangular perturbations of size Corollary. A complete investigation of sign epistasis for an Lemma 1. For Stirling numbers of the second kind Proof. The first formula holds since there are Each partition corresponds to six alternatives, which completes the argument.Lemma 2. Let Proof. For each vertex From the vertex We can now prove the main result.Proof of Theorem 1. The total number of  rectangular perturbations for an A rectangle with vertices in the The second part of the theorem states that the number of rectangular perturbations where exactly The positions of the As mentioned, a single signed circuit may correspond to two cases of sign epistasis. Indeed, a two-locus system with reciprocal sign epistasis is an example. It is thus of interest to identify all order perturbations, rather than identifying signed circuits only. Theorem 1 and its proof indicate how one can find all order perturbations. In particular, the complete list for identifying rectangular perturbations for Different rank orders may differ only 'by labels’. For instance, the rank ordersTwo rank orders are considered equivalent if a cube isomorphism induces a map between them. Differently expressed, for any given rank order (for a two-locus system), one can obtain an equivalent order by assigning the label 00 to a genotype of choice  and then 10 to one of its neighbors . After that, the adjacency condition determines the labels of the remaining genotypes. The new rank order is identical to the original one, except that the genotypes have new labels as described. It follows that each equivalence class consists of For general To determine whether a specific rank order is compatible with additive fitness, one has to solve a system of inequalities. For simplicity, we illustrate the argument for a two-locus system. Consider the order Additive fitness would imply The case As explained, in principle, one can check whether a given rank order is compatible with additive fitness. However, the theoretical understanding of rank orders and additive fitness is still limited.As noted in the main text, rank-order methods are sensitive for measurement errors. A sufficient number of tests is necessary for reliable results, and elementary probability theory provides some guidance. If, for instance, genotype Research on inferring rank orders from pairwise comparisons has a long history because of applications to sports and games . HoweverSimulations. The upper graph in  In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.Acceptance summary:eLife, where it was shown how the sign of certain epistatic interaction coefficients can be inferred from partial rank orders of fitness values. The purpose of the manuscript is to clarify the relation of this inference with the concept of sign epistasis; Crona et al., 2017, had shown that sign epistasis was a sufficient, but not necessary, condition for rank order to be informative about at least some signed interactions. Here the range of informative rank orders is extended by introducing the concept of rectangular perturbations, which generalizes the concept of sign epistasis by asking for the background dependence of the effect of mutational events that modify several loci at once. In the revision, the presentation was improved so that it has become more accessible for (evolutionary) biologists with a non-mathematical background.This work extends a previous paper in Decision letter after peer review:eLife. Your article has been reviewed by three peer reviewers, including Joachim Krug as the Reviewing Editor and Reviewer #1, and the evaluation has been overseen by Diethard Tautz as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Luca Ferretti (Reviewer #3).Thank you for submitting your article “Rank orders and signed interactions in evolutionary biology” for consideration by The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.Summary:eLife publication, Crona et al., 2017, where it was shown how the sign of certain epistatic interaction coefficients can be inferred from partial rank orders of fitness values. The purpose of the manuscript is to clarify the relation of this inference with the concept of sign epistasis. Crona et al., 2017 showed that sign epistasis is a sufficient condition for a rank order to be informative about at least some signed interactions, however it is not a necessary condition. Here the range of informative rank orders is extended by introducing the concept of rectangular perturbations, which generalizes the concept of sign epistasis by asking for the background dependence of the effect of mutational events that modify several loci at once. The exploratory application of the new concept to two empirical data sets suggests that it does give access to additional information about the topography of the landscape.This submission refers to the authors' previous Overall, the reviewers found that, while the manuscript contains some interesting ideas and results that are worth to be reported in the form of a Research Advance, substantial revisions are necessary. Most importantly, the results should be presented in a way that is reasonably accessible for (evolutionary) biologists with a non-mathematical background, and the presentation also needs to be more systematic. A detailed list of the issues that should be addressed is given below.Essential revisions:eLife publication of Krona et al., 2017), and the final answer that is achieved need to be spelled out more clearly and concisely. Moreover, it was not clearly stated how the new method of epistasis analysis goes beyond more established approaches, and in what sense it “add(s) precision to the analysis”.1) All three reviewers found that the manuscript was lacking in clarity and context. Specifically, they felt that both the motivating question of the manuscript  Reviewers #2 and #3 addressed the generalizability of the results. One was concerned that the scaling of the number of perturbations as 63) Another issue related to the practical application of the approach concerns the effect of measurement error. If measurement errors are not considered and there are many values close to each other, then the ranks could change significantly and the rank order could be heavily influenced by random noise, generating spuriously inferred interactions. The author states that “the results are not sensitive for a few false positives or negatives”, but this statement is not supported in the text aside from its assertion. This must be addressed (maybe through simulation?) before publication.4) Reviewers #1 and #3 were concerned that the text assumes too much previous knowledge about fitness landscapes and specifically the polytope theory of Beerenwinkel et al., 2007, on the part of the reader. One reviewer found that the introduction of the “circuits from polytope theory” is not helpful and probably incomprehensible to readers who are not already familiar with the concept, and the other wondered how to understand remark 2.3 about “the class of circuits with non-zero coefficients for exactly four elements” when the word “element” has never been used before in the text. Perhaps a formal definition of circuits can be avoided by simply saying that circuits are linear combinations of the 4 genotypic fitness values that with coefficients +/-1 arranged in such a way that the expression vanishes when fitness is additive?5) Reviewers #1 and #2 found that the discussion of the two rank orders that are compatible with additive fitness in the n=3 case in “Example 2.4” requires further clarification. What is “relabeling” in this context? And is it correct that the allowed orders are those where either the fitness is monotonic in the number of mutations, or at most one pair of genotypes violates this monotonicity, and this is a pair of genotypes with 1 and 2 mutations?6) Reviewers #2 and #3 addressed the comparison to empirical data. Reviewer #2 was not convinced that the proposed method allows one to gain “new insights” beyond previous findings, and asks for further specification of the unique contribution of the method. Reviewer #3 suggests to place the results into the context of previous analyses of the same empirical landscapes . Moreover, it was not clear whether the proposed method  singles out the wild type sequence, or whether it treats all genotypes on the same footing.7) Reviewer #1 requested further clarification of the precise relation between sign epistasis and signed interactions for different values of n (“Example 2.4”). It appears that the information contained in the manuscript regarding this point can be summarized as follows:i) For n=2 loci instances of sign epistasis and signed interaction coincide.ii) For n=3 and 4, any signed interaction corresponds either to an instance of sign epistasis or to a rectangular perturbation of higher order. If neither of these are present, the rank order is compatible with additive fitness.iii) For more than 4 loci, the characterization by rectangular perturbations is insufficient to decide whether a rank order is compatible with additive fitness.Statement (i) is obvious and well known, and the case n=3 has apparently been covered by the author through exhaustive enumeration. However, it is clear neither where the statement for n=4 comes from, nor in what sense the paper of Kraft et al., 1959, is relevant in the present context. Is there a characterization beyond rectangular perturbations that would apply for n > 4, and what kind of objects would it involve?eLife. Your revised article has been evaluated by Diethard Tautz (Senior Editor) and a Reviewing Editor.Thank you for resubmitting your work entitled "Rank orders and signed interactions in evolutionary biology" for further consideration by The manuscript has been improved but there are some remaining issues that need to be addressed before acceptance, as outlined below:Major issueThe Results section does not read well in its present form and needs to be reorganized. Presently the section begins with a partial explanation of the theory, followed by the discussion of experimental and hypothetical examples, and the complete account of the theory is found at the end. As a consequence, several concepts and definitions  are introduced twice. I suggest to subdivide the section into subsections to improve readability, and to present the theory fully in the first subsection. I would also suggest to emphasize the simple geometric meaning of the various expressions in terms of the squares and rectangles in Figures 1 and 2: The circuits in the set C are obtained by going around one of the square or rectangle and giving the genotype fitness values alternating signs along the way. Similarly the conditions for sign epistasis and higher order perturbations amount to comparing fitness differences (= directions of arrows) on two opposite sides of a square/rectangle. Regarding the hypothetical examples, there is something wrong with landscape B, since the genotypes with 3 1's are not peaks. Also, I did not find the discussion of the epistasis measure of Ferretti et al. very illuminating . It seems to me that both the hypothetical examples and the discussion of Ferretti et al. could be omitted without losing much content. Essential revisions:1) All three reviewers found that the manuscript was lacking in clarity and context. Specifically, they felt that both the motivating question of the manuscript , and the final answer that is achieved need to be spelled out more clearly and concisely. Moreover, it was not clearly stated how the new method of epistasis analysis goes beyond more established approaches, and in what sense it "add(s) precision to the analysis".In line with the reviewers suggestions, I have expanded text in the Introduction on how the results connect to open problems from Crona et al., 2017, added a separate Discussion section that summarizes conclusions, and also added a discussion about measurement errors to the Materials and method section. Most important, I have provided more evidence that rectangular perturbation goes beyond established approaches.A clarification regarding “added precision” is probably necessary. If the available information is complete fitness measurements of the preferred kind (say Wrightian fitness), then rank order methods cannot add precision. There are good reasons to consider rank orders anyway. Observations such as many peaks, prevalent sign epistasis and few trajectories to the global peak, provide intuition for the evolutionary potential.The “added precision” refers to how rectangular perturbations perform as compared to other rank order based concepts, such as summary statistics on sign epistasis, accessible mutational trajectories, peaks or fitness graphs:A) The new method has the ability to detect epistasis for landscapes, also if there is no sign epistasis. An empirical example has been added as proof of principle.B) Similarly, the new method reveals evolutionary important differences for landscapes that cannot be distinguished by frequently used methods. Explicit examples have been added.C) In addition, the revised manuscripts describes a case where one can apply rectangular perturbations for handling a problem with measurement errors in sign epistasis data.Aspergillus Niger) was deleted, since the added examples probably are more instructive. From the other original example , the more complete statistics on rectangular perturbations that includes replacements 00 → 11, seems more informative than statistics of sign epistasis alone. In general, it seems rather obvious that checking the effects of replacements 00 → 11, in addition to 0 → 1 (sign epistasis) is meaningful, especially since no effort is required.Note: One example from the original manuscript  Reviewers #2 and #3 addressed the generalizability of the results. One was concerned that the scaling of the number of perturbations as 6In many cases it make sense to consider selected perturbations, depending on context. For very large systems, one can also sample genotypes and sets of loci and check for perturbations. Rectangular perturbations can be defined for multi-allelic sequence spaces. However, a lot of theory on epistasis for biallelic n-locus systems  cannot easily be generalized to the multi-allelic case. Foundational work would be necessary for extending theoretical results to the multi-allelic case.3) Another issue related to the practical application of the approach concerns the effect of measurement error. If measurement errors are not considered and there are many values close to each other, then the ranks could change significantly and the rank order could be heavily influenced by random noise, generating spuriously inferred interactions. The author states that "the results are not sensitive for a few false positives or negatives", but this statement is not supported in the text aside from its assertion. This must be addressed (maybe through simulation?) before publication.I think the claim: “the results are not sensitive for a few false positives or negatives” caused some confusion . I have deleted the claim and added a discussion about measurement errors to the Materials and methods section. In addition, the revised manuscripts describes a case where one can apply rectangular perturbations for handling a problem with measurement errors for sign epistasis data .4) Reviewers #1 and #3 were concerned that the text assumes too much previous knowledge about fitness landscapes and specifically the polytope theory of Beerenwinkel et al., 2007, on the part of the reader. One reviewer found that the introduction of the “circuits from polytope theory” is not helpful and probably incomprehensible to readers who are not already familiar with the concept, and the other wondered how to understand remark 2.3 about “the class of circuits with non-zero coefficients for exactly four elements” when the word “element” has never been used before in the text. Perhaps a formal definition of circuits can be avoided by simply saying that circuits are linear combinations of the 4 genotypic fitness values that with coefficients +/-1 arranged in such a way that the expression vanishes when fitness is additive?The manuscript is supposed to be much easier than Beerenwinkel et al., 2007, and several other cited articles. Figure 2 is intended to explain everything a reader needs to know for understanding the main idea. I have followed the reviewers suggestion and replaced my original definition of the circuits we use with a brief description, and moved the discussion about general circuits to the Materials and methods section (some readers will appreciate the full context).5) Reviewers #1 and #2 found that the discussion of the two rank orders that are compatible with additive fitness in the n=3 case in “Example 2.4” requires further clarification. What is "relabeling" in this context? And is it correct that the allowed orders are those where either the fitness is monotonic in the number of mutations, or at most one pair of genotypes violates this monotonicity, and this is a pair of genotypes with 1 and 2 mutations?In brief, a pair of rank order differ by labels only, if there exists a cube isomorphism that induces a map between the rank orders. A detailed description has been added to the Materials and methods section.000 < ωg for each g ≠ 000, then additivity would imply that ω 111> ω g for all g ≠ 111. If we also assume ω 100> ω 010> ω 001, then the additivity assumption will impose further conditions on the rank order. A remaining question is whether ω 100 > ω 011or ω 100< ω 011. If that question has been answered, the order is completely determined.Yes, the claim is correct. For clarity, if we assume that ω6) Reviewers #2 and #3 addressed the comparison to empirical data. Reviewer #2 was not convinced that the proposed method allows one to gain “new insights” beyond previous findings, and asks for further specification of the unique contribution of the method. Reviewer #3 suggests to place the results into the context of prevous analyses of the same empirical landscapes . Moreover, it was not clear whether the proposed method  singles out the wild type sequence, or whether it treats all genotypes on the same footing.My response to comment 1 answers this question as well. In particular, an analysis of sign epistasis alone does neither have the same ability to rule out additive fitness, nor to capture global aspects, as a complete analysis of rectangular perturbation has.For clarity, I have rephrased the text in one place in the revised manuscript. The new text is: “As proof of principle, we demonstrate that the large rectangles give new insights for two empirical studies ”.I have added a remark about measures λ and λ* from Ferrettti et al., 2016. The proposed methods does not single out the wild type sequence. If the genotypes of highest and lowest fitness have maximal distance, it would be perhaps be most natural to assign the zero-string label to the genotype of lowest fitness (regardless if wild-type or not).7) Reviewer #1 requested further clarification of the precise relation between sign epistasis and signed interactions for different values of n (“Example 2.4”). It appears that the information contained in the manuscript regarding this point can be summarized as follows:i) For n=2 loci instances of sign epistasis and signed interaction coincide.ii) For n=3 and 4, any signed interaction corresponds either to an instance of sign epistasis or to a rectangular perturbation of higher order. If neither of these are present, the rank order is compatible with additive fitness.iii) For more than 4 loci, the characterization by rectangular perturbations is insufficient to decide whether a rank order is compatible with additive fitness.Statement (i) is obvious and well known, and the case n=3 has apparently been covered by the author through exhaustive enumeration. However, it is clear neither where the statement for n=4 comes from, nor in what sense the paper of Kraft et al., 1959, is relevant in the present context. Is there a characterization beyond rectangular perturbations that would apply for n > 4, and what kind of objects would it involve?If the rank order implies that υ > 0 for n = 2, we do not know whether or not the system has reciprocal sign epistasis. All we know is that the system has sign epistasis. In other words, signed circuits do not reveal complete information about sign epistasis, not even for n = 2. Similarly, a signed circuit interaction for c ε C tells us that there are one or two perturbations corresponding to c . Note that potential rectangular perturbations are exactly twice as many as the number of circuits in C.< 4. The case n = 4 is reasonably straight forward from a computational point of view . The reviewer’s last question is very interesting. Some relevant results are in Maclagan, 1999, and work that cites the paper (in particular detailed statistics for n = 5). However, the theoretical understanding for rank orders and additivity is limited. Because of the questions I have made several changes:As for additivity, checking rectangular perturbations is sufficient for n a) added explanations and clarifications for n = 3; 4.b) In addition to my reference to Kraft et al., 1969, a reference to Maclagan, 1999, has been added. Maclagan gives an explicit counterexample (which is probably easier to understand), and I have provided a “dictionary” that explains concepts and notation in the paper so that an interested reader can check.c) I have described how one can determine if a given rank order is compatible with additivity in the Materials and methods section.The manuscript has been improved but there are some remaining issues that need to be addressed before acceptance, as outlined below:Major issueThe Results section does not read well in its present form and needs to be reorganized. Presently the section begins with a partial explanation of the theory, followed by the discussion of experimental and hypothetical examples, and the complete account of the theory is found at the end. As a consequence, several concepts and definitions  are introduced twice. I suggest to subdivide the section into subsections to improve readability, and to present the theory fully in the first subsection. I would also suggest to emphasize the simple geometric meaning of the various expressions in terms of the squares and rectangles in Figures 1 and 2: The circuits in the set C are obtained by going around one of the square or rectangle and giving the genotype fitness values alternating signs along the way. Similarly the conditions for sign epistasis and higher order perturbations amount to comparing fitness differences (= directions of arrows) on two opposite sides of a square/rectangle. Regarding the hypothetical examples, there is something wrong with landscape B, since the genotypes with 3 1's are not peaks. Also, I did not find the discussion of the epistasis measure of Ferretti et al. very illuminating . It seems to me that both the hypothetical examples and the discussion of Ferretti et al. could be omitted without losing much content.I have restructured the result section accordingly. However, there are still some “repeats” of concepts, because I first discuss the two-locus case in some detail, and then (immediately after in the revised version) the general case, so as not to overwhelm the reader.I have removed the hypothetical examples and the discussion of the measures introduced in Ferretti et al., 2016, in line with the suggestions. I have also added the clarifications requested (see below).I would also suggest to emphasize the simple geometric meaning of the various expressions in terms of the squares and rectangles in Figures 1 and 2: The circuits in the set C are obtained by going around one of the square or rectangle and giving the genotype fitness values alternating signs along the way.A similar text has been added after the definition of rectangular perturbations.Similarly the conditions for sign epistasis and higher order perturbations amount to comparing fitness differences (= directions of arrows) on two opposite sides of a square/rectangle.A similar comment has been added right before the remarks in the result section.


In Bayesian machine learning, sampling methods provide the asymptotically unbiased estimation for the inference of the complex probability distributions, where Markov chain Monte Carlo (MCMC) is one of the most popular sampling methods. However, MCMC can lead to high autocorrelation of samples or poor performances in some complex distributions. In this paper, we introduce Langevin diffusions to normalization flows to construct a brand-new dynamical sampling method. We propose the modified Kullback-Leibler divergence as the loss function to train the sampler, which ensures that the samples generated from the proposed method can converge to the target distribution. Since the gradient function of the target distribution is used during the process of calculating the modified Kullback-Leibler, which makes the integral of the modified Kullback-Leibler intractable. We utilize the Monte Carlo estimator to approximate this integral. We also discuss the situation when the target distribution is unnormalized. We illustrate the properties and performances of the proposed method on varieties of complex distributions and real datasets. The experiments indicate that the proposed method not only takes the advantage of the flexibility of neural networks but also utilizes the property of rapid convergence to the target distribution of the dynamics system and demonstrate superior performances competing with dynamics based MCMC samplers. In machine learning, Bayesian inference  and Bayeo (MCMC) ,4 and vao (MCMC) ,6. MCMC o (MCMC) ,8,9. MCMo (MCMC) .Recent MCMC methods can be divided into two aspects. One class of the MCMC methods is slice sampling  and the HMC and its further developments ,21,22,23The disadvantages of the current methods motivate us to design a powerful sampler which can have not only low autocorrelation but also accurate estimation for the target distribution. In this paper, a new sampling method called Langevin normalization flows Monte Carlo (NFLMC) is proposed. We introduce Langevin diffusions to the normalization flows (NFs)  to constThe main contributions of this paper can be summarized as follows. (1) We introduce Langevin diffusions to normalization flows to construct a novel Monte Carlo sampler. (2) We propose the modified rocedure  to adjusrocedure ,29,30 inThe rest of this article is organized as follows. In q, which has the analytical probability density and then repeatedly apply an invertible transformation function The normalization flows  were firSince the Jacobian determinant of each transformation b is the scalar and n is the dimension of the parameter vectors. So To make Equation  tractablEquation , we neederminant  is defins represents the scale and t represents the translation. Both of them are neural networks. RNVP provides a more powerful and flexible posterior distribution for density estimation.Real-valued non-volume preserving (RNVP)  developsRecently, neural density estimators are widely used in the approximation of data distributions ,33 and vD-dimension Langevin diffusions are a time based stochastic process i-th term of the vector i-th row and the j-th column’s term of the matrix Langevin dynamics is a common method to model molecular dynamics systems. A  process . Equatio follows:dxt=b(xt) process . We assuIf we have x), ∀ϵ>0 . Langeviequation :(7)dxt=1The setting of Equation  makes ∂uto πg(x) .Generally, solving the stochastic differential equations exactly is intractable. Since stochastic differential equations usually have strong coupling and nonlinearity, it is difficult to calculate the exact expression of its solution. So it is necessary to utilize the numerical discretization methods to approximate the solution to stochastic differential equations. Euler-Maruyama discretization  is one oIt is noted that Langevin diffusions take advantage of the gradient information of the target distribution. The gradient information makes Langevin diffusions explore the state space efficiently. What’s more, Langevin diffusions contain the Wiener process that can be viewed as the random work. The random work helps to explore the state space extensively. The idea of Langevin diffusions are widely used in MCMC methods. Metropolis adjusted Langevin algorithm (MALA)  is one oSince NFs provide a more powerful and flexible posterior distribution for density estimation and MALA achieves rapid convergence to the target distribution, we maintain their advantages to develop a new sampler with appropriate training strategy, which can accurately sample from the target distribution with low autocorrelation.Normalization flows ,31 approConstructing the Langevin normalization flows has to satisfy two primary conditions. The first one is that the update of each step of the transformation function should be approximately invertible. The second one is that the determinant of the Jacobian and the inverse Jacobian of the transformation function must be tractable. In this way, we can ensure that the distribution obtained through the flows is able to converge to the target distribution.D is the number of the dimension of the sample. We first update a half of the sample. The transformation functions are as follows:U is the energy function of the probability density function. In addition, We then describe the details of our proposed transformation functions for a single Langevin step. We assume that  follows:y1:d=x1:d follows:y1:d=x1:d form as:zd+1:D=ydThe Jacobian matrices of these transformation functions are as follows:q represents the initial distribution. The inverse transformation functions It is noted that the Jacobian matrices of the transformation functions are upper triangular matrix and lower triangular matrix respectively, which simplify the calculation of the Jacobian determinants. In order to calculate the logarithmic probability of the transformation distribution, we need the help of inverse transformation functions and the inverse logarithmic Jacobian determinants. The logarithmic probability can be computed as follows.It is noted that Equation  is approEquation , z1:D isEquation , when itIn order to calculate the logarithmic probability of the transformation distribution, we have to compute the inverse logarithmic Jacobian determinants. The final formulas are defined as follows:Particularly, we introduce Langevin diffusions to normalization flows to construct the transformation function. Since the Langevin diffusions exploit the gradient of the target distribution, the transformation function is able to explore the state space efficiently.Hamiltonian dynamics introduce the auxiliary momentum variable to explore the state space efficiently. Through the transformation of the energy over potential energy and kinetic energy, the total energy remains unchanged. Since the change of the state is associated with the transformation of the energy, designing normalization flows which are based on Hamiltonian dynamics becomes complex, which will be our future work.q represents the approximation distribution and p represents the target distribution. Since the transformation function is invertible, the integral of There are two main differences between normalization flows and Langevin normalization flows. First, NFL cooperates with Langevin diffusions to construct an efficient and accurate approximation for the target distributions competing with the normalization flows. Second, when approximating the target distribution, the normalization flows are trained to minimize quations  and 13)KL(q|p), Probabilistic inference involving multi-modal distributions is very difficult for dynamics based MCMC samplers. Besides, samples generated from these samplers are still highly auto-correlated. In order to solve these problems, we develop a new Monte Carlo sampler using Langevin normalization flows which are called Langevin normalization flows Monte Carlo (NFLMC). Given the target distribution and the initial distribution, NFLMC learns the parameters of the conversion of the initial distribution to the target distribution of the sampler. In the following subsections, we begin to describe the main idea of the method and then we introduce how our method works. Finally, we give the loss function of the training procedure and the algorithm. When the value of loss function converges, NFLMC can precisely sample from the target distribution.N samples The procedure of NFLMC is elaborated here. Assume that the target distribution is denoted as We repeatedly utilize Equation  Ls timesEquation  is similEquation . After aEquation  and we oWe also repeatedly utilize Equation  Ls timesEquation  and finaAs Equation  shows, wEquation  and EquaEquation . Since tquations  and 13)πu and thHowever, in the progress of optimizing Equation , we findS and T, we choose to minimize As we have already discussed the transformation function in Langevin normalization flows, we do need a criterion to ensure that the final transformation distribution Equation  will enaSince we have Z is the true normalization constant and In Bayesian machine learning, we generally require sampling from the posterior distribution to approximate the complex probabilistic modal. Since Equation , we obsequation , the ill-conditioned Gaussian, the strongly correlated Gaussian, the Gaussian funnel and the rough Well. After that, we present the results on two multi-modal distributions. Finally, we demonstrate the results on a task from machine learning —Bayesian logistic regression.All our experiments are conducted on a standard computer with eight Nvidia RTX2080Ti GPUs. The nodes of each layer of the neural networks are set to be 512 with ReLU as the activation function. The number of the layer of the neural networks is set to be 3. Langevin steps are set to be 2 to 5. The number of transformation functions is set to be 8. The learning rate is set to be 0.05. The maximum iteration is set to be 10,000. We estimate scale parameter N represents the total sampling number, M is set to be 30 in our experiments and s-step autocorrelation. Autocorrelation is an index which considers the correlation between two samples. Let X be a set of samples and t be the number of iteration (t is an integer). Then t of X. The definition of the autocorrelation between time s and t is:Effective sample size—The variance of a Monte Carlo sampler is determined by its effective sample size (ESS)  which isM represents the sample number in X, N represents the sample number in Y and k represents the kernel function. Through MMD, we can analyze the convergence speed of the proposed methods.Maximum mean discrepancy—The difference between samples drawn from two distributions can be measured as maximum mean discrepancy (MMD)  which isSince RNVP performs well in density estimation, we utilize the loss function proposed in Equation  to trainRing: The ring shaped target density. The analytic form of the energy function of the ring is: Ill-conditioned Gaussian: Gaussian distribution with diagonal covariance spaced log-linearly between Strongly correlated Gaussian: We rotate a diagonal Gaussian with variances m Brooks .Rough well: A similar example from Sohl-Dickstein et al.  and its D funnel, whose energy function takes the form as: Gaussian funnel: We conduct our experiment on a 2-As In ring-shaped distribution, NNFMC has a significant fluctuation in the process of training, while NFLMC converges rapidly during the training procedure, which shows the stability of NFLMC. Since the loss of NNFMC has large fluctuation, we carefully tune the learning rate for NNFMC. As So what causes the large fluctuation of NNFMC? We think that the lack of strong guidance when exploring the state space makes NNFMC difficult to converge. Since the initial distribution is a standard normal distribution, samples from the initial distribution have large distance with the samples of ring-shaped distribution. So it is challenging for NNFMC to explore state space and the value of the loss function has large fluctuation. In order to verify this thought, we enlarge the size of the ring distribution whose energy function has the form: We then compare our method with HMC in terms of the autocorrelation on five different distributions.D mixtures of Gaussian distributions (MOG) whose probability density function are analytically available. First, we consider a MOG whose modes have the same probability and then we consider a MOG whose modes have different probabilities and further distance. The first distribution is defined as: We conduct our second experiment on two multi-modal distributions where we consider two simple 2-i et al. . The purWe compare HMC , MHMC 2, MGHMC  and NICEe et al. . We set We then discuss the circumstance in which the modes are far from each other and with different probabilities. When For parallel HMC, it seems that two modes have the same probability. However, the real probability of each mode is Logistic regression (LR)  is a traWe evaluate our methods on nine real-world datasets from UCI repository —Pima IndResults regarding the accurate rate of prediction and area under the receiver operating characteristic curve (AUC)  are summIn this study, we propose Langevin normalization flows and develop Langevin normalization flows Monte Carlo, a novel scalable sampling algorithm which exploits the flexibility of the neural networks and efficient exploration of Langevin diffusions. We design the appropriate loss function to train the sampler to ensure that the sampler is able to converge to the target distribution. We also discuss the unnormalized probability distributions and propose the appropriate loss function to these distributions. The experiments conducted on synthetic and real datasets suggest that our method is able to sample from the target distributions precisely and independently.Although HMC has various advantages, it is difficult for us to design the model based on HMC, because the auxiliary momentum variable should be carefully concerned in the transformation function of NFs. In the future, we plan to design the neural network sampler based on Hamiltonian dynamics.


In recent years, there has been significant developments in surgical robots. Image‐based sensing of surgical instruments, without the use of electric sensors, are preferred for easily washable robots.We propose a method to estimate the three‐dimensional posture of the tip of the forceps tip by using an endoscopic image. A convolutional neural network (CNN) receives the image of the tracked markers attached to the forceps as an input and outputs the posture of the forceps.The posture estimation results showed that the posture estimated from the image followed the electrical sensor. The estimated results of the external force calculated based on the posture also followed the measured values.The method which estimates the forceps posture from the image using CNN is effective. The mean absolute error of the estimated external force is smaller than the human detection limit. A minimally invasive surgery is an operation in which instruments such as endoscopes or forceps are inserted into the abdominal cavity through small ports. It offers patients the benefits of smaller scars, faster recovery, and fewer complications, as compared to conventional open surgeries. On the contrary, such operations are complicated due to the narrow field of view of the endoscope, pivot motion of the forceps centered on the insertion point, and the lack of tactile feedback.In the field of computer‐aided intervention, there are many studies that segment the forceps region from images. Attaching a marker to the forceps and extracting the forceps region is the easiest method for such estimations.Allan et al estimated the posture of a surgical instrument from a camera image, using a 3D model.Tanaka et al estimated the posture of a surgical instrument in real time.Du et al constructed a convolutional neural network (CNN) to estimate the 2D posture of a surgical instrument from images subjected to semantic segments.In this study, we propose a system to estimate the 3D posture of a surgical instrument by using a CNN, without the use of a position sensor. The system performs instrument tracking by using markers prior to inputting the image in the CNN such that any background data unrelated to the instrument posture is removed. This combination of the CNN and marker tracking enables posture estimation in unknown environments; the data set acquired in a dry environment can be used in vivo. The CNN outputs the estimated instrument posture, which is obtained from the position sensor in the learning phase. Moreover, it is possible to estimate the external force acting on the tip of the instrument by using the estimated instrument posture and a backdrivable pneumatic actuation system. The proposed method does not track the image temporally, and the error from the previous frame does not increase. Furthermore, it can estimate forceps posture from a large number of image databases at a constant rate because the computational speed of a CNN depends only on the structure of the CNN.This article is organized as follows. The robot system used in this paper is described in Section 2This section describes the system components used in this study. The proposed system consists of a master device, a slave manipulator, and an endoscope.2.1q1 (yaw), q2 (pitch), and q4 (roll), and a linear motion q3. The direction of the motion of each axis is defined to be positive when it moves along the arrow shown in Figure ϕ1 and ϕ2) and a grasping mechanism. The driving mechanism of the forceps is illustrated in Figure The slave manipulator used in this study is shown in Figure qref is the target value of the forceps posture, and X(q) is the displacement of the four pneumatic cylinders. We adopted cascade control for the posture control of the forceps, with the inner Proportional‐Integral (PI) control loop of the pneumatic pressure included in the pneumatic force control. Block and the outer Proportional‐Differential (PD) control loop of the position. To follow the rapid changes in the target value of the driving force, a feedforward compensation is applied by using the target posture and its derivative value. The driving force is obtained by multiplying the pressure in the cylinder, measured by the pressure sensor, and the cross‐sectional area of the piston.The control block diagram of the forceps is presented in Figure f^ext is an external force. Jp is a 3‐DOF Jacobian matrix, and JpT+ is a generalized inverse of JpT. Jq is a Jacobian matrix from angular velocity q˙ to cylinder velocity X˙. JpT+ and Jq are both functions of the posture q. Therefore, Equation (1) shows that the external force is estimated using the forceps posture, its differential value, and the driving force. Haraguchi et al verified the accuracy of the external force estimation.The pneumatic cylinders and wire transmission mechanisms in the forceps are backdrivable. The external force acting on the forceps tip is estimated using the following equation.2.2Sensable's PHANTOM Desktop, shown in Figure 2.3The endoscope used in this study  is a stereo camera. However, we only used one because similar images can be obtained from both cameras. In this study, we obtained 680 × 540 pixels images before applying the proposed image processing algorithm. The operator observes the 3D image on the display when teleoperating the slave robot from the master.3The proposed method combines the traditional marker tracking programing and a CNN.3.1During posture estimation, the background area, excluding the forceps, is redundant. Therefore, removing this background information may increase the robustness of the machine learning in an unknown environment.ϕ8‐mm forceps, respectively. A blue marker was selected because the red component of the organ is dominant in vivo. The tracking image of the marker was obtained using the following algorithm:Obtain an image of 680 × 540 pixels for each of the three channels of red, green, and blue from the endoscope.Flatten the brightness histogram.Binarize to extract a predetermined color range. The color range to be extracted is from  to , where the vector indicates the  values which range from 0 to 255. The color range was determined experimentally.The colors of unmarked metal parts partially fall into the predetermined color range, which results in noise on the binarized image. As the noise is less than the marked areas, we select the two largest areas as the markers and remove the remaining small areas.When the distance between the endoscope and the forceps changes, different tracking images are generated because the size of the projected marker on the image is different, even if the forceps posture remains the same. The image is trimmed to the minimum bounding rectangle of the set of markers, resulting in a tracking image that is normalized in size.The image input of the CNN needs should have the same size; however, due to the uneven size of the trimmed images, they are resized to 500 × 500 pixels.The roughness of the border of the marker differs according to the distance. Apply an average value filter of 20 × 20 pixels to the entire image.In this study, marker‐based forceps tracking is performed to remove the background information, followed by an CNN‐based posture estimation. The tracking image helps to accelerate training convergenceThe image output of each process is shown in Figure 3.2ϕ1 or ϕ2. We use two independent CNNs to estimate ϕ1 and ϕ2. The kernel and output shape of each layer are listed in Table We used CNNs to estimate the forceps posture from the tracking image of the marker. The CNN used in this study is shown in Figure The kernel size of the convolutional layer was empirically determined to improve estimation accuracy. To obtain the posture as an output, the convolutional layer is activated using the Relu function, and the fully connected layer is activated using the linear function.4This section verifies the estimation accuracy of the forceps posture in the master‐slave teleoperation. We also verify the estimation accuracy of the external force acting on the forceps tip, based on the estimated forceps posture.4.1z, ϕ1, and ϕ2 axes of the initial coordinate system are parallel to the xc, yc, and zc axes of the endoscope coordinate, respectively.Endoscopic surgery employing a surgical robot is controlled using a master‐slave method. However, creating the training data set from a real operation employing a master‐slave method results in a nonuniform distribution of the forceps posture in the data set. Therefore, we assigned the slave robot discrete target angles and created a uniformly distributed training data set. The layout of the devices when creating the training data set is shown in Figure ϕ1 and roll q4 of the holder robot to create training data equivalent to the 2‐DOF bending motion of ϕ1 and ϕ2. The motion range of the slave manipulator is defined in Table zc of the endoscope coordinate system, the marker at the forceps tip could not be detected as it is hidden behind the shaft or the flexible joint. Therefore, we excluded the postures wherein the area of the tip marker was less than 10% of that of the root marker. Finally, 107 166 data remained as the training data set.Additionally, we used one DOF of the bending motion 4.2The structure of a CNN suitable for learning from tracking images is unclear. Therefore, we select the number of the convolutional layers suitable for an experimental posture estimation.ϕ1 is achieved by the CNN with three convolutional layers, and the minimum MAE of ϕ2 is achieved by the CNN with two layers. These CNNs are used in the subsequent experiments.The created training data set consists of postures within the movement range defined in Table 4.3q3 of the holder robot or the posture ϕ2 of the forceps. However, when operating in the master‐slave mode, these two values change continuously. Therefore, to verify the accuracy of the estimated forceps posture when realistic trajectory data with continuous values are provided, validation data are created using the master‐slave method. In this study, two types of validation data were created, as shown below:Case A. Operates freely without a load.Case B. A palpation motion which an external force acts on the tip of the forceps.The training data were created without moving the linear joint In case A, the operator randomly created trajectory data, and in case B, the operator created trajectory data so as to press the forceps to the object simulating the organ, as shown in Figure 4.4To verify the accuracy of the forceps posture estimation, we compared the estimated posture with the posture measured using the sensor. The estimated posture was calculated from the validation data set using the CNN, based on the MAE of the entire training data set. For the comparison, we used sensor data instead of the ground truth  because the purpose of this study is the development of a system that can estimate the forceps posture without the use of a position sensor. We noted that previous reports suggested that the position sensors of the forceps have sufficient accuracy for practical purposes. For example, the forceps used in this study are able to perform the task of block transfer without problems.ϕ1 and ϕ2. In Figure ϕ1 and ϕ2 in case B are 6.7° and 6.5°, and the maximum error of ϕ1 and ϕ2 are 25.6° and 25.6°. The results of posture estimation and estimation error in case B are shown in Figure ϕ1 and ϕ2. Figure ϕ1 and ϕ2 in case B are 9.9° and 14.1°, and the maximum error of ϕ1 and ϕ2 are 23.9° and 14.2°.The results of posture estimation and estimation error in case A are shown in Figure 4.5To verify the accuracy of the external force estimation, we compared the estimated force with the force calculated by the position sensor and the driving force of the pneumatic cylinder with Equation (1). For the comparison, we used the calculated force instead of the ground truth  because the force estimation accuracy had been verified by previous research.x, y, and z external force estimations are 0.29, 0.14, and 0.32 N, respectively. The MAE of the norm of the external force estimation is 0.30 N, and maximum error is 0.82 N.We verified the accuracy of the external force estimation using the case B. The results of the force estimation and the estimation error for the case B are shown in Figure 5This section considers the accuracy of posture estimation and external force estimation.5.1ϕ1 and ϕ2 is 25.6°. The estimation error gives a deviation from the actual posture during position control. In particular, the deviation of 25.6°, which is the maximum error, is a gap that can be recognized by the operator through the video. Therefore, the estimation accuracy needs to be improved for use as a position sensor. In order to improve posture estimation accuracy, it is considered effective to use a high‐resolution endoscope. We investigated the relationship between image resolution and posture estimation error to examine whether it is effective to use a high‐resolution endoscope. We resized the images from case A video into 340 × 270 and 170 × 135 and acquired the tracking image with the same algorithm as the original image size. MAEs of posture estimation for each image size are shown in Table In posture estimation in case A, Figure ϕ^1 and ϕ^2 are the postures measured on the forceps coordinate system when the roll of the holder robot q4 = 0. We consider the relationship between the distribution of ϕ^1, ϕ^2, and the estimation error of ϕ^1, ϕ^2. The posture estimation error map of the training data and case A are shown in Figure ϕ^1 and ϕ^2. Figure ϕ^1 or ϕ^1 is large. Therefore, uniform training is also important for improving posture estimation accuracy.We plotted an error map to analyze the relationship between posture distribution and posture estimation errors. To plot the error map, we defined a coordinate system derived from the equation below, as follows:Figure 5.2During the external force estimation of case B, as shown in Figure 6In this study, we propose a system that estimates the 3D posture of the forceps of a surgical robot by using endoscopic images, without the use of position sensors. This system employs CNNs that receive tracking images of the markers attached to the forceps as an input and output the posture of the forceps. The accuracy of posture estimation was verified using the trajectory generated by the master‐slave operation. The experimental results show that the posture of the forceps estimated from the endoscopic image was accurate. Moreover, the external force acting on the tip of the forceps was estimated by combining the posture of the forceps estimated from the image and the driving force of the pneumatic cylinders. The validation results show that the MAE of the external force estimated using image processing is less than the required accuracy for force sensing in a master‐slave surgical robot. The experimental results indicate the effectivity of posture and external force estimation via CNNs.


Fabrication of a suitable flow model or phantom is critical to the study of biomedical fluid dynamics using optical flow visualization and measurement methods. The main difficulties arise from the optical properties of the model material, accuracy of the geometry and ease of fabrication.Conventionally an investment casting method has been used, but recently advancements in additive manufacturing techniques such as 3D printing have allowed the flow model to be printed directly with minimal post-processing steps. This study presents results of an investigation into the feasibility of fabrication of such models suitable for particle image velocimetry (PIV) using a common 3D printing Stereolithography process and photopolymer resin.4SCN). The images were of a quality that after applying common PIV pre-processing techniques and a PIV cross-correlation algorithm, the results produced were consistent within the aneurysm when compared to previous studies.An idealised geometry of a cerebral aneurysm was printed to demonstrate its applicability for PIV experimentation. The material was shown to have a refractive index of 1.51, which can be refractive matched with a mixture of de-ionised water with ammonium thiocyanate (NHThis study presents an alternative low-cost option for 3D printing of a flow phantom suitable for flow visualization simulations. The use of 3D printed flow phantoms reduces the complexity, time and effort required compared to conventional investment casting methods by removing the necessity of a multi-part process required with investment casting techniques. An aneurysm is a balloon-like bulge that forms on a blood vessel as a result of weakened vessel wall layers. The most dangerous aneurysms are found in the Circle of Willis and on the abdominal aorta. For cerebral aneurysms, rupture risk is 0.6–1.3% per annum.in vivo data would be the most physiologically representative means of validating computational models, however, observing phenomena like long-term thrombosis proves challenging. In vitro tools which can be used to generate data for validation are desirable as there can be greater control over the experimental process.in-vitro studies of haemodynamics.in silico and in vitro setups, the reconstruction of patient-specific anatomical models is beneficial. These are often derived from computed tomography (CT) or magnetic resonance images (MRI).Fluid dynamics is important in the investigation of the flow characteristics within an aneurysm and in cardiovascular diseases in general.in vitro fluid dynamic measurement method that is commonly used for external and internal flow studies and is particularly useful for bio-fluid flow problems, such as flow in aneurysms.Particle image velocimetry (PIV) is an in vitro flow phantoms out of transparent resin. There are two main types of 3D printing techniques that are commonly available commercially and can print optically clear resins. They are PolyJet and Stereolithography (SLA). Both uses a liquid resin but PolyJet prints in air in a raster pattern of straight lines whilst SLA prints in a liquid bath. This difference may have an impact on the clarity of the eventual printed model. Further discussion on this is given later in this paper. Different types of method-resin(s) combinations (from various manufacturers) are available on the market that are suitable for use in direct printing of optically clear flow phantoms, such as:PolyJet with Veroclear resin from StratasysXX) resins from FormlabsStereolithography (SLA) with FLGPCL series , printed with the SLA technique on a desktop Form 2 printer. In addition to being less expensive, SLA could potentially create smoother internal surfaces because of the lack of requirements for support material.One of the challenges of using 3D printed flow phantoms is in the matching of the refractive index (RI), due to the high refractive indices of transparent polymers compared with cast silicone. This limits the possible working fluids suitable for use. Recently Aycock XX. An idealised geometry of a cerebral aneurysm was adapted from the model developed by Mulder et al.The flow phantom was printed on a Formlabs Form 2 desktop SLA printer using clear photopolymer resin FLGPCLµm. The completion of the printing process is followed by cleaning in isopropyl alcohol (IPA) to remove uncured resin from the print surfaces. The model is then placed in sunlight for at least an hour in a post-curing process as recommended by the printer and resin manufacturer. No specific control of temperature was employed. Finally the model is finished by sanding and spraying of the outer planar surfaces. Sanding was performed using sandpaper of reducing grit size, with a final grit of P2000. The final process was the manual spraying of the external surfaces with a clear fast drying aerosol lacquer based spray paint (Spraymate Clear), which provides a drastic improvement to the final clarity. A picture of the models (pre-polishing and spraying vs. post-polishing and spraying) is shown in Fig. The printing process starts with a 3D Design of the model using computer-aided-design (CAD) software. Printing was performed at the finest resolution achievable which was 25 To further demonstrate the clarity of the material, a patient-specific model with highly complex geometrical features was printed and is shown in Fig. The refractive index (RI) of the printed material was measured with a Bellingham and Stanley Abbé refractometer and a white LED light source. Measurements were made on flat facets of approximately 12 mm × 35 mm cuboid samples printed especially for the RI measurement. The samples were coupled optically to the refractometer prism with a thin film of bromonaphthalene. The refractometer was calibrated with deionized water and a glass slide of known RI of 1.51. The uncertainty in RI ranged from ± 0.001 to ± 0.01 with the larger values resulting from uneven sample surfaces blurring the extinction angle.XX resin was 1.507 ± 0.003. Aycock et al. did not report the RI of Veroclear hence a measurement was performed using the same technique. The measured RI was 1.51 ± 0.01. Cast silicones can have lower RI.40The measured RI of the printed FLGPCL4SCN) and 34% de-ionised water at room temperature was eventually used as the working fluid that could match the RI of the model. NH4SCN is endothermic when mixed with water, and it should be allowed sufficient time to heat up before use. Some NH4SCN may be replaced with glycerol, to reduce the cost if required.At such high RI, the common working fluid used in cast silicones models (water-glycerol mixture) could not be used. Jermy compiled a list of suitable working fluids with zinc iodide, sodium iodide and ammonium thiocyanate shortlisted as potential candidates with the highest RI amongst those listed.Visual inspection to detect any distortion due to mismatch of RI was performed with a checkered grid placed behind the idealised flow phantom Fig. . Althoug−1 was applied.The viscosity of the working fluid used for this investigation is given in Fig. The experiment was repeated and two sets of measurements were taken and the line is the best-fit line from both sets of data combined.The viscosity of the working fluid is not exceptionally high thus allowing for the selection of easily available pumps and fittings for the flow set-up. In addition, the relative density of the fluid 1.131 at 23.7 °C which matches quite closely with the density of commonly available seeding particles thus reducing the effects of buoyancy of the particles during low flow applications.Although the working fluid’s viscosity and density are distinct from blood, this can be overcome by appropriate matching of Reynolds and Wommersley numbers to achieve dynamic similarity.Roughness of the surface in contact with the fluid is a very important parameter when conducting fluid dynamic experiments. Rougher surfaces induce laminar to turbulent transition early and affect both the bulk flow dynamics as well as the local wall shear stress (WSS) which is an important consideration in hemodynamic studies. Most previous PIV experiments on vascular flows using the conventional investment casting method only qualitatively describes the smoothness of the internal flow channel as being “smooth to touch”.To measure the surface roughness of the internal flow channel, a cuboid solid print of the same material (but an opaque version) was used and the surfaces were scanned using the SmartZoom 5 Digital Microscope from Carl Zeiss. The sample was not polished or sprayed so that it will be similar to the internal surfaces of the flow phantoms where polishing and spraying is challenging. The opaque version of the material was used to enhance the optical measurements of the microscope. Both horizontal and vertical surfaces were scanned to check if there were differences in roughness on surfaces parallel and perpendicular to the print direction. Scanning on both surfaces was performed at different locations to get a more representative gauge of the roughness. The on-board software of the SmartZoom 5 Digital Microscope was able to provide visuals but not calculate roughness values directly. Hence a visual estimate of the maximum height of the profile Rt was performed by adjusting parallel measurement lines on the software GUI. Horizontal lines were placed (within one-pixel accuracy) at the maximum valley depth (Rv) and maximum peak height (Rp) and Rt calculated as the differenceThe arithmetic average roughness Ra for the horizontal and vertical surfaces was calculated by the values extracted from the software for the entire surface and are 2764 and 5093 nm respectively. The Ra value for the vertical surfaces are higher than that for the horizontal surfaces which is expected as they cut across the printing layers. The Rt values corresponds to a Roughness N ISO Grade Number between N6 and N8. The Ra values corresponds to between N8 and N9.To assess the accuracy of the printer at printing channels in the chosen material, a separate sample was produced with descending channel size from 5 to 0.1 mm Fig. . An imagIn order to demonstrate the applicability of the flow phantom for PIV, a planar PIV experiment was set-up with a representative steady flow rate through the idealised flow phantom.The flow set up consists of a reservoir, a header tank with an overflow and a mini turbine flow meter. A submersible pump was used to pump the working fluid from the reservoir to the header tank located at an elevated height where the working fluid is being fed by gravity through a mini turbine flow meter and then through the flow phantom and back to the reservoir again. The header tank had an overflow, which maintained the flow rate at a constant value.μm borosilicate glass particles and PIV images were captured using a PCO pixelfly USB 14 bit CCD PIV camera with the flow phantom being illuminated by a Firefly PIV and Imaging Laser from Oxford lasers.The working fluid was seeded with 10 in-vivo conditions by matching the Reynolds number at 106. This simulated the upper limit of the flow rate expected through a cerebral aneurysm and corresponds to an average shear rate more than 100 s−1 within the blood in in-vivo conditions. At this high shear rate, the viscosity of blood is constant.Flow rate in the experiment was appropriately set to achieve dynamic similarity with As the focus of this paper is on demonstrating the applicability of the 3D printed flow phantom for use in PIV experiments, cross-correlations were made only in the idealised model although a physiologically accurate model was fabricated and PIV images taken.Images used for PIV analysis are presented in Fig. Processing of the results for the idealised model was performed using the open-source MATLAB code PIVlab 1.4.1et al.17The images are sufficiently clear with the particles easily distinguishable, especially after background subtraction processing. The cross-correlation code PIVlab is able to produce appropriate results clearly showing the flow recirculation about an axis that is off-centre within the aneurysm. This is consistent with the previous study by Hoi However, it is noted that there is an obvious observable difference in brightness between the “leading” and “trailing” portion of the idealised model from the point of view of the laser direction due to attenuation of the light sheet. This may cause difficulties when using a large model, but is appropriate for the current set-up. Note that the brightness difference in the physiologically realistic model is not due to the attenuation but due to the certain parts of the flow channel not being in the same plane.XX clear photopolymer resin can produce a flow phantom that compares well with the option of using a Stratasys PolyJet printer with Veroclear. Both the printer and the raw resin are significantly lower in cost when compared to the Veroclear option. Aycock et al.XX model had a Ra of 2.764 and 5.093 μm when measured aligned with and across the layers using an optical microscope. The Rt values corresponds to a roughness N ISO Grade Number between N6 and N8 and are acceptable for the current application. In comparison, it is still higher than cast silicone (1.27 μm) but better than two other 3D printed materials (8.87 and 16.07 μm) as reported by Cloonan et al.4SCN powder dissolved in 34% de-ionised water (by weight) produces a working fluid of comparable RI as the printed model and no observable distortions in grid lines were present when a checkered grid was placed behind the flow phantom. The resulting solution is compatible with glycerol, which could be used to further “control and refine” the RI of the working fluid. In addition, the current material does not exhibit the blurring of the seeding particles due to different print orientations as reported by Aycock et al.A Formlabs Form 2 Desktop SLA printer printing FLGPCLsilicone .27 μm buIn that study, limitations in-terms of clarity of the particles was found when using 3D flow models printed using the Polyjet technique with different print orientations even though the clarity and transparency is acceptable for PIV and the RI between the working fluid and printed material matched. Little explanation was provided as to why this happens. Our work made use of 3D printed flow models using the Stereolithographic (SLA) technique. Several prints were made and observations shows no loss of PIV image quality in different print orientations. A possible explanation to this phenomenon could be due to differences in how the two techniques work.Polyjet 3D printing uses a liquid resin that cures with exposure to air and the print head travels in a raster pattern of straight lines. It (as with FDM) prints in air whereas SLA, prints by curing the material in a liquid bath of the resin and successive layers are polymerized beneath each surface, resulting in a homogenous final material. Printing using the Polyjet method could lead to either alternating strips of cured resin material sandwiching rows of air (which does not happen in the SLA process as the entire operation is being performed submerged in the liquid) or the resin in each new printed line fusing with the side of a previous line (which has been exposed to air and has aged slightly) which may give a slight modulation in the RI of the cured polymer.An idealised illustration of the material printed using both processes highlighting the difference in the microstructure is given in Fig. et al.When conducting PIV experiments, light from the laser, or light scattered from the particles, strikes the flow phantom and depending on the position of the laser and camera with respect to the “printed material lines”, may result in modulation of the light intensity giving the smudged streaks and “spreading of the laser light sheet intensity” described by Aycock Although there are many advantages to this method of fabricating the flow phantom, the high RI of the resin, (measured as 1.507 ± 0.003) however, makes common fluids used for similar PIV experiments on cast silicone such as glycerol-water mixture unsuitable and salt solutions have to be employed. In comparison, the measured RI for Veroclear was 1.51 ± 0.01 and VisionClear from Luxexcel  is reported to have an RI of 1.53. The Luxexcel printer does not seem to be table-top size. Cast silicone has a lower RI of between 1.41 and 1.47.19In addition, the current method is not feasible for printing channels with resolution below 1 mm but has an error of not more than 0.2 mm difference from the original CAD model. This could pose problems when printing geometries with very intricate “spacings” but would be appropriate for other models. Users are recommended to take these inaccuracies into account when choosing to use this method to fabricate flow phantoms. Finally, the attenuation of light through the model may be a concern when very large models are required.et al.XX from Formlabs is suitable for this and similar applications.In conclusion, this study builds on the work of Aycock