Robotic-arm-assisted total hip arthroplasty (RoTHA) offers the opportunity to improve the implant positioning and restoration of native hip mechanics. The concept of individualised, functional implant positioning and how it relates to spinopelvic imbalance is an important yet rather novel consideration in THA. There is mounting evidence that a significant percentage of dislocations occur within the perceived “safe zones”; hence, in the challenging subset of patients with a stiff spinopelvic construct, it is imperative to employ individualised component positioning based on the patients’ phenotype. Restoring the native centre of rotation, preserving offset, achieving the desired combined anteversion and avoiding leg length inequality are all very important surgeon-controlled variables that have been shown to be associated with postoperative outcomes. The latest version of the software has a feature of virtual range of motion (VROM), which preoperatively identifies potential dynamic causes of impingement that can cause instability. This review presents the workflow of RoTHA, especially focusing on pragmatic solutions to tackle the challenge of spinopelvic imbalance. Furthermore, it presents an overview of the existing evidence concerning RoTHA and touches upon future direction. Total hip arthroplasty (THA) has been a well-established and successful procedure for over 60 years, with an ever-growing number of primary operations performed on an annual basis. However, changes in the demographics of patients requiring a total hip replacement, as well as limitations in the conventional method, present new challenges that necessitate a modern solution. With great advances in surgical technology, robotic-arm-assisted THA (RoTHA) presents a pragmatic solution to overcome barriers caused by human error through improving implant positioning and restoring normal hip biomechanics.Robots were first used in surgery in the 1980s, and subsequently implemented into orthopaedic surgery in the early 1990s for THA. Their evolution has yielded a spectrum of functional modes, ranging from fully automated to a semi-active instrument at the beckon of the surgeon’s hand . In all However, there was an 18% conversion rate to conventional THA (CoTHA), as well as higher rates of dislocation, revision surgery and soft tissue complications compared to manual THA . These iTraditionally, to plan the positioning of the acetabular component in CoTHA, a pre-operative supine anteroposterior (AP) radiograph of the pelvis is taken. However, this does not consider the dynamic relationship between the pelvis and the spine, and it does not address pre-existing spinopelvic pathologies. The importance of addressing these challenges is highlighted by the association between spinal deformity and increased dislocation . RoTHA cThe complex relationship between spinopelvic mobility and component positioning in THA presents a challenge for the arthroplasty surgeon. This review presents the workflow of RoTHA, incorporating spinopelvic measurements into pre-operative planning and aiming to achieve functional implant positioning. The workflow presented in this manuscript is pertinent to the Mako system; however, it could be translatable to other robotic systems. Furthermore, our review outlines the existing evidence concerning RoTHA and touches upon the future direction in the field.Since the pelvis is a dynamic unit within the body, spinopelvic motion should be considered when planning the functional implant positioning. The concept of a functional implant position is the combination of the anatomic cup position implanted in the bone and the postural orientation of the pelvis. An accurate functional cup position determined by patient-specific parameters improves hip stability, which is executed with greater precision by RoTHA . PreviouUnder normal circumstances of moving into the sitting position from standing, there is a posterior rollback of the pelvis upon the lumbar spine; this is accompanied by a loss of the lumbar curvature. This is to allow for the requisite flexion of the femur which is expected to be 55–70 degrees ,13,14. TThe lumbar spine X-rays are evaluated to determine the sacral slope, defined as the angle between the horizontal and the superior end plate of the S1 vertebrae. The difference in the sacral slope between sitting and standing is used as a measure of movement of the pelvis on the lumbar spine as the hip is flexed from standing to sitting position a,b.According to Stefl et al.’s  classifiThe stiff spinopelvic group is further divided into two subgroup patterns dependant on the position in which the spinopelvic joint is fixed: “stuck-standing” if the pelvis is stuck in the anterior position of standing defined by an absolute pelvic tilt of <30 degrees in the sitting position; and “stuck sitting” if pelvis is stuck in the posterior position with an absolute pelvic tilt of <30 degrees when standing because there is no anterior roll of the pelvis normally expected in standing from sitting.Vigdorchik et al.  suggeste“Measure twice cut once” is an old proverb , which iThe preoperative phase begins with obtaining a CT scan to define the bony landmarks that will later be captured intraoperatively to confirm the anatomy and generate a three-dimensional reconstruction of the femur and pelvis. Additional information regarding the bone morphology in terms of the version of the femur and acetabulum as well as any bony deficiencies can also be collected from the CT. Once the CT data are imported into the robotic software, the robotic product specialist works collectively with the arthroplasty surgeon to create a functional plan. In conjunction with the CT, standing and sitting lateral lumbar spine radiographs are organised to assess spinopelvic motion.The operative plan is normally finalised in the weeks leading up to the operation; however, adjustments can be made at any time, including during the operation. Last minute checks are usually made by the surgeon in theatres prior to scrubbing for the operation.The main surgical targets with regards to the acetabular cup positioning that need to be considered are reproducing the native centre of rotation as well as the cup version and size to ensure satisfactory bony coverage a–c. AnatAttention is then turned to the femur as the expected size of the stem, and the planned osteotomy site on the femoral neck can be determined. The robotic software allows careful planning of the femoral head centre of rotation. The estimated effect on changes to the offset and leg length can also be evaluated by comparing to the contralateral and preoperative hip within the robotic software .The cup and femoral stem positions can be adjusted in conjunction with the spinopelvic motion to ensure no bony impingement is evident, and the combined anteversion is between 25 and 45 degrees to prevent instability ,23.The results of the sacral slope calculations from the lumbar spine radiographs are imported into the robotic software, and a virtual range of motion of the joint is performed to evaluate impingement. The virtual range of movement (VROM) tool gives the ability to test the planned hip replacement into positions of maximal range of movement and receive instant feedback in relation to the presence of impingement.To perform the VROM, the hip is tested in two positions similar to sitting and standing: deep flexion and full extension. In our practice, we test deep flexion at 110 degrees flexion and 40 degrees internal rotation; extension is evaluated at 25 degrees extension and 15 degrees external rotation. In each of these positions, careful evaluation is performed to ascertain the presence of bone-on-bone, bone-on-implant, or implant-on-implant impingement.Changes in component positioning are subsequently dictated by the VROM, and examples include lateralising the cup or increasing the femoral offset if there is bone-on-bone impingement; increasing the femoral stem version if there is posterior impingement; or planning to remove osteophytes if they are deemed to be the source of impingement.Following the surgical approach, a screw for the removable femoral array is inserted into the greater trochanter away from the planned femoral osteotomy site. A check point is also placed next to the femoral array screw to confirm the system accuracy immediately before using the robotic arm to ensure that nothing has come loose during the operation.Bony landmarks are captured on the femur before dislocation on both trochanters, femoral neck, and femoral head. After registration, the femoral osteotomy can be planned using the probe on the femoral neck and marked with diathermy. After the femur is prepared, the femoral broach position can be checked by using ‘Broach Tracking’, which has three pre-registered divots on the trial neck. This allows the surgeon to compare the version, depth, and lateralisation of the broach to the preoperative plan.Attention is then turned to the acetabulum as landmarks are registered along the edge of the acetabulum rim and inside. A second checkpoint is also placed superior to the acetabulum with a tied suture for retrieval at the end of the operation.The robotic arm is used to prepare the acetabulum with attached hemispherical reamers. It is locked into position using stereotactic boundaries and is only able to ream in the plane of the planned cup version and inclination. The robotic software provides real-time feedback of the depth of the reaming. The definitive cup is then positioned within the acetabulum in the appropriate version and inclination, and the robotic arm is again locked within the stereotactic boundaries and impacted with a manual mallet. The robotic software displays the depth remaining until the cup is seated. After implantation, the cup position can be checked against the preoperative plan using three pre-registered divots on the definitive cup.Real-time feedback on the leg length and combined offset changes during the operation are obtained by performing a trial reduction . In thisThe patient had previously undergone a left total hip replacement complicated by two post-operative dislocations. She did not report a history of back pain.The difference in sacral slope between standing and sitting radiographs was noted to be 6 degrees. According to the Stefl classification, this is stuck sitting, as the sacral tilt does not tilt anteriorly beyond 30 degrees with standing, indicating a high-risk patient a–c.In this case, the native femoral retroversion (−6 degrees) posed a challenge in avoiding impingement . Upon asVROM was performed again, and impingement in flexion was eliminated . In exteThe robotic software also enables preoperative and intraoperative visualisation of the anticipated postoperative X-rays accounting for any changes to the plan. In addition, the software allows for calculation of changes to the leg length offset compared to the preoperative and contralateral hips. In this case, the leg length was 1 mm longer compared to the opposite hip, and the combined offset was 6 mm increased compared to that preoperatively a,b.The figures below depict pre-operative planning in a 68-year-old female patient with stiff spinopelvic construct and the changes that had to be made to overcome the challenge of spinopelvic imbalance.Historical targets mandated positioning of the acetabular component into 15 ± 10 degrees of anteversion and 40 ± 10 degrees of inclination as described by Lewinnek , which hA cohort study of 75 patients by Kayani et al. demonstrated statistically significant improvement in the accuracy of prosthesis placement within the defined safe zones with RoTHA. They found overall component accuracies to be 96% 24/25) and 92% (23/25) within Lewinnek’s and Callanan’s safe zones, respectively, whereas through conventional means, they were 68% (34/50) and 64% (32/50), respectively  and 92% , compareIt has been shown that RoTHA can result in more accurate implant positioning and improved restoration of natural hip biomechanics compared to CoTHA  and receNotwithstanding this, there are systematic reviews showing discordant findings. A systematic review and meta-analysis of 14 studies by Han et al. showed no differences in functional outcomes including HHS, Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) or the Merle D’Aubigne Hip Score . SimilarReported rates of intra-operative and post operative complications for both CoTHA and RoTHA remain low and comparable within the current body of literature in studies encompassing semi-active robots ,30,31,32Recent studies have shown reduced rates of dislocation of RoTHA compared with conventional methods. A large retrospective comparative review by Bendich et al. showed that there is a significant reduction in the rate of dislocations within the first year of index surgery requiring revision surgery . SimilarRoTHA is associated with substantial cost pertaining to the installation of the robotic device and software, maintenance and sterilisation, additional imaging, such as CT scans, and training of the staff. Notwithstanding this, the costs could potentially be offset if the documented improvement in implant positioning is shown to increase implant longevity and reduce length of stay. Maldonado et al. implemented a stochastic Markov model to compare the cost effectiveness of RoTHA and CoTHA. The authors found that the cumulative cost difference at 5 years for Medicare patients was GBP 945 less in favour of RoTHA and similarly for private healthcare patients, it was GBP 1810 at 5 years . The modThe evolution of surgical technology has resulted in the development of semiactive constrained robotic systems, and there is immense potential for this technology . RoTHA sTherefore, results from prospective randomised controlled trials with longer-term follow up are needed to accurately evaluate and unmask any potential functional benefits of RoTHA. Furthermore, the role of registry data should also not be overlooked.The cost effectiveness of RoTHA is an important topic, especially in relation to adopting this technology in publicly funded healthcare systems. There are significant costs associated with the equipment, staff, and training; however, stochastic models have shown this can be offset by a reduction in the length of stay and complications. As longer-term outcomes and complication rates emerge, the cost effectiveness will become clearer with further research into this area.Finally, the enhanced planning and accuracy with robotic technology has brought to the fore the concept of personalised component positioning based on the patients’ phenotype. Overcoming the challenge of spinopelvic imbalance and impingement risk necessitates careful planning and high accuracy in executing the pre-operative plan. In this vein, robotic technology can offer a pragmatic way to tackle the challenges posed by abnormal motion in the spinopelvic kinetic chain. The integration of the virtual ROM tool has also enabled instant intra-operative feedback, based on which the arthroplasty surgeon can change the plan to avoid impingement and achieve the surgical targets.


The behavior intention estimation and interaction between Autonomous Vehicles (AV) and human traffic participants are the key problems in Automatic Driving System (ADS). When the classical decision theory studies implicitly assume that the behavior of human traffic participants is completely rational. However, according to the booming quantum decision theory in recent years and actual traffic cases, traffic behaviors and other human behaviors are often irrational and violate the assumptions of classical cognitive and decision theory. This paper explores the decision-making problem in the two-car game scene based on quantum decision theory and compares it with the current mainstream method of studying irrational behavior-Cumulative Prospect Theory (CPT) model. The comparative analysis proved that the Quantum Game Theory (QGT) model can explain the separation effect which the classical probability model can’t reveal, and it has more advantages than CPT model in dealing with game scene decision-making. When two cars interact with each other, the QGT model can consider the interests of both sides from the perspective of the other car. Compared with the classical probability model and CPT model, the QGT is more realistic in the behavior decision-making of ADS. In order to drive safely and efficiently in this complex traffic surrounding, autonomous driving vehicles need to correctly estimate the behavioral intention of human traffic participants and interact with human traffic participants naturally just like human driving vehicles3. The behavior of human traffic participants and their interactions are very random in the real world actually. Osamu proposed that such randomness is characterized by obvious uncertainty and irrationality4. The "long tail" problem of autonomous driving includes various fragmented scenarios, extreme situations and unpredictable human behavior. This is related to the unreasonable behavior intention and uncertainty5, which needs to be studied by correct and effective cognitive and decision theory.For a long time in the future, autonomous vehicles will inevitably share urban roads with human traffic participants7 and the deep learning methods based on data drive are common. Traditional machine learning methods generally assume that the evolution process of traffic participants has the characteristics of Markov Decision Processes (MDP), Hidden Markov Model (HMM), Dynamic Bayesian network (DBN) and other methods to infer intentions. And the most extensive Partially Observable Markov Decision Processes (POMDP) is used to obtain correct behavior decision. However, the existing results of human behavior decision-making theory show that human behavior is incompatible with the complete rational hypothesis in classical decision-making theory8. And the cognitive and decision-making theory based on classical probability cannot accurately describe human behavior and its interaction9, which are the main bottlenecks restricting the safe and efficient driving of autonomous driving in actual urban traffic scenes10. Data-driven deep learning methods must rely on massive big data sample training  to deal with the “long tail” problem that may cause driving accidents12 and the explosive progress of autonomous driving technology being hindered. Daily, low probability of occurrence or emergent, dangerous, and the edge (corner) scene of scarce samples are often related to irrational behavior and interaction14. And pure deep learning method based on data driven is difficult to effectively cope with the “long tail” problem15.The mainstream methods of behavior decision-making, the traditional machine learning methods based on classical probabilistic reasoning16. In particular, the initial quantum inkling in the field of mobile robots, being most closely related to unmanned driving technology16, allows us to see the potential and possibility of applying quantum theory to solve the cognitive problems of autonomous driving. Quantum theory provides a new way to study the uncertain behavior  of human traffic participants and their interaction. How to correctly understand the uncertain behavior and interaction of human traffic participants based on quantum theory, and how to make correct interactive behavior decisions based on this is the focus of this paper waiting to explore and solve.What is gratifying is that quantum theory originated in the field of microscopic physics has been extended in the past two decades and it made great progress in many non-physical and macro fields such as cognition, decision making, information, communication, computing, etc. It has not only formed an increasingly mature theoretical system, but also been increasingly widely appliedAt present, there is no relevant literature to explain autonomous driving with quantum cognition and decision theory. Based on the Quantum Game Theory (QGT) model, this paper analyzes the behavior decision-making problem of human traffic participants (two-car game case), and compares it with the current mainstream method of irrational behavior research-Cumulative Prospect Theory (CPT), verifying the accuracy of the two models with data set experiments.17, including prediction intention and behavior selection, which regards the intention of road users as a potential representation in decision-making state space. Mehta18 proposed a navigation method based on POMDP, the evaluation decision was made by a set of pre-designed forward simulation predictions of closed-loop behaviors, and the social interaction between autonomous vehicles and other participants was also considered. Hubmann19 proposed a unified decision-making planning framework of online POMDP for complex signalless intersections, which considered the uncertainty of traffic participants' behavior intention, motion prediction and interaction with autonomous vehicles, but like most literatures, it still assumed that traffic participants' behavior decisions were rational. There are also some behavioral decision-making frameworks based on game theory that consider interaction, but the behavioral intention cannot be considered, or the participants are assumed to be rational. Fabiani20 proposed a multi-vehicle distributed hybrid coordination decision-making framework, which formalized the problem of non-cooperative behavior coordination as a generalized mixed integer game model. Coskun21 combines game theory with MDP, and describes the decision-making process of interaction between autonomous vehicles and other vehicles with MDP game model. Isele22 formalizes the interactive decision-making process as a random game (extended MDP) model, and realizes interactive decision-making by designing effective search and reasoning strategies of intent game tree. Tian23 established an interactive decision-making framework based on K-level game theory, while Li24 proposed a game theory framework integrating cognitive hierarchy theory and Bayesian reasoning, which can better describe the bounded rational interaction of participants. In addition, there is a negotiation framework based on heuristic strategy, which is different from the game decision-making framework, and can deal with interactive behavior decision-making problems3, but does not consider the irrational behavior of traffic participants.The most widely used traditional machine learning is Partially Observable Markov decision processes (POMDP)25. The two most representative paradigms of deep learning in motion planning are planning based on Deep Imitation Learning (DIL) and Deep Reinforcement Learning (DRL)26. DIL can be trained with data collected from the real world, but these data are scarce in corner cases, which makes the response of the trained network uncertain when encountering invisible data; DRL system can explore different driving situations in the simulated world, but when transplanted to the real world, these models often have biased behaviors12. In recent years, DRL and DIL have provided effective solutions to solve MDP problems in high-dimensional state space and decision space, and promoted the research on autonomous driving behavior decision-making to make important progress27. However, the established DRL/DIL model is difficult to verify and ensure safety in the actual complex environment. In the actual deployment of this model, there may be great challenges in terms of stability and robustness26, at present, it is basically in the stage of simulation research, and most of them are aimed at simple scenes28. Transferring the driving strategies learned from simulation to the real world is still an open research challenge29.In recent years, the method of deep learning motion planning based on data-driven has attracted more and more attention30, and their uncertain behavior decisions are mostly based on the classical probability decision-making theory and game theory, but the complete rationality assumed by the classical game theory cannot accurately describe human decision-making behavior31.At present, people have made quite a lot of research achievements in the behavioral decision-making planning of autonomous driving and safe navigation, but most of the researches are based on rational behavior assumptions, especially the interaction between autonomous vehicles and traffic participants32. Quantum logic was put forward by the famous mathematician Von Neumann, who defined the event as a subspace in Hilbert space32, so that quantum probability does not need to be constrained by many Boolean logic rules such as the rule of total probability. Therefore, quantum decision theory can allow those events that violate the law of total probability to exist. Busemeyer and Bruza pointed out that quantum logic is actually a kind of generalized Boolean logic, which does not have many constraints in Boolean logic, has greater flexibility and randomness, and is more conducive to explaining people's judgments and decisions16.Quantum mechanics is the greatest discovery in the last century, which greatly promotes the development of modern science and technology and becomes the theoretical pillar of emerging science and technology. Scholars in the field of cognition have found that the interaction between interference and entanglement in quantum mechanics and human cognition has many similar characteristics, which urges us to construct a mathematical expression method of quantum mechanics, introduce quantum probability into the cognitive field, try to use the unique characteristics of quantum mechanics to build a cognitive model, and explain the problems in the field of human cognition that can not be explained by the cognitive decision theory based on classical probability,quantum cognitive decision theory based on quantum probability is gradually born36. The quantum cognitive decision theory  produced by combining quantum probability with classical machine learning theories  provides a more advanced, effective and feasible theoretical tool for the cognitive decision research of autonomous driving system. Song and others for the first time quantum cognitive decision making theory is introduced into Autonomous driving field, aimed at the pedestrian crossing in Autonomous driving scene to build the kind of QLBN model, and compared with classical bayesian model made, in the case of the pedestrian crossing well explain the existence of irrational behavior, at the same time to build the kind of Quantum social force model and compared with the mainstream data-driven model, it also has a good advantage in pedestrian crossing trajectory prediction37, Catarina used QLBN to analyze "prisoner's dilemma" cases and obtain predictive results for similar events. Comparing with reality, Catarina proved the predictability of the quantum probability method38.In recent 10 years, quantum cognitive decision theory has made a series of breakthroughs in the field of human cognition, which has been recognized as a new way to explore human cognitive scienceTo sum up, there is no systematic method for automatic driving decision-making considering irrational behaviors of human traffic participants and their interactions. Although quantum decision-making theory has made great progress in recent years, it provides a new method to study the automatic driving decision-making problem considering the interaction of human traffic participants' behaviors , but there are no research cases applied in the field of automatic driving at present. In this paper, the two-car game case is analyzed by QGT, which is the first attempt to apply quantum decision theory in the field of ADS.Let's assume that a system has attribute A, and its value can be up and down. In addition, the system also has attribute B, and its value can be left and right. The biggest difference between quantum probability and classical probability is that there are incompatible attribute pairs, that is, two attributes cannot be measured at the same time. Correspondingly, if two attributes can be measured at the same time, they constitute a compatible attribute pair. For the measurement of an attribute, quantum probability and classical probability will get exactly the same result. Furthermore, for compatible attribute pairs, there is still no difference between quantum probability and classical probability. In other words, the compatible attribute operation in quantum probability has been able to cover all the contents of classical probability theory. However, for incompatible attribute pairs, many classical probability algorithms are no longer valid. The properties of classical probability system can be found in the measurement of compatible attributes of quantum probability, but conversely, the incompatible attributes in quantum probability have special properties, so it can be said that quantum probability contains more probability operation systems than classical probability.The advantages of quantum probability methods in decision-making will be demonstrated below by comparing Bayesian Network (BN) and Quantum-like Bayesian Network (QLBN) .A Bayesian Network (BN) is a directed acyclic graph in which each node represents a random variable, and each edge represents a direct influence from the source node to the target node. The graph represents independence relationships between variables, and each node is associated with a conditional probability table that specifies a distribution over the values of a node given each possible joint assignment of values of its parents.G be a BN graph over the variables Pr, over the same space factorises according to G, if Pr can be expressed as the product39.Bayesian networks can represent essentially any full joint probability distribution, which can be computed using the chain rule for Bayesian networks. Let In Eq. , \docume40.Each instantiation of the expression HG, but 40.QLBN can be defined by a pair Quantum probabilities are computed using projective rules that involve three steps. First, the probabilities for all events are determined from a state vector i and j, the entry i, column j of j, column i of Projection operators are characterized as being Hermitian and idempotent. To say P is Hermitian means that mentclass2pt{minimmentclass2pt{minim40, Jerome R. Busemeyer carried out a detailed derivation of the quantum probability distribution of single variable and multiple variables, which will not be repeated in this paper. QLBN is described in more detail below.In41. It can be described in Figs. Let entclass1pt{minimaentclass1pt{minimaIn Quantum theory, all independent quantum states contained in a Hilbert space are defined by a superposition state, represented by the quantum state vector 41:The purpose of the density operator n is the number of quantum states in the network, which contains the full joint probability distribution of classical probability if we sum the elements of the main diagonal.The density operator 42, 43 (Quantum) completely, reflects decision choices that lead to less efficient use (associated with contradictory decisions and cognitive biases).Density operators also contain quantum interference terms in non-diagonal elements, which are at the heart of the model. It is through these non-diagonal elements that one can obtain the quantum interference effect during inference, thus deviating from completely rational probabilistic reasoning. It can be seen that quantum states in QLBN allow different levels of deterministic representation, which can be concreted in Fig. 41:In the process of inference, subgroups of the quantum system need to be traced from the large system represented by the density operator e, the quantum edge probability of the discrete random variable is obtained, and the scores obtained are normalized:At the same time, the calculated complex probability amplitude is converted to the actual probability value. Given a certain evidence variable According to the expansion of the above formula, the quantum marginalization formula  is obtaiIn the above formula, if For normalization purposes, we assume that the decision maker is subjected to the same quantum interference term, i.e. 39.To sum up, quantum probability has wider physical meaning and properties than classical probability. Measurement is an important way to transform the illusory world of quantum into the real world, and human consciousness itself is transforming various possibilities into reality. This makes many scientists and philosophers think that quantum probability can not only describe the microscopic particle world, but also describe human consciousness and cognitive behaviorIn the context of related work in Part II, quantum decision model is the method of this paper to try to solve the irrational behavior and interaction of human traffic participants in autonomous driving. The following paper will compare the three models, so as to further illustrate the necessity of adopting quantum decision method in this paper.p and y. The income matrix is constructed as follows , proposed by Kahneman and Tversky, expounds many biased or irrational human behaviors in a unified way. Compared with the traditional Expected Utility Theory (EUT), CPT introduces two additional concepts in the definition of prospect: (1) P: value function CPT model assumes that the decision-maker chooses the behavior that produces the maximum value defined in , that isIn the QGT model, decision maker T is in a superposition state before observing I, after observation, the superposition state is transformed into a possible ground state, the probability of its transformation is the square of the magnitude of the probability amplitude , th, th\docuThen write the decision of target vehicle as45 obtains the parameters in CPT by Inverse Reinforcement Learning (IRL), assuming that The method in referenceFrom the simulation results Fig. , the CPTCompared with the classical Markov decision model, the results obtained by the QGT model can better explain the separation effect. MATLAB simulation platform is used for simulation, when the interact vehicle’s intention is clear, the standard quantum game model is used. The variable parameters are quantum entanglement factor γ and utility function When the intention of the interact vehicle 's vehicle is uncertain, the probability of pass/yeild the vehicle of the target vehicle is simulated by improved QGT, and the benefits of both parties are considered from the perspective of the opponent's vehicle, the parameter variables are quantum entanglement factor γ and risk aversion index In the improved model, when quantum entanglement is 0, the probability of pass or yeild the target vehicle collapses to the classical model, which is 0.5, and then fluctuates with the increase of quantum entanglement and benefit function, in the case of maximum quantum entanglement, the improved QGT model is more inclined to pass , the irrational behavior and the interaction between the two vehicles are considered in the QGT model, which is more suitable for the actual situation than the CPT model.In order to verify the effectiveness of CPT and QGT, the experimental data set scene is similar to the simulation model, in the real scene, vehicles on the two main roads are allowed to change lanes in the red box, so this is the center of game decision-making. We trained and tested two models on a data set Fig.  containi45, inverse reinforcement learning (IRL) for parameters in CPT did not take into account the influence of interaction between the two sides of the interaction, resulting in a relatively low success rate; (3) Compared with CPT, QGT takes into account the superposition state in the action set, which is more consistent with the actual situation without completely rational assumption.The verification results of the two models are shown in the following table Table . The res45, while QGT does not require a large number of data-driven methods. If our results are similar to or even more advantageous than the neural network model, the QGT proposed by us will have more theoretical value.However, we also need to demonstrate the benefits of QGT in terms of data efficiency. Neural network model achieves good results in specific scenarios through a large number of data-driven methodsExperiment 1: randomly shuffle all the trajectory pairs and select 80% of them for training and the other 20% for testing. The success rate is 65% for testing.Experiment 2: directly shuffle all frames for the neural network and randomly select 80% for training and 20% for testing. The success rate is 97% for testing.The neural network algorithm is applied to the data set Fig. . To achiThe large discrepancy between the testing accuracies of the two experiments with the NN model is mainly due to the over-fitting problem cause by the data insufficiency. In experiment 1, it showed that the NN model learned on 80% of the trajectory pairs cannot be well generalized to other interaction pairs.Table In this study, the QGT model was used to analyze the interaction between two vehicles, compared with the classical Markov dynamic decision model. This model is more practical and successfully explains the separation effect. Compared with the CPT model, the QGT model takes the superposition state in the action set into consideration and abandons the assumption of complete rationality, which is more consistent with the actual situation. When the opponent's vehicle intention is known, only the uncertainty and irrational behavior in the environment are considered, and the probability value fitting the actual situation is obtained. Otherwise, the profit value of both sides of the game will be also considered from the perspective of the opponent's vehicle, and the actual result is obtained. According to the case analysis, the QGT model owns more advantages than the classical Markov dynamic decision model and CPT model in explaining the uncertainty and irrational behavior and interaction of other traffic participants. The CPT model and QGT model are further verified by data sets, showing that the QGT model has more advantages in dealing with game scene decisions. At the same time, compared to neural network model, the result of QGT model is close to and the QGT model does not need to be driven by a large amount of data, which makes the QGT model more efficient than the neural network model.In the following work, more complex traffic scenes  are cited, and their interaction with other traffic participants in automatic driving is further explored by combining quantum theory with deep learning.Although the application research is only carried out at the simple two-car game scenario, the research method adopted is also instructive and referential for more complex scenes in automatic driving. This paper firstly attempt to apply QGT to automatic driving, providing a new reference frame for the study of decision-making problem of bounded rational behavior interaction of human traffic participants. We believe that with the further development of quantum decision theory and the continuous exploration of researchers, its application in autonomous driving will be more popular and in-depth.


Chromosome karyotype analysis is of great clinical importance in the diagnosis and treatment of diseases. Since manual analysis is highly time and effort consuming, computer-assisted automatic chromosome karyotype analysis based on images is routinely used to improve the efficiency and accuracy of the analysis. However, the strip-shaped chromosomes easily overlap each other when imaged, significantly affecting the accuracy of the subsequent analysis and hindering the development of chromosome analysis instruments. In this paper, we present an adversarial, multiscale feature learning framework to improve the accuracy and adaptability of overlapping chromosome segmentation. We first adopt the nested U-shaped network with dense skip connections as the generator to explore the optimal representation of the chromosome images by exploiting multiscale features. Then we use the conditional generative adversarial network (cGAN) to generate images similar to the original ones; the training stability of the network is enhanced by applying the least-square GAN objective. Finally, we replace the common cross-entropy loss with the advanced Lovász-Softmax loss to improve the model’s optimization and accelerate the model’s convergence. Comparing with the established algorithms, the performance of our framework is proven superior by using public datasets in eight evaluation criteria, showing its great potential in overlapping chromosome segmentation. Human chromosome karyotype analysis is of great diagnostic and prognostic value in diseases. It is usually performed in clinical diagnosis, cancer cytogenetics, and the detection of genetic abnormalities such as Edwards syndrome and Down syndrome ,2. The mSince manual segmentation is both time and effort consuming and the accuracy highly depends on the experience level of the analyst, over the past few decades, many algorithms have been proposed to automatically segment the chromosome from the images on computers . Based oWe present an adversarial multiscale feature learning framework to improve the accuracy and adaptability of overlapping chromosome segmentation.We use the conditional generative adversarial network to penalize the difference between the generated decision map and the source image, pushing the generator to produce a higher-confidence decision map for the segmentation task.Instead of using the single-scale features to represent the chromosome images, we carefully design a nested U-shaped network with dense skip connections as the generator to capture multiscale features to explore the better representation of the chromosome images.We replace the common cross-entropy loss with the advanced Lovász-Softmax loss to improve the model’s optimization and accelerate the model’s convergence.We carry out extensive experiments and analyze different objective functions that provided baselines for chromosome segmentation.In this paper, considering the various scales and overlapping regions of chromosome images, we demonstrate an adversarial multiscale feature learning (AMFL) framework that employs a nested U-shaped convolutional neural network (NestedUNet) , conditiResults show the superiority of the present AMFL method and the loss function adopted in this work in terms of visual perception analysis and quantitative score comparison.In this paper, we consider the overlapping chromosome image segmentation as an image-to-image task, in which the source overlapping chromosome images are translated to a confidence map to indicate the category information in the source images. Compared with the existing methods, we bring the adversarial learning to penalize difference between the generated decision map and source image, pushing the generator to produce a higher-confidence decision map for segmentation task. Moreover, the generator in our framework is deeper than that of CNN-based methods, possessing higher representational capacity. Specifically, as depicted in j = 0 receive only one input from the previous layer of the encoder, whereas the nodes at the level of j > 1 receive the up-sampled output of j + 1 nodes from the lower skip connection and all the outputs of the previous j nodes in the same skip connection. Therefore, a dense skip connection was constructed and multiscale features are integrated to provide better feature representation for the segmentation of overlapping chromosome regions with different scales. Meanwhile, in order to better describe network parameters, the number of filters was defined as: (1) Generator: As shown in ijth node, respectively. Note that the middle channels are the output channel of the first convolution layer and also the input channel of the second convolution layer in the nested convolution block. To describe the network structure in detail, we denote the convolution layer, batch norm layer, and rectified linear unit  except Hausdorff.To quantitatively evaluate the performance of our method, we selected eight evaluation metrics including the pixels accuracy (Acc) , dice coAcc: It indicates the pixel accuracy of the predicted results in the segmentation. In other words, it represents the proportion of pixels in an image that is correctly predicted. The Acc is calculated using Equation (8):j but actually belongs to pixel i, and c is the categories.(1) Dice: This metric represents the similarity of the predicted image O to the ground truth G. The Dice is calculated using Equation (9):G| and |O| represent the numbers of elements in the arrays.(2) IoU: This metric represents the intersection area between the predicted image O and the ground truth G, the IoU can be calculated using Equation (10):(3) Precision: It indicates how reliable the prediction is. This metric can be calculated using Equation (11):c, while c but do not actually belong to class c.(4) Recall: It indicates how sensitive the prediction is. Therefore, it is also called sensitivity, which can be calculated using Equation (12):c but actually belong to class c.(5) FNR: It is also called the under-segmentation rate, which measures the proportion of the positive classes that are predicted to be negative. It is defined as in Equation (13):(6) FPR: It is also called the over-segmentation rate, measuring the proportion of the negative classes that are predicted to be positive. This metric is calculated using Equation (14):c.(7) Hausdorff: It represents the shape similarity between the predicted images O and the ground truths G. It is calculated using Equation (15):x and y. The smaller the Hausdorff distance is, the greater the similarity between the predicted segmentation maps and the ground truth is.(8) FNR, Hausdorff, and FPR, where a lower score gives a better segmentation resultNote that, for each metric, a higher value indicates better performance, except for We validate the effectiveness of our method by comparing it with 10 recent state-of-the-art algorithms, including efficient neural network (ENet) , bilaterIn order to show the superiority of using Lovász-Softmax as the loss for overlapping chromosome segmentation, as shown in In Due to the difficulty in obtaining clinical data, we use Pommier’s overlapping chromosome datasets ,47 to deIn the training stage, all the training sets were shuffled, and all input images were normalized to the range of 0–1, and the batch size was set to 64. We optimized the generator and the discriminator alternately, both applying the Adam solver with a fixed learning rate of 0.0002 and momentum parameters of β1 = 0.5 and β2 = 0.999. Then, we set the random seed to 123. We trained our framework from scratch with the training sets to produce the “optimized” model. The training was stopped when training losses did not decrease for 200 consecutive epochs. We saved the generator model weights when the training Dice scores were at their highest. For the inference stage, we used the well-trained framework to segment the images. All the experiments were conducted in Pytorch  under anMoreover, we can also see that the various scales of chromosome individuals and overlapping regions were correctly segmented in all the images, indicating that our method performed well in the multiscale segmentation task. In order to further highlight the superior performance of our algorithm, we show confusion matrices of average accuracy scores on all the testing sets in In this section, we visually compare the performance of our method with baseline methods. In this part, we quantitatively compare the performance of our method with others and show the results in Moreover, in order to further highlight the superior performances of our present method. In To evaluate the computation efficiency, we present the total number of model parameters and the average running time of CPU and GPU when using different methods on all the testing sets in In order to analyze the role of different parts of the proposed framework, we present the average quantitative results of the proposed method by using and without using GAN with different objective functions in In this paper, we propose and demonstrate the AMFL framework for overlapping chromosome segmentation. In the network, instead of using single-scale features to represent chromosome images, we carefully designed a nested U-shaped network with dense skip connections as the generator to capture multiscale features to explore a better representation of the chromosome images. Then, we utilized cGAN to provide prior information for better discriminating features and producing highly accurate chromosome segmentation images. In addition, we replaced the common cross-entropy loss with the advanced Lovász-Softmax loss to improve the model’s optimization and accelerate the model’s convergence. In addition, we utilized the least-square GAN objective to replace the original GAN loss to stabilize the training and avoid model collapse. As for the objective function, we chose Lovász-Softmax after experimentally comparing it with others on their performance in overlapping chromosome segmentation. At last, to show the superiority of our AFML, we compared it with 10 state-of-the-art semantic segmentation methods. The results show that our AFML performed better in both visual perception and eight quantitative metrics.Currently, our AFML performs well in public overlapping chromosome datasets. However, due to the difficulty in obtaining clinical data, our well-trained model may fail to meet situations where the images exist with severe morphological inconsistency in the clinical overlapping chromosome. In future research, we intend to collect annotated clinical data and design a generalized, fully automatic system for the segmentation, classification, and karyotype analysis of chromosomes.


Living sample viability measurement is an extremely common process in medical, pharmaceutical, and biological fields, especially drug pharmacology and toxicology detection. Nowadays, there are a number of chemical, optical, and mechanical methods that have been developed in response to the growing demand for simple, rapid, accurate, and reliable real-time living sample viability assessment. In parallel, the development trend of viability measurement methods (VMMs) has increasingly shifted from traditional assays towards the innovative atomic force microscope (AFM) oscillating sensor method (referred to as nanomotion), which takes advantage of the adhesion of living samples to an oscillating surface. Herein, we provide a comprehensive review of the common VMMs, laying emphasis on their benefits and drawbacks, as well as evaluating the potential utility of VMMs. In addition, we discuss the nanomotion technique, focusing on its applications, sample attachment protocols, and result display methods. Furthermore, the challenges and future perspectives on nanomotion are commented on, mainly emphasizing scientific restrictions and development orientations. The development and evaluation of new drugs take several years of investigations on living samples to explore drug pharmacology and toxicology. Compared to in vivo investigations on living sample viability, in vitro investigations are easier to execute and duplicate, the experimental settings are easier to regulate, and they are less morally problematic and costly . In the Recently, various chemical, optical, and mechanical methods possessing high accuracy and sensitivity have been developed in response to the demand for living sample viability determination . ObviousThis review aims to address the development of VMMs from traditional assays to nanomotion and to shed light on the novelty and practicability of nanomotion VMMs. Above all, various common VMMs are discussed and summarized in order to assess the potential areas of future development by discussing their most significant advantages and drawbacks. The use of nanomotion for monitoring living sample viability is discussed extensively through a comprehensive literature survey which summarizes the applications, the methods of sample adhesion on the microcantilever, and result display methods, and concludes with a consideration of the challenges and deficiencies that need to be addressed in the future. Finally, we hope that the review can promote the development of VMMs and present a promising innovative direction.Generally, In previous reviews, according to the measurement principle or measurement procedures, VMMs have been classified in previous reviews as direct or indirect, labelled or label-free, and endpoint or real-time . While iChemical viability assays work according to a common principle—the injection of living samples with one or more compounds. For instance, an anticancer drug’s effectiveness or toxicity is evaluated through living sample interaction with drug’s compound(s) . TherefoChemical viability assays are divided into five main categories: dye exclusion assays, fluorometric assays, luminometric assays, flow cytometry, and colorimetric assays. The principle of dye exclusion assays is based on the determination of membrane integrity. Dye exclusion assays determine the viability of suspension samples, with nonviable samples appearing in blue cytoplasm and living samples appearing in clear cytoplasm. Trypan blue ,12,13,14Fluorometric assays include three methods: resazurin  assay, 5-CFDA-AM assay, and fluorescein diacetate-propidium iodide. In the resazurin  assay, a healthy sample undergoes a non-reversible enzymatic reaction that turns the resazurin or alamarblue into a pink color resorufin that spreads in the medium such that, by measuring color change, healthy samples can be calculated ,22,23. ILuminometric assays include three methods: ATP (adenosine triphosphate), luciferase, and bioluminescent-nonlytic methods. The viability of living samples is determined in the ATP assay by using luminometers to assess intracellular ATP levels after the cells have been lysed to release intracellular ATP, which interacts with the luciferase enzyme ,29,30. TFlow cytometry includes three methods: membrane asymmetry assays, membrane permeability assays, and mitochondria assays. Membrane asymmetry assays are based on detecting changes in a cell membrane’s outer surface . HoweverColorimetric assays are based on the determination of metabolic activity and can be applied to both suspensions and adherent living samples. Colorimetric assays include ten methods: MTT, MTS, XTT, WST-1, WST-8, SRB, NRU, CVS, colonogenic assay, and LDH. The MTT assay is converted into a colorful formazan by the active metabolism of viable samples, and the intensity of the colored formazan is proportional to the number of live samples ,41. UnliAn optical measurement strategy is a non-invasive way of measuring viability and monitoring the effect of drugs or toxicity  which prIn the case of living cells, the mitochondrial network is dynamic   but the Raman spectroscopy (RS) is one of the most popular optical methods used to measure living sample viability ,63. OptiRecently, an automated platform approach for high-throughput screening RS was created to overcome human factor errors, reduce test time, and increase the number of samples under measurement ,69. The Flow imaging microscopy (FIM) is a rapid, label-free method used to determine living sample viability ,74,75. ICurrently, the most significant limitation is the speed of data analysis . Micro-fHolography is a method for detecting living samples by observing morphological changes under stress or vibration resistance. Digital holographic microscopy provides a quantitative, contactless, non-destructive, and marker-free real-time monitoring method of living sample migration, adhesion, and dynamic change. It offers the possibility of measuring the efficacy of drugs in living samples ,79. DeadDue to its advantages, such as high efficiency, low cost, and flexibility to combine with other components, lens-free digital in-line holographic microscopy has become a valuable tool in the characterization and viability analysis of microbiological entities such as cancer cells . RecentlOn-chip, lensless video microscopy technology is a label-free, real-time, and non-destructive VMM technology with a field of view twice that of a conventional microscope ,90,91,92The combination of microfluidic microscopy and high pixel resolution eliminates the need for expensive lenses, light sources, and mechanical microscanning . The iteSeveral mechanical or physical techniques have been developed to quantify living sample viability. These methods are based on the principle of measuring or monitoring one of the vital activities of living samples. For any living organism, adhesion, respiration, proliferation, electrical charge, and thermogenesis process activities are vital signs of life. Several methods have been developed to measure these activities based on monitoring viability.Respiratory activity is an essential metabolic activity. The ability to absorb and consume oxygen can be an important factor in indicating the ability of a living sample to survive. Monitoring the harm produced by chemical agents to the breathing activity of a living sample is used to determine the viability of living samples based on respiratory thermodynamic features. The percentage of dead samples relative to healthy samples can be calculated by comparing the oxygen absorbed by live samples with that absorbed by the controls . MeasuriThe measurement of oxygen consumption by living samples in tissue culture flasks has been carried out using an optical oxygen sensor . The phoOxygen-sensing microplates have been used to measure living sample viability using empirical correlations between fluorescence intensity and viability . AlgoritAll metabolic processes in living organisms are heat-producing reactions . Thus, hTwo central problems with the polydimethylsiloxane (PDMS) materials used today in the manufacture of microfluidic chips show the need to use thermal materials suitable for measuring the oxygen consumed by biological organisms. The high oxygen permeability of PMDS makes respiration and viable oxygen measurements difficult. Owing to its lipophilic nature, it has well-known absorption capacities for biomarkers and medications. Thermoplastic polymer materials with low oxygen permeability, such as polyethylene terephthalate (PET) or cyclic olefin copolymer (COC), are required as chip materials.On the other hand, the process of manufacturing these thermoplastic polymer materials remains a significant challenge. The use of the manufacturing process for thermoplastic sensor integration is critical; it allows for repeatable measurements across a series of experiments. This technology, which can be easily integrated into existing thermoplastic microfluidic systems and enables living sample respiration monitoring, may pave the way for a more uniform and controlled means of monitoring culture conditions in cell-on-a-chip microfluidic systems .A microcalorimeter is an instrument designed for measuring the heat produced by microorganisms in closed bioreactors. By measuring the resulting heat employing a microcalorimeter, it is possible to directly monitor living organisms by comparing them with control samples. The percentage of dead samples relative to healthy samples can be calculated by comparing the heat energy produced by live samples to that produced by the controls . MicrocaClosed ampoule isothermal microcalorimetry has been used to evaluate the vital activities of samples in continuous real-time monitoring. In closed ampoule isothermal microcalorimetry, the heat flow between a sample and a heat sink is measured and compared to the heat flow between a reference sample and the heat sink. The measurement is carried out isothermally. The heat flux is recorded as an electrical signal after the calibration , as showThe indirect measurements of heat energy produced by living samples include non-contact temperature mapping by temperature-sensing methods. However, besides the complexity of results interpretation, accuracy is affected either by radiation absorption or the limitation of temperature resolutions. In the case of direct methods, the temperature change inside living samples can be measured by nanoscale thermal probing. However, this can cause additional heat production due to the stress response of sample rupture. The limited accuracy of the sensor’s energy resolution does not reach the level of single-sample temperature, which is at a level of pW. Therefore, most measurements are made by calculating the average heat generated by colonies of living samples, making determinations based on the calorimetry technique challenging.The calorimeter principle makes it impossible to set up a sample development environment for an extended time. As a result, calorimeters are unsuitable for research in which living sample viability must be maintained for an extended period of time . The mosDespite the literature indicating the possibility of using this method to measure cellular activities, only a few laboratories use microcalorimeters to measure vital activities. The reason for this may be high costs, which, according to the producers, are due to limited production levels. Still, perhaps by using accurate thermal electronic sensors, the demand for this technology will increase and costs will decrease .The path of research on (bio)sensors has recently turned, with great interest, to micro-nanomechanical systems as precise measurement and monitoring systems. Micro-nanomechanical oscillator sensors appear as experimental real-time measurement techniques. They have enabled the exploration of biological, mechanical, and chemical properties in vital samples and the testing of molecular interactions, biological activities, and dynamic properties at the level of a single molecule and the temporal changes in these properties .Cancer-marker detection and anticancer drug testing are increasingly needed for rapid, real-time, and high-sensitivity techniques. Micro-nanomechanical oscillator sensor techniques may provide the desired alternative . Micro-nSingle-living sample measurement is one of the most important advantages of micro-nanomechanical oscillator sensors. Measurement experiments with a sample population can only give an average measurement. This neglects the individual differences between single samples and considers them as homogeneous. A better understanding of biological processes is achieved by considering the heterogeneity of samples, especially in toxicological and anticancer drug tests . Micro-nMicro-nanomechanical oscillator sensors are vibrating mechanical structures that are often cantilevers. This microscale cantilever vibrates as an oscillating mass sensor to which the vital part adheres. The sensitivity of the vibration frequency or the oscillation amplitude depends on the mass of particles adhered to the cantilever. In oscillating mass sensors, either molecular receptors  stick on the surface of the microscale cantilever or the living sample  adhere to the surface of the microscale cantilever.Antibodies or other molecular receptors stick to the microscale cantilever surface, which is vibration movement-controlled. By moving it towards the sample, molecular recognition occurs between the sample target molecules and the sensor’s molecular receptors. This leads to mechanical, optical, or electrical interactions through which vital processes can be monitored ,123,124.Micro-nanomechanical oscillator sensors have been used to measure the interactions of surface receptors in vital samples. The chemical reactions of these receptors appear in the form of surface stress that can be traced and measured through micro-nanomechanical oscillator sensors. Surface stress caused by the receptor or the ligand causes micro-nanomechanical sensor deflection and changes in oscillation amplitude . Some stOther studies have used mechanical sensors to monitor the activities of vital samples, such as cells or bacteria, including viability ,127,128.Changing the health status of living samples alters some mechanical properties, such as weight and stiffness, and some biological properties, such as adhesion ,130,131.One piece of advanced nanomechanical oscillator sensor technology that enables real-time, direct, label-free measurement is the atomic force microscopey (AFM). Living samples are connected to an AFM cantilever and placed in a test chamber. The vibrations of the cantilever are tracked over time. An optical transduction system detects and records cantilever dynamic oscillation deflections by reflecting a laser beam from the oscillation cantilever to a detector, as seen in AFM has been used in many measurement techniques and can be described as having three main components: an imaging mode, a force spectroscopy mode, and an oscillating sensor mode. AFM can directly image single membrane proteins and living samples at nanometer resolution in a buffer solution—a crucial advantage over other microscopy techniques. Real-time AFM imaging of single living samples can provide novel insights into dynamic processes . The AFMIn measuring adhesion by single-cell force spectroscopy, a living sample is attached by force, which is contrary to the natural adhesion phenomenon. Natural living sample adhesion takes a longer time. It occurs to a lesser degree due to natural factors, such as gravity or the self-propulsion of the living sample . Living On the other hand, the traditional AFM methods provide end-state visualizations of sample fates as effects of chemical agents. They do not show the instantaneous effect of chemical agents on living samples. The real-time quality of nanomotion gives another additional advantage over AFM single-cell force spectroscopy methods.The operating principle of this method is to take advantage of the high sensitivity of the AFM cantilever. Its high flexibility allows high sensitivity to the nanomotion resulting from the change in mass due to the adhesion of nanoparticles to the cantilever surface. The fundamental principle behind these biological oscillating frequencies is the measurement of the change in frequency response caused by the additional load of biomolecular mass attached to the cantilever surface. In general, AFM oscillating cantilevers measure cantilever deflection or frequency response changes caused by a mass of adhered biomolecules . As a ref is the spring constant, and F is the acting force. In case a force is applied on a harmonical oscillation, the amplitude variation depends on the loading force, while the frequency remains constant, as shown in Equation (2) [cell on a cantilever of length L and g, E, and I are the gravity coefficient, Young’s modulus of the cantilever, and the moment of inertia of area, respectively.The deflection of a cantilever is proportional to the force. It results from the interaction of the cantilever with the sample according to Hooke’s law :Δz = kf−tion (2) :(2)z =F·Biomolecules are attached to the surface of an AFM cantilever that is implanted into a test chamber. The transformation of the cantilever oscillations over time is then tracked. An optical transduction system detects and records the dynamic oscillation frequencies of the cantilever via a laser beam reflected from the oscillating cantilever to a detector. This system’s time resolution and sensitivity are ideally suited to studying living organisms at the nanoscale ,146. TheThe AFM oscillating sensor mode or AFM nanomotion makes for easier procedures than AFM single-cell force spectroscopy techniques. Still, the method neglects the heterogeneity of cells, though this is an advantage, since we are rarely interested in single-cell behavior but instead in statistical values. The AFM single-cell force spectroscopy methods that measure a single cell’s adhesion do not yield high productivity in the measurement of anticancer drugs and cell viability. Furthermore, because of the irregular shape of single cells, theoretical models cannot describe the resulting changes in their shapes due to applying compressive or shear forces to the cells, making methods for calculating cell adhesion strength by surface stress very complex. The shape of a cell strongly influences the applied force, making the calculation of the exact adhesion force difficult.This novel method serves as a new technique for monitoring cell viability by measuring cell adhesion and thereby as a new technique for testing drug efficacy and toxicity. The changes in the values of the cantilever’s deflection and the frequency of vibration appear as a result of changes in cell state and cell detachment from the cantilever’s surface as a result of death. As a result, this method could be used to assess the efficacy of drugs and toxins. It has provided many advantages over traditional methods for measuring cell viability, including direct, real-time, and label-free measurement in tens of minutes. In contrast, traditional methods take days or even weeks. This technique complements traditional VMMs. It may be promising for the long-term development of cell viability measurement. Unlike optical and electrical measurement methods, this test can give a quick and reliable direct measurement of the viability of biological organisms, even if they are not characterized ,147.The AFM oscillating cantilevers were initially used as sensors for detecting the presence of bacteria and germs by measuring the effect of bacteria and germs attached to their surfaces in air or liquids. The presence of bacteria or germs was indicated by a change in a cantilever’s vibration amplitude and frequency ,143.AFM oscillating cantilevers have been used to study the biological activities of bacteria as well as the effects of antibiotics and medicines on them, as shown in more detail in In the case of bloodstream infection, an innovative, rapid early detection of infecting microorganisms was obtained along with an accurate determination of their antibiotic susceptibility using the AFM oscillating cantilever . NanomotThe AFM oscillating cantilever technique has been used to monitor cell viability ,151,152 2 until they adhered to the cantilever. The positions of the adhered cells on the cantilever surface were controlled without cells on the free end, which were used to reflect the laser beam [Wu et al. examined the effect of paclitaxel on the MCF-7 breast cancer cell line. The cells settled down onto the microcantilever due to gravity. The attached cells were cultured and incubated for 4 h at 37 °C in a humidified atmosphere of 5% COser beam . Withoutser beam . To asseser beam . Three dser beam .biosensThe sample’s adhesion to the surface of the microcantilever is crucial to the success of the measurement method. The measurement process principle is based on the adhesion of the cell or the bacteria to the surface of the cantilever as an indication of its viability, with non-adhesion as a sign of death. Choosing the appropriate adhesion protocol depends on the sample’s nature, size, and concentration. There are four main factors to consider when choosing a protocol for sample immobilization on the cantilever surface: the process of adhesion should take place in an environment that preserves the live sample; maintenance of the environment under the same conditions during all stages of the process; the possibility of controlling the location and number of cells or bacteria on the cantilever surface; and the risk of contamination, sample death, or cantilever damage. The direct attachment method is the most commonly used, the easiest, and the least expensive method. The cantilever is incubated in the live sample medium outside the AFM test room; see To overcome the drawbacks of direct immobilization, a high concentration of live sample medium is injected inside the test room ,150,157.AFM single-cell force spectroscopy (the micrometric motors of the AFM technique)  was usedHowever, this technique is more complex and requires expensive equipment. A sample is limited by its size and by cantilever size. There is substantial potential for misinterpretation of data due to cell damage during the adhesion process . StudiesInk-jet printing has been employed to immobilize samples on cantilevers, demonstrating its superiority as a method for nanomotion and real-time monitoring measures . The inkMany sensitive displacement sensors have been developed in order to read out and display the minute deflections of microcantilevers. Optical beam deflection, piezoelectric, and piezoresistive read-out techniques are the ones most commonly used . The mosiz) [Many studies have used the variance of cantilever deflections to compare results from different experiments, as shown in zi) :(3)Var =As the number of samples adhering to the cantilever increases, the variance bars used to show the cantilever’s deflection increase. The variance decreases as the number of samples adhering to the cantilever decreases. If the number of samples on the cantilever remains constant, the cantilever oscillates with a steady variance .Expressing the results by variance is an easy and practical method that enables clarification of the effects of chemical agents on living samples. Still, the variance shows the end state of the effect and does not show the instantaneous effect during the process. The variance from when the living samples were adhered to the cantilever and before the chemicals were injected is compared with the variance after the living samples died and detached from the cantilever. Referring to the deflection curve in A0 is the amplitude at t = 0. The damping coefficient (B) can then be estimated by solving Equation (4) as:To overcome drawbacks in displaying the nanomotion results by variance, several methods have been used to display the instantaneous effect of the nanomotion method, as illustrated in The B value is equal to the negative logarithmic amplitude ratio at any point t in the chemical agents’ effect step and the amplitude at the beginning of the chemical agents’ effect step (t = 0). The B value represents the amplitude damping rate; the higher the B value, the higher the amplitude damping rate. Hence, the more significant the chemical agents’ effect. As shown in The deflection values of the cantilever were divided into 10-second chunks. Then, a violin plot was plotted for each chunk . The vioPower spectral density (PSD) was used to represent the deflections of the cantilever, as shown in In the studies on nanomotion, several cantilever types have been used, as shown in The period in which the measurements were made ranged between ten minutes and seven hours, as shown in The AFM oscillating sensor method enables the calculation of the total sample adhesion force by recording the total value of cantilever frequency. The cantilever frequency changes with the number of valid samples still attached to the cantilever surface. The locations of these samples on the surface of the cantilever and away from the free end significantly affect the deflection and frequency of vibrations, but in the AFM oscillating sensor mode, with the final sample, the viability result is not affected by the height of the amplitude but rather by the shape of the drop in amplitude . So, if you assume the localization of samples is neglected in statistical distributions, experiments should be comparable. Thus, we propose that a method be developed in the future which will allow the location of samples on cantilever surfaces to be controlled. This will enable the value of adhesion forces to be calculated. Working on a mathematical model that shows the relationship between adherent samples’ masses and locations on the cantilever surface and the cantilever deflection values may develop the measurement of the viability of a single sample as a focal point for increasing accuracy and reliability.Nanomotion’s ability to distinguish between living samples based on their propensity to adhere to a surface opens up a wide field of other applications that enable fingerprinting of different living samples according to their health status. Nanomotion was utilized to collect brain tumor vibration signals from cultured cells based on their vibration, allowing for the differentiation of various brain tumors from the normal brain based on nanomotion characteristics .Nanomotion showed significant applicability in the real-time monitoring of the viability of live samples. However, displaying the results so as to enable the real-time monitoring of the effects of chemical agents on living samples still presents a challenge. The result display methods discussed previously are limited either to the end state of the monitoring process or to a specific point during the measurement. Deflection variance, the most commonly used method, gives a visualization of the final state of chemical agents’ effects on living samples but does not show the state of samples instantaneously during the chemical agent step. On the other hand, the deflection damping coefficient value, for example, gives a constant value for the effect on living samples, which may enable a final comparison  for the state of the living sample at a specific concentration of the chemical agent or comparison between two different chemical agents. Hence, finding a novel display method that enables the display of real-time monitoring of the viability of living samples under the influence of chemical agents may be a promising avenue for future research in this field.The same living sample and the same chemical agent have not been used in a single study with different result display methods, nor have the different methods been compared. The use of more than one result display method may give a better real-time visualization of chemical agents’ effects on viability as measured by nanomotion. A time-dependent study (amplitude modulation (AM) and frequency modulation (FM) over time) was used to show the single-cell force spectroscopy measure of yeast cell metabolism . For allIn this review, the most common methods used for living sample viability measurement have been classified and presented, and their benefits and drawbacks, as well as potential utilities, have been discussed and evaluated. The principles and features of VMMs are summarized in Additionally, cantilever type and functionalization are important factors that affect the success of nanomotion methods, while the numbers and positions of adherent samples on cantilevers are significant factors affecting the reliability and repeatability of the process. We sincerely hope that this review will drive the future development of nanomotion and provide significant thoughts for novel VMMs.


Foreground detection is an essential step in computer vision and video processing. Accurate foreground object extraction is crucial for subsequent high-level tasks such as target recognition and tracking. Although many foreground detection algorithms have been proposed, foreground detection in complex scenes is still a challenging problem. This paper presents a foreground detection algorithm based on superpixel and semantic segmentation. It first uses multiscale superpixel segmentation to obtain the initial foreground mask. At the same time, a semantic segmentation network is applied to separate potential foreground objects, and then use the defined rules to combine the results of superpixel and semantic segmentation to get the final foreground object. Finally, the background model is updated with the refined foreground result. Experiments on the CDNet2014 dataset demonstrate the effectiveness of the proposed algorithm, which can accurately segment foreground objects in complex scenes. Foreground detection is a fundamental lower-level task of computer vision. The performance of foreground detection directly affects the performance of a series of subsequent assignments, like object recognition and tracking. The primary purpose of foreground detection is to extract foreground objects from video frames and obtain feature information of foreground objects. Foregrounds are usually moving objects, such as moving human bodies and vehicles. The foreground detection process can be regarded as a classification process of image pixels, looking for differences in continuous image sequences  and extracting moving objects of interest according to the differences. The moving objects of interest are in the foreground, and the rest are classified as background. The foreground object detection is generally realized by background subtraction, which usually contains unsupervised and supervised algorithms. In recent surveys –4, we caSuperpixel is an image segmentation technology proposed and developed by Ren and Malik , which rThe rest of the paper is organized as follows: Over the past 30 years, plenty of foreground detection approaches have been studied. They can be grouped into unsupervised and supervised methods.Unsupervised background subtraction algorithm usually combines handcraft features with advanced background modeling rules to achieve foreground separation. The unsupervised background subtraction algorithm usually consists of three parts: the initialization of the background model, the extraction of the foreground region, and the update of the background model.The first unsupervised algorithm widely used for foreground detection was the Gaussian Mixture Model (GMM) proposed by Stauffer and Grimson , which iTo overcome the problems arising from the above GMM, nonparametric statistical background models were proposed to improve foreground detection performance effectively. The nonparametric estimation background subtraction methods do not need to preassume the model and estimate the model's parameters. The classic methods are Kernel Density Estimation (KDE)  and VisuSupervised background subtraction algorithms typically rely on a trained deep neural network (DNN) to segment the foreground from the background. All of these methods are divided into four steps: (1) use a particular algorithm to get a clean background image; (2) generate a dataset of specific scenes for training; (3) exploit the dataset to train the deep neural network; (4) utilize the trained neural network and the background image to perform foreground detection on the image sequence. Supervised methods can achieve good results for foreground detection tasks in many complex scenes, significantly improving the accuracy of the foreground detection.∗27 image patches is extracted from the background image and the corresponding original image and are input into the constructed CNN model. The output is the foreground probability of the center pixel in this input image patch. Because ConvNet is one of the most straightforward approaches to model the differences between the background and the foreground using CNNs, there are several limitations listed below: (1) the high-level information is challenging to learn through patches [Braham and Van Droogenbroeck  proposed patches ; (2) the patches  designedA fully convolutional network (FCN) was first proposed to process the semantic segmentation of an image by Long et al. . The FCNIn addition to the algorithms mentioned above, other supervised algorithms such as generative adversarial networks (GANs) –28, deepSSSFD combines an unsupervised algorithm based on multi-scale superpixel segmentation (MSS) and a supervised algorithm based on semantic segmentation. The detailed implementation steps are introduced in this section. We first apply a Gaussian filter to the current input image. Gaussian filtering is a linear smoothing filter that can eliminate Gaussian noise and is widely used in the noise reduction process of image processing. Gaussian filtering is a process of weighted averaging of the entire image. Each Gaussian filtered image pixel value is obtained by the weighted average of the original image pixel and its neighboring pixels. The template defines the weight and neighborhood.i is the scale subscript used in multiscale segmentation, K is the number of multiscales used, Ni is the number of superpixels under scale i, and j is the corresponding index of the superpixel block under a particular scale.Next, we use different scales to perform superpixel segmentation on the Gaussian filter image to get foreground probability maps at each scale. The number of pixels in a superpixel block, that is, how many superpixels an image is divided into, is significant. Suppose the size of the superpixel block is large. In that case, the pixels in the superpixel block may contain both foreground and background object pixels, which will reduce the accuracy and sensitivity of the final detection result. If the size of the superpixel block is small, compared with the pixelwise foreground detection algorithms, the accuracy of the last detection result is not significantly improved, and the robustness is poor. Therefore, we perform multiscale superpixel segmentation on video sequences and then execute foreground detection. We use the SLIC algorithm  to segmeR around a pixel and find the minimum value of the result as the variation of the pixel. This measure can effectively improve the robustness of the background subtraction algorithm and cope with the foreground detection of complex scenes such as dynamic backgrounds. IFD can be defined using the following formula:x, y) is the coordinate of a pixel point in the superpixel . It represents the pixel value of the current frame, Ib represents the pixel value of the background image,  is the coordinate of pixels in the neighbor region of , and R is the radius of the neighbor region.In foreground detection, the feature that distinguishes foreground and background is the RGB color feature. We use improved frame difference (IFD) to compute the difference between two corresponding superpixel patches in the current and background images. The IFD approach is not to calculate the feature values pixel by pixel but to perform a subtraction operation with a region of D is obtained from formula . |SPi,j| represents the number of pixels in the superpixel block.We use a superpixel instead of a single-pixel for background subtraction. The superpixel, including the similarity and spatial relationship between pixels, can improve the performance of the proposed unsupervised algorithm. The foreground image consists of superpixels with foreground or background labels. Whether each superpixel is marked as foreground or background is determined by the variation in the RGB color characteristics of the superpixel itself. The color feature variation of a superpixel can be calculated by averaging the color feature changes of each pixel that composes the superpixel. This average, in turn, acts as a feature variation value for each pixel in the superpixel. Formula  can repr formula , and repMSPi, j) means the foreground of the pixel with coordinate   at scale i, and Tsp is the threshold to judge whether a superpixel belongs to the foreground or not.After calculating the average value of superpixel,  formula , so as tFg is the foreground probability of the image, and K is the total number of used scales.The number of superpixels in an image will affect the results of foreground detection. We use a multiscale strategy to effectively reduce the performance degradation of the algorithm caused by the uncertainty of the number of superpixels and better detect foreground objects. The formula is as follows:Tfg is the threshold for judging the foreground.Finally, we compare the foreground probability map with a threshold to obtain the foreground detection result of the image. The formula is as follows:BSt, which are shown in the second column of We use a region-based strategy when using the IFD to calculate the pixel difference between the current frame and the background model. This strategy will lead to the detection result being always slightly larger than the ground truth, called Region of Moving Object Slightly Larger (RMO-SL), as illustrated in . TherefoC={c1, c2, c3,…, c150}. Following the same procedure as [Fc={person, car, cushion, box,book,boat, bus,truck,bottle, van,bag, bicycle}, and the background class is Bc, Bc∩ Fc=∅,  Bc ⊂ C.We adopt an advanced semantic segmentation network called DeepLabv3plus  and pretedure as , we diviN) of frames from CDNet2014 to form a training set when training a semantic segmentation network. There are usually three strategies for selecting frames [M/N, where M is the total number of video frames. The second is to choose N frames randomly from the video. The third is to choose N frames manually. Considering that the video contains many redundant frames, it will cause the problems of insufficient foreground information and an unbalanced number of positive and negative samples, which will affect the training effect of the network. Therefore, we use the manual selection of frames and select frames that contain foreground objects as much as possible for training, which will improve the accuracy of the network for foreground segmentation. Considering that there are a considerable amount of redundant frames in the video, actively screening valid frames instead of randomly selecting the training data can not only effectively reduce the amount of redundancy but also reduce the severe imbalance problem of positive and negative samples. We employ the dataset Sub-CDNet2014 provided by [Since there is no unified training dataset for foreground detection, we generally choose a certain number SStThe third column of SSt, stationary semantic foreground such as cars in a parking lot will be incorrectly classified as the foreground. This problem can be addressed by a similar approach to [SStFG and SStBG two information with high confidence in foreground and background, respectively, are derived from SSt. SStFG can be used to help reliably detect foreground pixels. At the same time, SStBG provides necessary information for detecting background pixels. They are defined as follows:SBMt(x) represents the semantic background model at the pixel x at time t. At first, initialize SBMt(x) with the first frame. We then continuously update the semantic background model to adapt to scene variations using a conservative update strategy that alleviates the problem of slow and intermittently moving objects blending into the background model. Specifically, the foreground pixels FG of the final detection result DR are not updated, and the background pixels BG are updated according to the learning rate ∂.In the semantic segmentation result, roach to , that isSBMt(x).Using the detection results of MSS and semantic segmentation, we can determine the class of a pixel according to the defined rules.SStBG, pixels with a low semantic probability should be classified as background points, regardless of the background subtraction result BSt. Depending on this rule, challenges such as dynamic backgrounds, illumination variations, and cast shadows that seriously affect the performance of unsupervised foreground detection algorithms can be solved, and false positives in such scenarios can be greatly reduced, as shown in the first and second rows of ThBG is the decision threshold value of background pixels.First, in SStFG, pixels with high-semantic probability should be classified as foreground points. This rule reduces holes in foreground objects due to occlusion or model absorption, which tends to be false negatives, as shown in the third row of ThFG denotes the decision threshold value of foreground. Specify ThFG > ThBG.Second, in BSt as the pixel class:Finally, if the value of a pixel does not meet the above two conditions, the semantic segmentation result cannot provide enough decisions to determine the classification of the pixel. At this time, we directly use the MSS result The complete collaboration process is summarized in In brief, this process can also be represented by the following formula:The background update is significant. A timely and accurate update can make the algorithm better cope with complex scene changes and improve the robustness and accuracy of the algorithm. We adopt different update strategies for the pixels detected as foreground or background.pbm is the pixel value in the background model at the same position as the foreground pixel of the current frame, and t.If a pixel is detected as a background pixel during foreground detection, the pixel value of the corresponding pixel in the background model is directly replaced with the current pixel value. We use a statistical histogram model to update the foreground pixel if a pixel is detected as foreground. The detailed procedure of the SSSFD algorithm for detecting moving objects is summarized in Algorithm 1.CDNet2014  dataset F-measure [F-measure is the weighted harmonic average of recall and precision, which is the comprehensive evaluation metric that can best reflect the performance of the foreground detection algorithm. The higher values of recall, precision, and F-measure, the better performance of the algorithm is. They are defined as follows:During the quantitative evaluation, we choose recall, precision, and -measure  to evaluThis section executes several experiments on the CDNet2014 dataset to analyze the proposed SSSFD algorithm in detail. We conducted an ablation study of SSSFD and performed qualitative and quantitative comparisons with other fundamental and state-of-the-art foreground detection algorithms to evaluate SSSFD. In this work, all experiments are implemented on a PC with a 3.6 GHz CPU, 16G RAM, and NVIDIA GeForce RTX 2080 Ti GPU in the Matlab 2020a environment under Windows 10.A few hyperparameters should be set before initializing a background model, as shown in F-measure by a significant boundary of 12.5% on the baseline category. This is mainly because the semantic segmentation network can completely segment potential foreground objects encountered in the training process. It is given a lift to F-measure by improving the foreground detection rate and more correctly recalled foreground pixels. SSSFD provides around 11.4% improvement compared to MSS in the bad weather category. MSS may be wrongly detected large snowflakes as foreground. Semantic segmentation can eliminate this false detection, reduce false positives, and improve precision and F- measure. Since MSS exploits the RGB color feature, it is easy to detect moving shadows as foreground. The high-performance semantic segmentation network can more accurately determine the shadow position through the extracted global information, eliminating the shadow. SSSFD also has a remarkable improvement relative to MSS in the shadow category, around 19.9%.SSSFD improves the F-measure across the dynamic background category is 8.3%. Although MSS can recall most of the foreground moving object pixels in the camera jitter category, some false foreground pixels are also identified simultaneously due to drastic camera jitter, resulting in low precision of recalled pixels. However, SSSFD will segment the correct foreground object for each input frame, ignoring camera shake. Compared with MSS, the F-measure value of SSSFD is improved by an astonishing 80.9%.Dynamic background category is the different kind of challenge as camera jitter. There is the movement of some background objects in the dynamic background, such as curtains moving with the wind, swaying leaves, and sparkling water. The camera jitter video background contains no moving objects. The position of the entire camera changes irregularly due to external forces , resulting in the blurred video. Camera jitter will cause the pixels on the object's edge to be detected as foreground points in the previous frame, the current frame as background points, and the next frame may be detected as foreground points. In the “fountain01” video sequence with dynamic background, the MSS algorithm detects the gushing fountain as the foreground while missing small natural moving objects, making this sequence detection fail. When the SSSFD is adopted, the false foregrounds are eliminated, and the correct ones are recalled. The F-measure has a surprising improvement on this sequence. The improvement in The MSS is removed from SSSFD, and the quantitative foreground results obtained by the semantic segmentation network (SSN) and SSSFD are compared, as shown in This section presents the exhaustive evaluation of SSSFD. The proposed algorithm is compared with eleven fundamental and state-of-the-art approaches, including unsupervised methods like GMM , KDE 1111, ViBe It is apparent from In conclusion, it is evident from the qualitative and quantitative results that SSSFD obtains better performance and is closer to the ground truth in most video sequences.This paper proposes a foreground detection algorithm called SSSFD. It utilizes an unsupervised algorithm based on multiscale superpixel segmentation to acquire background subtraction results. We train a semantic segmentation network on a subset of manually selected frames from CDNet2014 and employ this network to perform semantic segmentation on input images. Updating the background model with fine foreground results composed of superpixel and semantic segmentation dramatically improves the accuracy of foreground detection. The experimental results on the CDNet2014 dataset verify the effectiveness of the proposed algorithm. SSSFD achieves state-of-the-art performance in visual and qualitative evaluation compared with other algorithms.The limitation of the SSSFD algorithm is that the real-time performance is poor. This is because, first, multiscale superpixel segmentation takes up a lot of computational costs. Second, the processing speed of the high-precision semantic segmentation network is slow. Therefore, our future work will investigate more accurate and faster semantic segmentation networks and unsupervised background subtraction algorithms, as well as efficient fusion strategies for semantic information and background subtraction results.