In this paper, two novel and practical regularizing methods are proposed to improve existing neural network architectures for monocular optical flow estimation. The proposed methods aim to alleviate deficiencies of current methods, such as flow leakage across objects and motion consistency within rigid objects, by exploiting contextual information. More specifically, the first regularization method utilizes semantic information during the training process to explicitly regularize the produced optical flow field. The novelty of this method lies in the use of semantic segmentation masks to teach the network to implicitly identify the semantic edges of an object and better reason on the local motion flow. A novel loss function is introduced that takes into account the objects’ boundaries as derived from the semantic segmentation mask to selectively penalize motion inconsistency within an object. The method is architecture agnostic and can be integrated into any neural network without modifying or adding complexity at inference. The second regularization method adds spatial awareness to the input data of the network in order to improve training stability and efficiency. The coordinates of each pixel are used as an additional feature, breaking the invariance properties of the neural network architecture. The additional features are shown to implicitly regularize the optical flow estimation enforcing a consistent flow, while improving both the performance and the convergence time. Finally, the combination of both regularization methods further improves the performance of existing cutting edge architectures in a complementary way, both quantitatively and qualitatively, on popular flow estimation benchmark datasets. Optical flow estimation is a prerequisite step in a variety of computer vision problems, ranging from obvious ones, such as object tracking , action Training a Deep Neural Network (DNN) network for a complex problem is generally a cumbersome procedure, highly sensitive to the training set and the parameters used. In the relevant literature, different strategies have been followed to improve accuracy by either modifying the objective function or controlling the biases, also known as regularization methods. Regularization can be defined as any strategy employed to improve the training procedure of a neural network by imposing problem-specific restrictions. It has been shown that regularization can improve the behavior of DNNs , mitigatMotion consistency is a well-established principle used for the regularization of optical flow estimation networks. Neighboring points of the scene are assumed to move uniformly; a hypothesis that is accurate in most cases. However, this assumption fails significantly on object boundaries, as depicted in Optical flow and semantic segmentation may be seen as disparate modalities given that the former is about motion and the latter is about scene understanding. They are intertwined, though, as we can easily reason about the possible motions of a scene just by looking at its semantic segmentation  and, on the other hand, understand the semantics of an object just by looking the patters of its motion . The close relationship of the aforementioned modalities is confirmed in the literature, as they are jointly exploited ,11. The The current work presents two regularization methods: one explicit and one implicit. The first one explicitly penalizes motion irregularities in the resulting flow field, leading to smoother and more accurate results. The second method indirectly affects the resulting flow field and is, thus, referred to as implicit. Its regularizing effects are thoroughly analyzed quantitatively and qualitatively.The explicit regularization method aims to guide any optical flow estimation network to better model the correlation between motion flow and texture on the edge of an object. The training procedure is regularized by selectively imposing the motion consistency constraint on semantically coherent local regions of the image. For this purpose, a new loss function is introduced that takes into account the semantic segmentation mask of the examined scene to reason on the validity of motion consistency. Based on this approach, the network learns to distinguish inter- and intra-object edges and enforce smoothness accordingly. At the inference stage, the network produces sharper flow with less prevalent motion field leaks across objects and higher motion consistency within each object, without requiring any additional input or modification.The implicit regularization method aims to improve the training procedure by reducing the legitimate solution space. DNNs are designed to be translation invariant in order to improve their ability to detect textures and objects within an image. This translation invariance stems mainly from the pooling layers of the network, while convolutional layers are also contributing to it. While in some applications this is a desirable property, a recent work  exploredMoreover, an exploration of both regularization strategies, explicit and implicit, as well as their complementarity, is explored in order to better understand the role of regularization in optical flow estimation. For this purpose, the widely adopted flow estimation network FlowNet2  and its A new loss function for explicit regularization is proposed that takes into account the semantic segmentation mask of the examined scene, enforcing motion consistency within the object and sharpness on the boundaries. Different loss designs are extensively explored and tested at multiple stages of the baseline architecture.The use of pixel coordinate information as an implicit regularizer in optical flow estimation networks is proposed. An in depth exploration of the role of translational invariance on the quality of flow estimation is performed to validate our claims.The proposed regularization methods, along with their complementarity, have been extensively assessed on the optical flow estimation task in popular publicly available datasets and insights on the training procedure are provided.The contributions of this work can be summarized as follows.Regularization of optical flow estimation methods goes back to Horn and Schunck . They fon et al.  utilizedn et al.  were then et al.  combininWith the advent of deep learning, optical flow estimation methods using DNNs were proposed and the effort of the research community shifted to their optimization.The first work on optical flow estimation using CNNs was presented by Dosovitskiy et al.  who modeTran et al.  implemenFlowNet2 from Ilg et al.  focused Explicit regularization is more commonly used in unsupervised learning methods, as the common reconstruction loss function is an unstable training objective. This can be implemented in various ways, such as a consistency loss term ,31, utilExplicit regularization methods try to express some prior knowledge about the characteristics of real optical flow fields. This is typically implemented as an additive loss term. Unlike the works in ,18,19,20Several attempts have been made to exploit the interdependence of semantics and motion in order to apply some semantically-driven regularization. Ha et al.  propose The correlation of optical flow and semantic masks has been explored in literature by jointly training a deep neural network to produce both of them. However, such approaches require significant structural modifications of the architecture and induce the additional burden of having to deal with multiple steps or training objectives.As already mentioned, regularization can be defined as any strategy employed to improve the training procedure of a neural network and to reduce its generalization error. In the relevant literature, different strategies have been followed, imposing restrictions and penalties that can lead to improved performance by either modifying the objective function or controlling the biases. Regularization techniques can pursue their aim among others by (1) encoding specific prior knowledge, (2) promoting a simpler solution, (3) transforming the problem into a determined one, or (4) combining multiple hypotheses to model the training data. In this work, we are mainly focusing on (1), exploring the use of semantically-driven local motion consistency in a semantic context, and (3), reducing the spatial invariance of the network to better determine the solution space. The exploration process performed for the proposed approaches is also presented, as it can offer significant insights in the discussion of the results.Explicit regularization techniques impose some constraints directly on the output of the estimator, which in the case of deep neural networks, takes the form of a loss term. The simplest method is global consistency or global total variation, under which large, abrupt changes of the predicted motion field are penalized. Naturally, such a penalty decreases the accuracy in regions where the transition of the ground truth flow is sharp, such as on the boundary of two independently moving objects, as depicted in In such cases, a local regularization term that selectively regularizes motion consistent regions is required. These regions are bound by edges with sharp motion change, also known as motion edges. The identification of motion edges was performed by utilizing low level features such as intensity edges and gradients in their modeling , based oGiven any optical flow estimation network, we modify the training branch by adding to the average endpoint error (AEE) loss the proposed Local Smoothing Loss, as depicted in i and j refer to the rows and columns of the optical flow field, The loss is implemented as an element-wise multiplication of the consistency term with a binary mask that contains the edges extracted from the semantic instance segmentation. Motion spikes filtered by the mask are excluded from the consistency constraint, and are thus not regularized. Let us define the semantically-driven local motion consistency loss asThe idea of integrating pixel coordinates as features in CNN architectures was explored by Liu et al. . They inThe methodology was tested on the toy problem of mapping coordinates in  Cartesian space to coordinates in an one-hot pixel space, significantly improving performance and convergence. While it seems intuitive to utilize coordinates as inputs on such a task, the expected effect is heavily domain-dependent and should be evaluated on a task by task basis. In image classification, for example, the impact of CoordConv is not significant, while in object detection, generative modeling and super-resolution , importaWe claim that the accurate pixel correspondence calculation required in optical flow estimation inherently relies on such a feature. We focus on proving our claim on both small and large scale experiments and architectures. We believe that the usage of pixel coordinates can be integrated in optical flow estimation in the future. To assess its fundamental merit, we test the impact of CoordConv on a toy optical flow estimation problem. We construct a simple dataset, containing 3136, cal flow . Each reWe train a simple autoencoder-based network, depicted in The training of the network using CoordConv converges after 85 epochs, in contrast to the baseline network which fails to converge even after 175 epochs . The expwarp, and the error between Img2warp and Img1. Img2warp is computed using backward bilinear warping on Img2 using the flow field from the previous stage. As the inputs pass through the stacked modules of FlowNet2, the predicted flow gets progressively refined and approaches the ground truth, where Img2warp should approximate Img1 and the error should reach zero.After demonstrating the theoretical merit of the proposed regularization techniques, in this section, they are extensively evaluated. FlowNet2  has beenIn order to evaluate our contributions, we apply the proposed regularization methods to FlowNet S and proceed with training the network, replicating the gradual training procedure of FlowNet2. The modified FlowNet S module is constructed as depicted in The FlowNet2 building blocks were implemented in Pytorch, namely, FlowNet C and S. Due to the fact that we cannot reproduce accurately the augmentation process followed in the original work, we demonstrate results comparing our implementation and the modified version proposed. Both networks are trained identically, using the same train/test split ratio, training schedule (epochs), and learning rate.The training of the models is performed using FlyingChairs , containFollowing the procedure proposed in the original implementation, each network is first trained on FlyingChairs (Long scheduling) and then on FlyingThings3D (Fine scheduling) for a total of The evaluation is conducted on MPI Sintel , KITTI dWe conducted our experiments on commodity hardware, i.e., a Linux-based PC with an NVIDIA 1080GTX GPU, 32 GBs of RAM, i7–6700 K CPU. For our implementation, we used Python 3.6 and the Pytorch library. The training hyperparameters are the same as in the baseline reference in order to ensure fair comparability of the results.In this section, we explore the independent value of each of our contributions, by testing them on a smaller scale. We train FlowNet2 CS on FlyingChairs using different types of regularization.a in . This is attributed to the fact that Sintel has similar characteristics to the training set and small non-regularized networks tend to overfit to the training data. The use of the semantically driven local consistency loss produces results superior to the global one in all cases. As it is validated in the next sections, fine-tuning the network with a dataset rich in semantic entities generalizes better and the regularized network has better performance even in simple datasets. Similarly, networks with a bigger capacity perform better at all cases. Overall, regularizing the network using consistency constraints improves the optical flow estimation over the baseline.The qualitative difference between the two methods, is highlighted in After evaluating the effect of our novel consistency constraint as a regularizer, we move forward to incorporate the proposed modification of FlowNet S with the CoordConv layers. FlowNet2 CS and the proposed modified network are trained following the procedure described in The modified FlowNet2 CS architecture benefits from our contributions, outperforming the baseline both in terms of AEE and OP in all used datasets. The improvement is consistent in both training stages and is more evident in the realistic KITTI 2012 and 2015 datasets, where the proposed network improves the baseline by 0.5 to 1 pixel in terms of AEE and by 1 to 2.5 points in terms of OP. Moreover, comparing In the previous section, the merit of both implicit and explicit regularization has been demonstrated on the FlowNet2 CS model, as well as their complementarity. It has not yet been explored, though, how the proposed regularization methods are performing on deeper architectures that include more stacked modules. It has been shown that increased depth acts as an inherent regularizer, and therefore it is important to analyze the behavior of the proposed methods on a deeper model. For this purpose, we employ the FlowNet2 CSS architecture that stacks another Flownet2 S module on top of FlowNet2 CS. The network is trained following the same procedure, employing both a long and a fine training schedule. The evaluation is performed on four datasets and the results are presented in The proposed modified FlowNet S module is attached to the CS stack forming FlowNet2 CSS. With the weights of first stage fixed, training is continued on FlowNet2 CSS, following the same procedure. Both a long and a fine training schedule are employed and the results are shown in Quantitatively, the proposed CSS network outperforms the baseline FlowNet2 CSS on all available datasets by a significant margin. KITTI sets are the most representative ones of a real world scenario and constitute the de facto test for modern algorithms, while Middlebury is a renowned dataset with real samples. It is worth noticing that the results of the proposed CSS are lacking in performance when trained only with the simple FlyingChairs dataset, qhereas training on FlyingThings3D drastically improves the results. Despite the fact that stacking is implicitly regularizing the network, the proposed approach is further improving the results increasing the lead from the baseline in the KITTI datasets.Qualitative inspection of the results depicted in Shadows constitute a particularly interesting case for optical flow algorithms, because they break the brightness constancy constraint. More specifically, unsupervised methods that are trained with reconstruction loss produce erroneous flows at shadows, as they appear to move together with the object casting them. This is another manifestation of the flow leakage problem, as the motion vectors of the moving object erroneously leak to its shadow. The proposed method enforces flow consistency within semantic entities, reducing leakage and errors in those areas. Examples of this behavior can be seen in The aim of this work was to explore means of regularization that can improve optical flow estimation networks in terms of generalization and ease of training. Towards this objective, two regularization strategies were proposed and evaluated.The first one includes the infuse of semantic information in the training process to enable a better understanding of the scene context and, subsequently, to achieve better flow estimation at inference. The proposed semantically-driven local consistency regularization method utilizes ground truth semantic information to learn to implicitly identify the semantic edges of an object and better deduce the motion vectors around them. In the context of the optical flow estimation, semantic edges proved to be a better approximation of the real “motion edges” of an object compared to edges from a traditional edge detector, as used in the literature. The proposed method is highly versatile and can be easily integrated into existing training pipelines without modifications in the inference step.During the experiments, it was noticed that the accuracy of the results was highly related to the quality of the semantic information. The available ground truth semantic segmentation is not fully aligned with the respective images, causing inconsistencies when combined. It is believed that training with finer segmentation masks would prove even more beneficial. Moreover, the use of instance segmentation masks could further improve the results.The second method proposed deals with the use of pixel coordinates to regularize the training process. An implicit regularization method using the normalized coordinates of each motion vector as a feature was also proposed and experimentally assessed for optical flow estimation. This method is adding spatial awareness to the data input of the network, breaking the spatial invariance properties of the CNN. Extensive experimental evaluation proved that this is beneficial for the training process improving speed and reducing variance, while converging to a better solution. Although this regularization method requires some architectural modifications, minimal processing overhead is added during the inference stage.Finally, it was proven that both proposed methods are complementary and can be combined to further improve the results. The well-established FlowNet2 was used as a reference architecture to evaluate the added value of the proposed solution. The accuracy of the baseline flow estimation network is improved on both synthetic and real datasets, while minimally affecting the underlying architecture.


Enzymes are the most efficient catalysts known to date. However, decades of research have failed to fully explain the catalytic power of enzymes, and most of the current attempts to uncloak the details of atomic motions at active sites remain incomplete. Here, a straightforward manner for understanding the interplay between the complex or irregular enzyme topology and dynamical effects at catalytic sites is introduced, by revealing how fast localized vibrations form spontaneously in the stiffest parts of the scaffold. While shedding light on a physical mechanism that allowed the selection of the picosecond (ps) timescale to increase the catalytic proficiency, this approach exposes the functional importance of localized motions as a by-product of the stability-function tradeoff in enzyme evolution. From this framework of analysis—directly accessible from available diffraction data—experimental strategies for engineering the catalytic rate in enzymatic proteins are proposed. However, enzymes and, more generally proteins, are highly dynamical entities1. The knowledge of their structure’s dynamical nature is thus essential to establish a complete understanding of their functioning. From the search for catalytically optimal conformations to the transition over the barrier that defines the chemical step, protein kinematics occur over multiple timescales, ranging from ps to a few 3. Collective domain motions on the 4). In enzymatic proteins, however, faster atomic-scale fluctuations take place at active sites, influencing on reaction chemistry6. In a recent and controversial debate, these fast dynamical modes—termed rate promoting vibrations (RPV)—have been recognized as playing a key role in increasing the chemical rate in enzymes7. Several works have highlighted that RPVs correspond to localized energy that couples with the chemistry in the form of mechanical compressions between the enzyme residues and the reactant10. Active site residues have—on average—lower temperature factors than the non-active site ones. This implies that the former are in general stiffer than the latter12 and thus, active sites are subjected to higher cohesive forces15. This observation was first interpreted as functional improvement of the molecular structure through folding, as modulating the rigidity at the active site controls the thermal stability of enzymes18. Also, direct active site mutations have demonstrated to be a very efficient way of increasing protein stability19.Proteins derive their tremendous range of biological functions from the intricate patterns of interactions between each other and with the myriad of other molecules in living organisms. These interrelations have been early understood from studies of the three-dimensional (3D) structures in complex with their binding partners-known as 20. The backbone (BB) rigidity is a fundamental physical feature preserved by enzymes throughout evolution21. Consequently, it is commonly admitted that catalytic sites are energetically strained to maintain the stability-function tradeoff throughout evolution22, but is that all? This broad set of discoveries raises a central question that remains unexplored: What is the most basic physical or molecular mechanism that unifies the fast (i.e picosecond) dynamical properties with the emergence of an enzyme function  and the conserved structural rigidity? As the flexibility or the disordered nature of enzymes is well understood in conformational changes, high-strained residues, which are often regarded as order, contain an undisclosed fundamental feature of the functioning of enzymes that emerges from an intermediate time scale between the change of shape and the crossing of the transition barrier.It is further interesting to note that the secondary and tertiary structure (folded) of enzymes is more conserved in evolution than the primary one (sequence)23 and consequently produce the same quantity of modes of ps timescales  with a period of 0.38 nm, as depicted in Fig. The harmonic solutions of pulsation i is obtained from a second derivative of the energy of structure.Rigidity and flexibility have a broad sense in protein science and engineering. These concepts—borrowed from mechanical engineering—are often regarded as a static attribute. Current research associates the rigidity of the protein scaffold with thermostability, but there is more to claim. A definition of the backbone rigidity is proposed here, derived from the internal molecular degree of freedom i, the beta factor is expressed asThe temperature factor, also termed B-factor , describes the attenuation of radiations (X-ray or neutron) induced by the thermal motions. The B-factors have been extensively exploited in a wide range of studies dealing with the importance of rigidity, flexibility, and domain motions, which—as mentioned in the introduction—are primordial in proteins in general. This quantity mirrors the atomic displacement fluctuations and delivers important information about internal dynamics. From the definition introduced above for the n in Eq.  by repla25, it has been reported that enzymes are subjected to the phenomena of wave localization26 that manifests itself as a striking correlation between the locations of active sites and the formation of thermal hot spots (vibrations). This central quantity is obtained by solving the linear systemN. A real constant c is introduced such that the eigenvalues of c can be taken slightly greater than the greatest eigenvalue i can be thus expresses asa separating two residues i and j along the AA sequence. The ability of this quantity to identify where energy localizes can be assessed in Fig. From the concept of localization landscape introduced for electronsj, due to the nature of the effective potential that generally depends on The term ion, Eq.  ensures nce, Eq.  approximThe correspondence between energy localization and the rigidity profile is quantitatively validated on Fig. T,By virtue of the equipartition theorem, at thermal equilibrium with a temperature quations  and 12)T,13\docuTo sum up, when discussing structural aspects, enzymes are not just ‘ordered’ or ‘disordered’ sequences of residues as it is commonly discussed in biophysics. The particular symmetry associated with the periodic chain of AA allows wave propagation to occur . From this consideration, a mathematical operator is constructed to detect whether this energy localizes or spread everywhere in the system. Wave localization is the particular feature where vibrations are spatially confined within domains much shorter than the full sequence of residues. This mathematical approach provides a quantitative criterion to identify functional residues associated with potential fast dynamical effects. Indeed, by analyzing the so-called localization landscape, we found that fast-vibrations are clamped at the stiffer part of the enzyme scaffold. More specifically, the localization amplitude is a linear function of the local rigidity, and rigidity modulation drives the characteristic timescales associated with the dynamics of each residue. Hence, to probe fast-dynamical effects in an enzyme, one must focus on the stiffer parts of its structure.THz energy is trapped. An important finding in this work is the similitude between the BB rigidity and the energy (LL) of the protein structure  of 0.98 as exemplified in Figs. The energy localization landscape (LL) offers an accurate representation of how thermal energy of the BB is partitioned within the scaffold . In ordure (Eq. ) dictateure (Eq. ), the twThis planned comparison reveals first that, discerning the backbone rigidity profile allows one to precisely delineate where all of the fastest vibrations (i.e in the ps range) appear in the 3D scaffold . In fact, if the presence of RVPs is a prerequisite for catalysis in enzyme then, all proteins are potentially enzymes. From now, the question asked in the introduction shall be re-expressed thus: How do we accommodate topologically the recognized importance of protein rigidity with the emergence of functional residues associated with ps dynamics? In doing so, can we provide a physical basis for the existence of RPVs?To a high degree, it has been observed that more than 80% of the BB energy is localized in enzymes. In other terms, for 603 modes describing the picosecond dynamics of HIV1-protease, more than 480 have a structure predicted by the LL Fig. ; BesidesAn enzyme domain is said to be ordered if its hydropathy is enough to drive spontaneous folding. In this work, a complementary vision of the concept of order is introduced when studying ps dynamics. Indeed, thermal energy transport arises from the long-range ordering of the ues Fig.  have beeWhile RPV’s (or at least hot-spots) are highly local (dash lines), they are not uniformly formed along the amino acid chain. Indeed, the cartesian distances between ASP25 and three of the four main hot-spots are shorter than 0.5 nm. This finding suggests that multiple segments of the protein, separated by vast distances in amino acid location (> 50 residues), can be involved in creating RPVs. Interestingly, this information is partially accessible from the observation of radiation attenuation (B-factor) in a diffraction experiment, provided that structural errors associated with crystal packing are limited.In enzymes, most of the energy coming from the backbone vibrations is confined within the most rigid parts of the structure. Active sites in enzymes, because they are usually stiff, are subjected to RPVs.The geography of hot-spots is defined by the shape of the folded protein.The localization landscape is a feature conserved throughout evolution.These findings not only match the state of the art methods to understand the role of rigidity and the importance of order/disorder in enzyme-catalyzed reactions, but they also validate the debated physical origin of RPVs, which now appears as a solution to solving the problem of increasing the catalytic proficiency while maintaining the protein stability-function tradeoff using an available energy source arising from thermal fluctuations. As a by-product of this tradeoff, stiffness-driven localized vibrations provide a first glimpse into how the ps timescale was possibly selected and explain the molecular and physical origin of RPVs9.Our approach provides a physical rationale that explains the emergence of a biological function  as a tradeoff between order and disorder, which would be more appropriately termed continuity/discontinuity. Order/continuity allows energy to propagate, while disorder/discontinuity portions the structure into distinct strings of mechanically coupled residues, fed with an enhanced density of high (ps) frequency motions. Besides, a moderate correlation between our model and the temperature factor establishes the central role of the full topology in the THz regime. Although there is no explicit requirement for a universal theory of enzyme catalysis, one can extract the following features from the interpretation of the results.30. Indeed, in light of the LL, differentiation and low connectivity produce a partitioning of the BB’s thermal energy, which induces the mechanical decoupling of all AA motions, a prior step to the establishment of functional residues. Many biologically active proteins do not need to have a unique fixed and rigid 3-D structure to execute a function32. Examples are the abundance and functional importance of intrinsically disordered protein (IDP) but not exclusively. Much than 44% of the human proteome contains intrinsically disordered peptide segments more significant than 30 residues in length33. Surprisingly, the majority of disordered peptide segments have yet no known function34. Though, if one considers protein dynamics from its vibrational energy property , disordered regions could be a prerequisite for allosteric pathways: By damping mechanical waves, they promote coupling between specific residue. Therefore, an additional by-product associated with the rigidity would allow solving the problem of regulation with an evolutionary strategy that consists in modulating to the pattern of mechanical damping, such that signals propagate between selected groups of distal residues.Previous works have hypothesized that differentiation and low connectivity between a protein’s scaffold and its active-site is a crucial prerequisite for innovabilityHence, from this study, it appeared that energy/information transport requires, on the one hand, long-range symmetry to emerge from random thermal fluctuations. To take advantage of this available energy in a functional form, folding produce discontinuities of the scaffold rigidity that comes to dictates the configuration of energetic hot-spots, or possible functional residues. These hot-spots fuels the enzymes with potential RPVs and would eventually become active sites throughout the natural selection. Putting forward this analysis in the line of a canonical Darwinian vision, the energy localization pattern—as much as its counterpart, the free energy landscape—appears as an essential  attribute of enzymes evolution extensible to all proteins in general. Hence, the conventional paradigm of the enzyme structure that encodes the function is prolonged here to the structure that encodes the dynamics and the catalytic rate acceleration. Together, both structure and dynamics encode the biological function. We finally propose a new class of experiments that can be achieved from the basis of this theory.To determine the ideal sites for a small-molecule to bind and influence the catalytic rate, one first extracts the coordinates of the localization hot-spots . By correlating these strings of hot-spot residues to a measure of the solvent’s accessibility, one can predict potential candidates (residues) that will affect the catalytic rate upon binding. We tested the possibility of this instrument on active sites residues Asp25Thr26Gly27 located on the localization hot spot corresponding to residues ranging from 21 to 31. Potential binding site candidates such as GLU21 as well as ASP30 have been extracted Fig. .Considering the extremely light computational costs of this rigidity-based LL, it is possible to investigate almost instantaneously the effect of distal mutations on the dynamical properties of catalytic sites. Such a direct approach to modulate RPVs’ effects allows narrowing the results of large-scale screenings considerably.However, since this work is currently based on ENM, there are several limits, notably regarding the possibility to investigate the influence of co-factors on the LL. Additional investigations are currently conducted to extend this dynamical analysis using more realistic force fields with MD simulations to produce more accurate dynamical matrix.In this paper, we have introduced an atomistic and robust structure-based approach that identifies the structural rigidity of the BB as a key quantity to understand and control RPVs at catalytic sites. Our results clarify the deep connection between an enzyme structure, the rigidity arising from the molecular interactions, and the subsequent picosecond dynamics that emerge to directly modulate the catalytic rate at active sites in a fold-encoded fashion. Hence, in enzymes, the observation of discontinuity of the BB stiffness provides a unified picture of how the search time to actives configurations splits into two distinct but complementary dynamical scales: slow conformation changes by soft domains, and fast promoting vibrations occurring at the stiffest parts. RPVs or fast dynamical effects appear as a by-product of the stability-function tradeoff in enzymes and possibly in proteins in general. This molecular theory paves the way toward a rational design of catalytic proteins and provides simple conceptual tools to guide experiments addressing the influence of vibrations on chemistry.


The achieved relative accuracy is at least 4X better than the existing state of the art. The SSFC shoe prototype is built using readily available soccer shoes and piezoresistive FlexiForce sensors. To improve the wearability and comfort of the instrumented shoe, a semi-computational sensor design methodology is developed based on an equivalent-area concept that can accurately account for SSFC’s with arbitrary shapes. The search space of the optimal SSFC design is shown to be combinatorial, and a high-performance computing (HPC) framework based on OpenMP parallel programming is proposed to accelerate the design optimization process. An optimal sensor design speedup of up to 22X is shown to be achievable using the HPC implementation.Traditional pedobarography methods use direct force sensor placement in the shoe insole to record pressure patterns. One problem with such methods is that they tap only a few points on the flat sole under the foot and, therefore, do not account for the total ground reaction force. As a result, body weight tends to be under-estimated. This disadvantage has made it more difficult for pedobarography to be used to monitor many diseases, especially when their symptoms include body weight changes. In this paper, the problem of pedobarographic body weight measurement is addressed using a novel ergonomic shoe-integrated sensor array architecture based on concentrating the applied force via three-layered structures that we call Sandwiched Sensor Force Consolidators (SSFC). A shoe prototype is designed with the proposed sensors and shown to accurately measure body weight with an achievable relative accuracy greater than 99%,  Pedobarography is the art of measuring and analyzing pressure forces exercised by the foot on the ground during the walk cycle. Wearable pedobarography is a highly promising research area whose results are expected to impact a range of healthcare applications such as gait biometrics, activity monitoring, weight estimation, prosthetics, physical therapy, Parkinson’s disease, neuro-motor disorders, obesity control, and sports medicine. Other engineering areas should profit from wearable pedobarography, including robotics, security, and entertainment.In the specific context of healthcare applications, it is well known that the short-term change patterns of body weight are correlated with several physical, mental, and emotional health problems. A partial list of medical conditions with such correlations is shown in Researchers have used optimization to suitably place shoe-integrated FlexiForce pressure sensors under the feet. However, such optimizations cannot be reliably generalized due to person-to-person gait and foot variabilities. A comparative study of footwear-based wearable systems  lists seThe main purpose of this paper is to address the above challenges in wearable body weight estimation using a novel approach to shoe-integrated sensor design and characterization. The main sensor design innovation is the integration of force sensors in three-layered structures that act as force consolidators and channel the total applied force to a set of critically placed points under the foot. The second design innovation is a semi-computational methodology for the fast extraction of transduction characteristics of the sensing layer in the multi-layer structure. Driven by ergonomic and comfort considerations, such semi-computational approach helps accelerate the design iterations of multi-layer sensing structures of arbitrary shape, while mitigating the inefficiencies of manual designs based on custom fabrication or automated designs based on full 3D finite-element-method (FEM) analysis of the shoe-integrated sensor.Sensor Design: A novel method for designing the force sensors is proposed. A layered structure called Sandwiched Sensor Force Consolidators (SSFC) is used under the shoe sole to concentrate and channel the applied forces. Each SSFC is a three-layer structure made of a top and bottom capping layers, and a middle sensory layer, as shown in Feasibility Analysis and Proof-of-Concept: A fully instrumented proof-of-concept shoe prototype has been built and tested on a human subject. It has been shown to achieve a weight estimation accuracy greater than 99%, which is at least 4X better than the state-of-the-art [-the-art ,8,9.Ergonomic Shoe Design and Custom Sensor Characterization: To improve the wearability comfort of the proof-of-concept prototype, the paper introduces a streamlined, efficient, equivalent-area methodology for the extraction of the transduction characteristics of SSFC sensor structures of arbitrary shapes. The methodology is fully self-contained and does not require external or vendor 3D FEM codes for shape-from-force analysis.High-Performance Processing: To speed up sensor characterization and the design optimization loop of the semi-computational methodology, a parallel programming interface has been implemented and tested using C and OpenMP. It has been shown to reduce the shoe design optimization runtime from few days to few hours.The main contributions of this paper are as follows:The remainder of this paper is organized as follows. The literature on shoe-integrated sensors is rather large and goes back to the seminal work of . Given tBroadly speaking, the prior art in this area can be categorized into two categories. The first comprises the sensing systems that end up collecting only a portion of a GRF while the second comprises those that collect the total GRF. To the first category belong the applications that target the observation of relative foot pressure patterns without much dependence on the exact magnitude of the ground reaction force. Research in gait analysis and recognition as described in ,13,14,15total GRF, has more variety in its transduction modalities. This is because the main goal of shoe-integrated, total GRF measurement has been to develop an ambulatory equivalent of the static force plate system, which is considered the golden standard in bio-mechanical and sports medicine research [The second category of the prior art, where the measurement targets the research ,24. As aresearch  in whichresearch , where tresearch . The optresearch ,27, wherresearch , where fA counterpart of  for the Bodyweight measurement using GRF has been directly addressed in ,9 where Sandwiched Sensor Force Consolidators (SSFC) under the shoe sole, as shown in We propose a method for shoe-integrated sensor array design to mitigate the “spreading-out” effect of the total GRF. The main idea is to concentrate the applied forces on a set of critical points placed under the foot. To do so, we build The SSFCs are attached to the bottom side of the shoe sole, as shown in To facilitate contact with electrodes, the shoe insole is used to pass flexible wires with sideway connection with the electrodes, as shown in As will be explained shortly, we have determined that 6–8 consolidators, properly located in the sole, are sufficient to collect the total GRF. This is the design guideline we have implemented in our prototype and used in all our experiments. It should, however, be understood that under one shoe, there can be as many SSFCs as needed. A larger number of SSFCs can be used to fabricate a grid of sensory consolidators that help to capture the two-dimensional distribution of the local GRFs. Such distributions can be analyzed in a plethora of applications other than body weight measurement and walk cycle analysis. These applications include fall detection, body balance, and gait analysis. In our prototype, each SSFC sensor has an applied-force rating in the range of 220–445 Newton, but of course, other ratings are possible as may be the case for the physical therapy of a toddler or the monitoring of post-stroke body balance of an obese patient.In this subsection, we briefly highlight the implementation of a proof-of-concept prototype shoe where we have used the cleat-like structure of soccer shoes and the piezoresistive force sensors, FlexiForce, to create SSFCs according to the concept proposed in Sensor shape pixelization and mapping onto a square of equal area.Z-direction discretization of the sensing material thickness.Rotationally symmetric force distribution over the equivalent square area.The soccer-like shoe used in the prototype of the previous section is of course not suitable for comfortable daily wear, especially in the context of continuous monitoring of a patient’s weight and gait. In this section, we show that the SSFCs don’t have to be embedded in cleats but that a more ergonomic and comfortable sensor array design is feasible. We do so by introducing a semi-computational approach for designing SFFCs of arbitrary shapes. The challenge is to show that the transduction characteristics of such custom SFFCs will satisfy the requirements of an accurate ground-force reaction sensor array. As mentioned, SSFCs are necessary to address measurement errors due to the spreading-out effect of the total ground reaction force. However there are other sources of measurement errors. One such source is the potential inaccuracy in the transduction coefficient of the SSFC sensing layer itself. This transduction coefficient depends on the shape and size of the consolidator, the materials used therein, and their layer thicknesses. The transduction coefficient may be determined either empirically or computationally. The empirical method is based on the direct measurement of a fully fabricated consolidator sample. Such a method is tedious, wasteful of materials, and is not compatible with design iteration or optimization. The computational method typically uses 3D FEM codes which require the meshing and analysis of complex 3D shapes such as the ones shown in The end goal of the SSFC design process is to conjure up, in the shortest possible time, an ergonomic, comfortable shoe design capable of accurate body weight measurement. An important outcome of the SFFC design process is the transduction model of the sensor array that enables the instrumented shoe to determine in real-time the ground reaction force. The steps involved in the design and characterization process are explained in the following subsections.nD), the number of SSFCs per design (DnS), the number of shoe sizes (nSS), the physical size of each design (nM), the sensing layer thickness range (nT), the expected force range (nF), and the FEM grid size  are extracted from the image using the start and end pixels of the columns  of the individual SSFC and its dimensions are extracted as shown in This pixelization is needed to map the shoe sole Computational Efficiency: As will be explained shortly, the SSFC is meshed into small elements, and the square transformation eliminates the need to access elements outside the sensing area. For example, the shape in Algorithmic Simplification: The quantity of interest is the cumulative change in the physical property of an SSFC rather than the change gradient across its surface. Therefore, preserving the SSFC geometry is unnecessary. This helps in simplifying the computation of additional effects such as the fringe effects in capacitive sensors. It is well known that it is quite complex to account for the fringe capacitive effects of an irregular shape, while a regular one allows the use of existing true-and-tried formulas.Realistic SSFC Simulation: The shape transformation further enables the use of force distribution models that result in realistic vertical displacement maps and, therefore, realistic SSFC transduction characteristics.Reduction of Approximation Errors: Finally, it helps in reducing the approximation errors inherent in the meshing of a complex shape and the discretization of its boundary.Each SSFC shape is then transformed into a square  whose area e in the grid are given as:The force distribution model is meant to mimic the forces that are applied across the SSFC during a given phase of the walk cycle as, for instance, during heal impact. The model is a two-dimensional distribution whose definition is under the control of the designer. It is defined over the equivalent square area as introduced in the previous paragraph. As shown in F is applied, it will be distributed across the equivalent square according to N mini capacitors between the top and bottom electrodes of the SSFC, as shown in The force distribution function itself may be either deterministic or stochastic. For illustration purposes, a rotationally symmetric normal distribution function is used in this paper as shown in loop3 iteration , the computation of transduction characteristics is based on a set of four nested iterations. The outermost iteration, loop4, is over the materials, M, provided by the designer. For each M, the second iteration, loop5, is over the thickness of the material, d, for a range also set by the designer  can be measured and converted to a suitable electrical change using appropriate electronic circuitries. As will be shown in loop1, the computation results in The change in thickness,  follows (4)Δdi,jIn this section, we will present the implementation of both the feasibility prototype to validate the proposed sensor design approach, and the computational framework to accelerate the process to design ergonomic shoes based on our sensor approach.The prototype data acquisition system, shown in A computational efficiency component to illustrate the significant reduction in processing time for ergonomic shoe design process.A characterization component to illustrate the accuracy of the sensor characterization technique.In this section, the semi-computational methodology based on the area-equivalent approach is implemented using a high-performance computing (HPC) framework. The implementation comprises two components:As highlighted earlier in To validate the area-based computation of transduction characteristics, we have selected a circular capacitive sensor similar to the one from SingleTact . The maigeometry .(7)ΔL=d include ,40.The change in geometric shape during the shape transformation process does also affect the fringe capacitance, which must be adjusted accordingly. This is because while the area remains the same, the perimeter changes. To account for such change, the fringe capacitance  at the Masdar Institute, now part of Khalifa University, Abu Dhabi, UAE, under Protocol HSREC#2013-16.), wearing the instrumented shoes. Under the second test, the measurements are taken whether the subject is walking or standing. The subject is 5The functionality of the OpenMP parallel implementation of sensor characterization is verified both on a personal computer with four cores and at Khalifa University’s HPC cluster. A range of data sets of various sizes is used in the numerical experiments to compute the SSFC transduction characteristics and their computation times. Several runs are conducted while varying the number of processor cores. The number of processor cores and data sizes are shown in An alternative compute infrastructure to the HPC cluster is a graphics processing unit (GPU) running on a personal computer with minor GPU-specific changes in the OpenMP directives of the code.D = 15 mm with a capacitive sensor of dielectric thickness d = 0.15 mm, and dielectric constant To validate our sensor characterization method, we have selected a circular SSFC of diameter ngleTact . In one or as in . The errThe sensor characterization method results in several output data sets for the user-defined sweeps over materials, designs, and sizes as well as over force and thickness ranges. Such data sets can be used to analyze the transduction behavior of the SSFCs. These data sets also include the even in the presence of motion. The achieved relative accuracy is at least 4X better than the existing state of the art.This paper has addressed the problem of accurate body weight estimation during walk using a novel shoe-integrated sensor architecture. The main concept is to concentrate the applied force using multi-layered, cleat-like structures that are called sandwiched sensor force consolidators (SSFC). Such structures are crucial in the reduction of the spreading-out effect of the ground-reaction forces. The paper then addressed the efficient characterization of such force consolidators using an equivalent-area methodology that can accurately account for SSFC’s with arbitrary shapes. In particular, a direct mapping has been derived that relates the sensed force to the SSFC dimensions and material properties. The issue of the computational efficiency of the equivalent-area characterization method has also been tackled and an OpenMP parallel programming interface has been implemented and shown to reduce the total processing time of a full shoe-integrated sensor array from few days to few hours. The design methodology has been implemented in a working prototype that has been fully tested using both static weights and the walking subjects. The achievable relative accuracy of body weight estimation was shown to be greater than 99%, While the major objective of the present work is the development of ergonomic, shoe-integrated sensors for medical body weight measurement, it is conceivable that the accuracy attained will enable more precise methods for several adjacent applications such as gait analysis, sports medicine, and physical therapy.


With the rapid development of information and communication technology, the wireless sensor network (WSN) has shown broad application prospects in a growing number of fields. The non-line-of-sight (NLOS) problem is the main challenge to WSN localization, which seriously reduces the positioning accuracy. In this paper, a robust localization algorithm based on NLOS identification and classification filtering for WSN is proposed to solve this problem. It is difficult to use a single filter to filter out NLOS noise in all cases since NLOS cases are extremely complicated in real scenarios. Therefore, in order to improve the robustness, we first propose a NLOS identification strategy to detect the severity of NLOS, and then NLOS situations are divided into two categories according to the severity: mild NLOS and severe NLOS. Secondly, classification filtering is performed to obtain respective position estimates. An extended Kalman filter is applied to filter line-of-sight (LOS) noise. For mild NLOS, the large outliers are clipped by the redescending score function in the robust extended Kalman filter, yielding superior performance. For severe NLOS, a severe NLOS mitigation algorithm based on LOS reconstruction is proposed, in which the average value of NLOS error is estimated and the measurements are reconstructed and corrected for subsequent positioning. Finally, an interactive multiple model algorithm is employed to obtain the final positioning result by weighting the position estimation of LOS and NLOS. Simulation and experimental results show that the proposed algorithm can effectively suppress NLOS error and obtain higher positioning accuracy when compared with existing algorithms. A wireless sensor network (WSN) is a self-organized network consisted of a large number of inexpensive sensor nodes, which is increasingly widely applied in industrial production and social life . LocalizThe NLOS identification strategy is proposed to detect the severity of NLOS and further divide the NLOS situation into mild NLOS and severe NLOS. Compared with the traditional algorithms, the proposed algorithm can obtain more NLOS environment information through NLOS identification, which is beneficial to improve the positioning accuracy and robustness of the algorithm.The proposed algorithm has strong robustness even in a severe NLOS environment. A severe NLOS mitigation algorithm based on LOS reconstruction is proposed to deal with severe NLOS. When severe NLOS occurs, the average value of NLOS error is estimated and the measurements are reconstructed and corrected for subsequent positioning, so our algorithm has better performance in severe NLOS environment when compared with the traditional algorithms.By contrast with many algorithms, the proposed algorithm does not assume the prior information of NLOS error. Even so, simulation and experimental results show that the proposed algorithm still effectively eliminates NLOS error when NLOS error obeys different distributions. Without losing generality, the proposed algorithm uses TOA measurement, but it can be extended easily to other signal features such as TDOA, RSS or AOA.The clipping parameters In order to obtain the accurate position of the target node, many algorithms have been proposed to solve the localization problem. The EKF  algorithThe organizational structure of the remaining papers is as follows. The related works and problem statement are described in In order to reduce the influence of NLOS transmission, many scholars have carried out a series of studies. Haigh proposed a novel NLOS ultrasonic signal classification and rejection algorithm , which uData fusion is also one of the research hotspots. In , by combMany conventional algorithms need to know the prior information of the environment in advance. How to reduce the dependence on the prior information of the environment is also one of the research directions. In , Yu estarithm in  to constM beacon nodes whose coordinates are known, and the signal transmission between the mobile node and the beacon nodes can be LOS or NLOS depending on whether there are obstacles between the mobile node and the beacon nodes. The mobile node moves randomly in the localization area. In the process of mobile node moving, the signal transmission channels between the mobile node and the beacon nodes switch between LOS and NLOS conditions, and the switching is regarded as a second-order Markov transition process [l. The movement of the mobile node can be described by the change of the state vector, which can be modeled as:A is the state transition matrix of the mobile node, and G is used to describe the random acceleration caused by noise In a two-dimensional localization area, we assume that there is one mobile node surrounded by  process  as shownM beacon nodes at time step l. The element m-th beacon node and the mobile node. Then,m-th beacon node and the mobile node at time step l, which is defined as:m-th beacon node. Noise vector Let We assume that the sensor noise variance The IMM localization algorithm regards the signal propagation channel as a second-order Markov transition process . Four maN different subgroups, and each subgroup separately calculates its own position estimation by the least-squares, and then NLOS identification is carried out on N different position estimation results using the validation gate. The position estimation that can fall into the validation gate is a positioning estimation with higher positioning accuracy. We count the number The flowchart of the proposed algorithm is illustrated in l-1 are initialized respectively [i-th mode. i to mode j. Note that First, the state vector ectively . Then, wj-th mode-matched filter at time step l − 1. After the interaction is completed, the algorithm starts to perform model matching.Then input interaction is carried out:l − 1, which are employed to compute the state prediction value and the error covariance matrix prediction value for LOS/NLOS at time step l.Prediction: through the interaction process, we obtain the mixed state estimation and the mixed state covariance matrix at time step Before suppressing NLOS noise, a NLOS identification strategy is proposed to detect the severity of NLOS.M beacon nodes and mobile node into n-th subgroup at time step l. After calculating the position estimation of each subgroup, we conduct NLOS identification to detect the severity of NLOS pollution.Since the position estimation requires at least 3 measurement values, we first divide the distance measurement values of n-th subgroup at time step l, calculated as follows:When there is no NLOS error in all range measurements of the subgroup, then the following formula holds:To test Equation (17), we define hypothesis n-th subgroup are in LOS condition, the position estimation n-th subgroup is in NLOS condition, the position estimation If all beacon nodes of the By comparing We count the number of If If If j = 1, EKF filter is used to update the state estimation and error covariance matrix.l, l.Update: when j = 2, the REKF algorithm and a proposed severe NLOS mitigation algorithm based on LOS reconstruction are employed to filter mild NLOS and severe NLOS, respectively.When The REKF algorithm is described as follows:l.(1) Rewrite EKF equation: it was proved in available research works ,25,26 thThen, the state equation and the measurement equation in (1) and (3) are rewritten as follows:At the same time, l, by solving the following coupled equation, we can obtain the maximum likelihood estimation of (2) Robust regression: considering the case of the linear regression model 37) at time step -squares , we can echnique  to solveechnique  as the l at time r-th iteration.In the first step, an initial state estimation In the second step, the initial state estimation The third step is to estimate the noise scale The fourth step is to update the state estimation by a Newton–Raphson step. The new state estimation is computed as follows:In the fifth step, the convergence of the algorithm is checked according to a preset convergence condition. If the norm of the current state estimation and the last estimation is less than the required value, the iteration is stopped to obtain the final state estimation, otherwise, the above steps are repeatedly executed until convergence.After obtaining the state estimation, the likelihood function is calculated as follows:The severe NLOS mitigation algorithm based on LOS reconstruction we propose is described as follows:LOS reconstruction: When NLOS pollution is especially serious, if we can estimate the average value of NLOS error and perform LOS reconstruction on the measurements from each beacon node, the negative impact of NLOS error on positioning performance can be alleviated to a large extent. We first estimate the average value of NLOS error l. when l can be computed by using the position estimation Update l.Since the NLOS error is a deviation greater than zero, we take the average value of all values larger than zero in the array Distance reconstruction:l, l.Then the NLOS EKF filter is used to estimate the position using the measurements after LOS reconstruction.c denotes the updated mode probability and the normalization factor respectively.See belowThe output state vector estimation The simulation environment and parameter settings for simulation are introduced in this section.Firstly, we introduce the simulation environment. The simulation platform is MATLAB, and every simulation result is based on 1000 Monte Carlo experiments. The simulation scene of a Monte Carlo experiment is depicted in onal EKF , IMM-EKFonal EKF  and R-IMonal EKF  algorithIn this algorithm, we assume that the sensor noise variance M, as in . In addiALE refers to the average distance deviation between the estimated position and the real position. l. The performance of the algorithm is evaluated through two indexes: cumulative distribution function (CDF) of the average localization error (ALE) and root mean square error (RMSE).In order to simulate a complex positioning environment, simulations are respectively taken under the condition that NLOS errors follow different distributions: Gaussian distribution, uniform distribution and exponential distribution. The robustness and positioning accuracy of the proposed algorithm with the change of the mean value of NLOS error, the standard deviation of NLOS error, the probability of NLOS error, and the number of beacon nodes are explored based on the comparative scheme designed. Simulation results are shown and evaluated in We first assume NLOS error and sensor noise follow Gaussian distribution To better illustrate the robustness and positioning accuracy of the proposed algorithm as the mean value of NLOS error The positioning accuracy and robustness of the proposed algorithm when the probability of NLOS error varies from 0.1 to 1 are shown in We assume that that NLOS error and sensor noise follow uniform distribution The RMSE result of each algorithm with the parameter max changing from 8 to 15 is described in We assume that NLOS error and sensor noise follow exponential distribution The RMSE of each algorithm with the parameter To describe the robustness of the proposed algorithm as the NLOS error probability From In order to test the performance of the proposed algorithm, we conducted a localization experiment in an indoor environment. The signal transmission between mobile node and beacon nodes was based on ultra-wideband (UWB) technology. UWB technology has been widely used in indoor positioning in recent years because of its low complexity and high positioning accuracy. The principle of UWB ranging is to approximate the distance between nodes based on the two-way propagation time of signals. In this experiment, both the mobile node and the beacon nodes adopted UWB node. The difference was that the mobile node was connected with the power supply as shown in The experimental scene is depicted in The localization error at each sampling point is illustrated in This paper proposes a robust localization algorithm based on NLOS identification and classification filtering for a hybrid LOS and NLOS environment. Firstly, an NLOS identification strategy is proposed to detect the severity of NLOS, and then the NLOS situation is further divided into mild NLOS and severe NLOS, which is beneficial to the subsequent classification filtering. After that, we performed classification filtering to obtain respective position estimates for different NLOS situations. For the LOS case, the EKF algorithm was applied to generate accurate positioning results. For the NLOS case, REKF algorithm and the proposed severe NLOS mitigation algorithm based on LOS reconstruction were employed to filter out mild NLOS noise and severe NLOS noise, respectively. Through NLOS identification and classification filtering, the proposed algorithm could deal better with various complex positioning environments and had stronger robustness. Finally, the IMM algorithm was employed to obtain the final positioning result by weighing the position estimation of LOS and NLOS, which further improved the positioning accuracy and robustness. Simulation and experiments showed that the proposed algorithm outperformed EKF, REKF, IMM-EKF, R-IMM in the hybrid LOS and NLOS environment. In addition, under the unfavorable conditions of fewer beacon nodes participating in the positioning and higher NLOS probability, the performance of the algorithm did not drop significantly, which shows stronger robustness. In the future, we need to further improve the positioning accuracy when the number of nodes participating in the positioning has enough redundancy.