The modelling result showing the measurement and transient behavior of different parameters is presented. The numerical results would be both qualitatively and quantitatively validated. The presentation of unstable state profiles and heat variables along the annulus are also discussed.Computational fluid dynamics (CFD) has become effective and crucial to several applications in science and engineering. The dynamic behavior of buoyancy induced flow of water in partially heated tall open-ended vertical annulus is analyzed based on a CFD technique. For a vertical annulus, the natural convective heat transfer has a broad application in engineering. The annulus is the most common structure used in various heat transmission systems, from the basic heat transfer device to the most sophisticated atomic reactors. The annular test sections of such a large aspect ratio are of practical importance in the design of equipment’s associated with the reactor systems. However, depending on the geometrical structure and heating conditions, it exhibits different flow behavior. The annulus may either be closed or open-ended. In this study, we carry out CFD analysis to examine the thermodynamics properties and the detailed thermal induced flow behavior of the water in Tall open-ended vertical concentric annuli. The purpose of this study is to evaluate the impact of a partially heating on mechanical properties and design parameters like Nusselt number, mass flow rate and pressure defect. For Rayleigh number ranging from 4.4 × 10 The process of energy transfer on a surface to a fluid flowing over it; as a result, the difference between them is referred to as convection heat transfer. Since the rate of convective heat transfer is influenced by the flow field of the fluid, it will strongly depend upon how the flow is generated. For forced convection, the flow of fluid is due to some external means such as fan or pump. While in free convection, the body forces are responsible for the flow and that occur due to the density difference arising caused by the changes of temperature in the flow field. The body forces are generated due to pressure gradients imposed on the whole fluid due to gravity. This results in the buoyancy force that causes the lighter fluid to rise upward. Therefore, the buoyancy induced flow is generally due to density gradient in the flow, which can also occur due to concentration gradient and temperature gradient. Thermo-siphon action is induced either in the cavity or in closed loop geometry due to heating and cooling of a fluid differentially. Energy systems (solar collectors), wall with multi-layers, windows with double glasses, in nuclear reactors, furnaces of distillery plant and several heat exchangers is also possessing thermally induced flow. Some practical structures like tubes of circular, annular cavities and parallel plates often require convective transfer of heat, mass flow in heated vertical open-ended annuli. The practical importance of such system are particularly the fuel elements of nuclear reactors, arrangements of double pipes, process of chemical distillery, solar energy collector for channel type, thermo protection system, electrical cables (gas cooled). Significant interest has been shown in these problems with natural convective heat transfer in recent years.5, 0.5 < Pr < 5, radius ratio from 1 to 4 and aspect ratio varies from 1 to 20. The motion was found to be produces by the gradient of radial density induced by thermal boundary conditions, maintaining the internal cylinder heated and the outer cylinder cold. The motion also consists of a single cell with low Rayleigh numbers, while a multicellular motion on large Rayleigh numbers can be observed. Mochimaru [3 to 105. The inversion parameters were 0 to 1, the eccentricity was 0 to 0.8 and the cylinder orientation angle was 0 to π. The results indicates that the characteristics of transfer of heat and flow pattern are mostly affected by a combined effect caused by the water density inversion and the inner annulus cylinder position. El-Shaarawi and Al-Attas [Thomas  analyzedochimaru  carried Al-Attas  studied Al-Attas  conducteAl-Attas ,10. MohaAl-Attas  performeAl-Attas  conducteThe schematic diagram of the problem for numerical analysis is shown in 1 = 21 (dimensionless height). From Z = Z1 = 21 to Z = Z2 = 300 is the heated region for inner cylinder while Z = Z2 = 300 to Z = A = 352 is non heating zone near the exit for inner cylinder while the outer cylinder assumed adiabatic. In radial directions Ri = ri/b and Ro = ro/b.(a)Full HeatingPartial HeatingInner cylinder ((b)Outer cylinder ((c)Inlet ((d)Outlet . So in the first steps (predictor step), Equations (2)–(4) are marched advance in time by considering the diffusion terms implicitly to yield provisional estimates of the velocity field In the corrector step, a vorticity preserving and irrotational correction pressure field is used to correct the non-solenoidal provisional approximation for velocity field. The free velocity field of divergence at the latest time level, Subtracting Equation (5) from Equation (7), we haveThen, Equation (8) is reduced as,For making the velocity free from the divergence at new time level, take the divergence of Equation (10), hence,(i)at inflow and solid walls of the annulus: (ii)at out flow of the annulus: where n is normal to the boundary of the domain at any location.So, by solving the Poisson Equation (11) we obtained the correction of the pressure. As suggested by Saad and Schultz , Cheng aOnce by obtaining the pressure correction field, Equations (9) and (10) can be used for the inside the flow domain to correct the pressure and predicted velocity field.In current time the pressure field can be obtained as:The corrected velocity field will be recovered at the current time as,At the solid walls, for all the velocities no-slip and no penetration condition is specified explicitly. The ordinary momentum equation is used to update the pressure at the inlet boundary and solid walls. To obtain pressure at the outflow, the traction free condition proposed as fallows by Gresho  are emplThis condition has also been employed by Cheng and Armfield  and HasaThe 4th order central-difference scheme employing five grid points is expressed as:The 3rd order upwind scheme employing five grid points is expressed as:The fluctuation-pressure gradient terms are first-order derivatives which are discretized as second-order centrally accurate terms near the wall and fourth-order accurate terms inside the flow. They are respectively expressed as:For Neumann condition at the boundary, the expression for discrete derivative should be one sided differencing expression for better accuracy. These are as follows:Forward-difference scheme using three grid points is expressed as:Forward-difference scheme employing five grid points is expressed as:The parallel code has been developed which runs on distributed memory devices using MPI (Message Passing Interface). The straight-forward method of parallelization is to use the decomposition of the domain. In which the computation is equally split between the n processors. Using the MPI commands like MPI SEND, MPI RECV and MPI WAIT, explicit point-to-point communication is introduced. While in operations for data set at two different parts of the code, it is necessary communication between processors. When derivatives terms are calculated, each processor needs data for the interface neighboring horizontal planes so that each processor exchanges one dimensional data from the other processors. The schematic view of the domain decomposition employed is shown in Non-equidistant meshes are used for most near-wall for simulation as shown in Now for any physical quantity to be solved, which may be either U, W or The numerical scheme used in this analysis has been validated with the data achieved by Desrayaud et al.  and AminFirst, the temperature profile along the radial length in the middle of the annulus is shown for different selected times and at this position, the temperature overshoot can be observed in Other results like wall and lquid temperature were also generated to compare the non-dimensional wall variation and mean bulk fluid temperatures along the axial length at selected non-dimensional times are shown in The time measurement of radial velocity at mid-height of the annulus for various radial positions is indicated in Change in radial velocity along the radial length is presented in The transient development of radial velocity contours at the inlet as well as exit of heated cylindrical wall are shown in This also indicates that, for the same Rayleigh number, the width of the contours of radial velocity is decreasing over time. Yet the width of the contours of radial velocity increases as Rayleigh number increases. Near the place where the heating begins, the magnitude of radial velocity formed is negative and its amplitude falls over an axial length, which reach nearly zero value when flow becomes fully developed. Furthermore, positive radial velocity is established near the place where heating stops and its strength falls over the axial length up to the exit. The steady-state contour of radial velocity for different Rayleigh numbers at the beginning and end of the heating zone shown in The axial velocity variation over the radial length is displayed in So as to overcome the viscous force, resulting in an increase of the pressure defect. This phenomenon is due to the overshooting of temperature, as described earlier. It is shown that in the case of partial annulus heating, the pressure defect is initially positive near the annulus exit (which is the non-heating zone) and subsequently becomes negative.The phenomenon of transfer of heat has been characterized as for the average and local Nusselt number, that are estimated at the heated region of the inner wall. The local Nusselt number (Nu) for the inner heated cylinder is defined as:Thus, such variations in the Nusselt number show the development of boundary layer at the annulus entrance, while for the rest of the annulus length the flow is fully developed. The greater the Rayleigh number, increases the Nusselt number for both fully and partially heated cases. The percentage mean deviation in each case for n number of data points were evaluated using the following relation:The non-dimensional volume flow rate Similarly, mean bulk axial velocity is define as:h which will be at the same point in the absence of any motion that would result in the departure of the pressure field from the hydrostatic variation imposed due to gravity by Gebhart et al. [h), is the pressure change that arises through fluid motion. It is due to acceleration, viscous force and buoyancy force. The difference (p − ph) is called the ‘motion’ pressure field or ‘pressure defect’ by Mohanty and Dubey [. That is the actual static pressure p is decomposed into ph and pm, asHowever, the numerical value of the pressure defect gets calculated. The gradient of the difference between two pressures, the actual static pressure p at any location and the hydrostatic pressure pt et al. . With bund Dubey . That isThe experimental and Numerical values of the pressure defect thus obtained are shown in 3 to 6.61 × 104). The study results show the rapid increase in the temperature for any radial location due to conduction having a maxima. As the convection becomes significant, the axial flow is developed and a steady state is observed below the maxima. The flow becomes parallel and fully developed with this eventual steady state which is known as temperature overshoot. The above phenomenon was observed for all Raleigh number. In addition, the convective radial velocity is getting higher by increasing Rayleigh number resulting in a negative effect on the fully developed region and positive effect on the thermal entrance length. The time taken to achieve equilibrium decreased with Rayleigh number due to the positive effect of this number on the buoyancy force. The numerically determined Nusselt number at steady state for fully and partially heated cases comes out to be 3.09 to 3.58 and 3.03 to 3.57, respectively, for Rayleigh number, Ra is 4.4 × 103 to 4.4 × 104. At the beginning of heated wall, local heat transfer coefficient and Nusselt number decrease from very large values and then for the remaining length of annulus it gradually decreases and become constant. The variations in the Nusselt number along the annulus height represent the developing boundary layer at the entrance and fully developed flow in the remaining length. The average Nusselt number increases gradually with Rayleigh number and can be represented in terms of Rayleigh number by the following correlations.Numerical investigations have been conducted for different Rayleigh numbers (Ra = 4.4 × 10The fully developed region decreases as there is an increase in Rayleigh while the thermal entrance length increases. Due to the non-heating zones at inlet and outlet of the annulus, the axial velocity decreases as compared to when the annulus is fully heated. The difference in axial velocity between the fully heated and partially heated annulus increases with increase in Rayleigh number. As the fluid gets heated up, the buoyancy force increases upward in the annulus and the viscous force decreases simultaneously. This is why the Pressure defect first decreases from the inlet value at the entrance, reaches minima where the two forces become equal and then increases again to attain value zero at the annulus exit when the buoyancy force overcomes the viscous force. Finally, the mass flow rate has been shown to increase initially and attain maxima and then decrease to a constant value at the steady sate. For the same Rayleigh number, the mass flow rate decreases in case of partial heating as compared to those in the fully heated case.


Simultaneous Localisation And Mapping (SLAM) has long been recognised as a core problem to be solved within countless emerging mobile applications that require intelligent interaction or navigation in an environment. Classical solutions to the problem primarily aim at localisation and reconstruction of a geometric 3D model of the scene. More recently, the community increasingly investigates the development of Spatial Artificial Intelligence , an evolutionary paradigm pursuing a simultaneous recovery of object-level composition and semantic annotations of the recovered 3D model. Several interesting approaches have already been presented, producing object-level maps with both geometric and semantic properties rather than just accurate and robust localisation performance. As such, they require much broader ground truth information for validation purposes. We discuss the structure of the representations and optimisation problems involved in Spatial AI, and propose new synthetic datasets that, for the first time, include accurate ground truth information about the scene composition as well as individual object shapes and poses. We furthermore propose evaluation metrics for all aspects of such joint geometric-semantic representations and apply them to a new semantic SLAM framework. It is our hope that the introduction of these datasets and proper evaluation metrics will be instrumental in the evaluation of current and future Spatial AI systems and as such contribute substantially to the overall research progress on this important topic. Intelligent transportation: whether talking about autonomous vehicles on open roads or Automatic Guided Vehicles (AGVs) within campuses, office buildings, or factories, transportation is a time and safety-critical problem that would benefit strongly from either partial or full automation.Domestic robotics: demographic changes are increasingly impacting our availability to do household management or take care of elderly people. Service robots able to execute complex tasks such as moving and manipulating arbitrary objects could provide an answer to this problem.Intelligence augmentation: eye-wear such as the Microsoft Hololens is already pointing at the future form of highly portable, smart devices capable of assisting humans during the execution of everyday tasks. The potential advancement over current devices such as smartphones originates from the superior spatial awareness provided by onboard Simultaneous Localisation And Mapping (SLAM) and Artificial Intelligence (AI) capabilities.Research at the intersection of robotics and computer vision currently targets the development of a handful of important applications in which some form of artificial, intelligent device is employed to assist humans in the execution of everyday tasks. A few examples are given by:Common to these applications is their mobility and requirement of real-time operation and interaction with the real world, no matter if active or passive. The last major road-block towards the achievement of such functionality is believed to be the perception problem. More specifically, the perception problem asks for a real-time solution enabling an intelligent, embodied device to localise itself relative to its immediate environment as well as perceive the latter in the sense of generating a virtual scene representation that is useful towards the execution of a specific task.In robotics, this problem is traditionally called Simultaneous Localisation And Mapping (SLAM). A typical realisation of a SLAM system would use either one or several exteroceptive sensors  attached to a moving body  in combination with interoceptive sensors such as an Inertial Measurement Unit (IMU). The incoming sensor stream is processed in real-time to incrementally recover the relative displacement of the sensor as well as a geometric representation of the environment. While there have been many successful systems presented in the past, the generated model of the environment traditionally only covers physical boundaries in the form of low-level primitives such as a point-cloud, a mesh, or a binary occupancy grid. Such representations may be sufficient to solve the problems of path-planning and obstacle-free navigation for robots but are not yet amenable to the solution of complex tasks that require understanding the object-level composition of the environment as well as the semantic meaning of its elements.Over the past decade, the identification of scene components  and their semantic meaning has seen substantial progress through the rise of deep learning based image processing techniques. In an aim to increase the semantic, object-level understanding reflected by 3D environment representations, the SLAM community has therefore recently begun to include such algorithms for object detection and semantic segmentation in the front-end of SLAM systems and reflect the added information in an object-level partitioning and a semantic annotation of the generated 3D models. Real-time systems generating such joint geometric-semantic representations from image sequences represent an evolution of more traditional, visual SLAM into something referred to as Spatial AI .We provide a concise review of currently existing Spatial AI systems along with an exposition of their most important characteristics.We discuss the structure of current and possible future environment representations and its implications on the structure of their underlying optimisation problems.We introduce novel synthetic datasets for Spatial AI with all required ground-truth information including camera calibration, sensor trajectories, object-level scene composition, as well as all object attributes such as poses, classes, and shapes readily available.We propose a set of clear evaluation metrics that will analyse all aspects of the output of a Spatial AI system, including localisation accuracy, scene label distributions, the accuracy of the geometry and positioning of all scene elements, as well as the completeness and compactness of the entire scene representation.In recent years, we have seen a number of exciting advancements in the development of Spatial AI systems. However, the performance of the latter is thus far mostly evaluated in qualitative terms, and we currently lack a clear definition of quantitative performance measures and easy ways to compare relevant frameworks against each other. An automated benchmark as has been introduced for traditional visual-inertial SLAM frameworks appears as an eminent gap to be filled in order to warrant research progress on Spatial AI systems. Our contributions are:Our paper is organised as follows. In One of the first occurrences of the SLAM paradigm dates back to 1991, when Leonard and Durrant-Whyte  exploiteTwo landmark contributions on monocular visual SLAM date back to the same year and are given by Davison’s  ExtendedWhile the most established representation for real-time SLAM systems is a 3D point cloud, the community has explored various alternatives. Delaunoy and Pollefeys  use a trA commonality to all the above-mentioned algorithms is that they employ either explicit or implicit low-level primitives for representing the environment . Each element is constrained directly by the measurements, and resilience with respect to noise and missing data may be increased by enforcing local smoothness constraints. However, the mentioned approaches do not impose less localised constraints given by semantic or other higher-level information about certain segments of the environment. As a simple example, consider a situation in which we would know that an entire set of 3D points lies on a plane. We would prefer to model this part of the environment using a single instance of plane parameters rather than many 3D points. However, such strategy assumes the existence of a front-end measurement segmentation module able to detect meaningful parts of the environment , establish correspondences between such measurement and scene segments, or even understand certain geometric properties about them . Examples of such front-end modules are given by the Yolo9000 object detection framework , the MasA system that is able to then incorporate higher-level knowledge such as learned semantic priors into the 3D representation of a real-time mobile visual perception system is defined in Davison’s studies FutureMapping 1 & 2  as a SpaAs also outlined in Davison’s studies ,27, therOne obvious disadvantage of frameworks such as SLAM++  is that Approaches that finally achieve dense, optimisable environment representations by low-dimensional graphical models and implicit, class-specific shape representations are given by Dame et al.  and EngeSpatial AI systems that aim at a real-time understanding of both the geometry and the semantics of an environment generally contrast with more time consuming offline approaches that start from known camera poses and known low-level geometry only to infer an object-level representation of the scene. For example, Gupta et al.  and Li eClassical solutions to camera-based SLAM rely on least-squares optimisation objectives that consist of minimising many localised photometric or geometric residuals over many views. They depend on an explicitly parametrised, high-dimensional, and low-level representation of the environment. Examples of the latter are given by geometric primitives such as 3D point clouds, sets of lines parametrised in 3D, voxel-based occupancy maps or distance fields, or even surface representations in the form of a triangular mesh. The resulting optimisation problems often have a graphical representation in which nodes represent optimisation variables  and edges represent correspondences between 3D points and frame observations and thus locations in which the relative positioning between the related variables can be cross-validated against the actual measurements. As introduced by Dellaert and Kaess , such grWhile possibly giving us accurate information about the physical boundaries of an environment, such representations generally do not reveal anything about the composition or semantics of a scene. As the entire modelling process purely relies on the online measurements, the robustness of classical methods is furthermore easily compromised by unmodelled effects such as occlusions or violations of assumptions made on surface reflectance properties . In the following, we will discuss a series of forward-looking environment representations, their use within SLAM algorithms, as well as expected resulting benefits.It gives us more flexibility in choosing different but more appropriate parametrisations for the geometry of each segment.It gives us a reasonable partitioning according to human standards enabling us to assign a class or—more generally—semantic attributes to each segment.Our first step towards the proposal of a new environment representation employed in future spatial AI algorithms consists of introducing naturally occurring segmentations. For example, indoor man-made environments are naturally composed of a background structure filled with objects that have certain geometric and semantic properties. It makes sense to adopt this partitioning for the internal environment representation used within a SLAM algorithm for the following reasons:h).Two instances of a thin conical object with location and shape parameters. Due to rotational symmetry, the pose One set of 3D points A plane vector Using an algorithm to establish correspondences between views, we can finally define the graphical structure of this optimisation problem. A natural subdivision of the graph into multiple subgraphs, which is a requirement in larger-scale applications where the complete map can no longer be loaded into memory. The hierarchical, tree-based structure of the map permits loading submaps of different scales .An agreement between the tree structure of the map and the natural segmentations and hierarchical subdivisions used by humans, which simplifies man-machine interaction.Parent structures appearing as separate nodes in the optimisation graph and enabling the construction of higher-order residual errors.One further aspect that is still missing in the envisaged representations is the naturally occurring hierarchical nature of man-made structures. We again start from the example of indoor man-made environments, which have a hierarchical, multiscale subdivision into buildings, floors, and rooms cf. . EstabliTo illustrate the latter argument, we again picture a concrete scenario in which a floor cleaning robot navigates in an office building. Let us imagine that—after traversing the first room—it passes through a corridor and enters another room to continue its task. Each room has a rectangular footprint with vertical walls and is furnished with one table and four chairs, the chair shapes being the same within each room. The resulting graphical model that exploits the hierarchical structure of the environment is indicated in The envisaged environment representations aim at a radical change in how we represent and optimise environments in visual SLAM. Inspired by the recent success of deep learning in computer vision, the hierarchical object-level representations of the environment leverage artificial intelligence to identify the segmentation and composition of an environment. However, deep learning may not only be employed for solving front-end tasks such as the segmentation of images and the initialisation of a graphical model as outlined in Compact map representations: whether talking about a planar surface representing a wall or a more complex object shape, the object-level partitioning permits the choice of class-specific implicit low-dimensional representations. While choosing a simple three-vector to parametrise a plane, more complex objects could be modelled using the above-introduced, artificial intelligence based low-dimensional shape representations. The compact representations have obvious benefits in terms of memory efficiency, a critical concern in the further development of next-generation Spatial AI systems.Low-dimensional representations: by employing class-specific, we implicitly force optimised shapes to satisfy available priors about their geometry. In contrast to current approaches which use high-dimensional, semantically unaware representations that only employ low-level priors such as local smoothness, the (possibly learned) higher-level priors we may employ for certain objects of known classes are much more powerful and implicitly impose the correct amount of smoothness for each part of the object. It is conceivable that such techniques will have a much better ability to deal with measurement disturbances such as missing data or highly reflective surfaces.Current low-level representations enable the automated solution of low-level problems such as the efficient, collision-free point-to-point navigation in an environment but do not yet enable the execution of more complex instructions that would require an understanding of semantics and composition. Hierarchical hybrid graphical models would give machines a much more useful understanding of man-made environments and notably one that is presumably much closer to our own, human reasoning.The expected benefits of the generated representations therefore are:We now proceed to the introduction of our new benchmark for Spatial AI systems. After a review of existing datasets in the literature, we present our new synthetic datasets along with their main advantages, creation, and evaluation metrics.Numerous visual-sensor-based datasets have been publicly released for different purposes and made great contributions in the SLAM and the deep learning communities. Within the SLAM community, the TUM-RGBD  and the Predictions from 2D images (even if complemented by depth information) easily suffer from the challenge given by occlusions in the data, which is why the current research trend shifts the focus from an evaluation over 3D instead of only 2D data. Novel datasets containing ground truth information about the contained 3D objects along with their semantics, poses, and shapes are therefore in great demand. A solid starting point in this direction is given by Stanford’s 3D Scene Datasets  and Scanhttp://mpl.sist.shanghaitech.edu.cn/SSSBenchmark/SSS@MPL.html.Our novel benchmark datasets are motivated by this circumstance and inspired by the ICL-NUIM datasets , which iRendering engines model the imaging process according to some preset parameters and with a unified computer clock for each virtual sensor. This effectively saves the effort for intrinsic and extrinsic calibration of the multisensor system, including the tedious time stamp alignment between RGB, depth, and semantic images as well as the IMU readings.The convenience given by the software-based adjustment and replacement of objects and cameras vastly increases the efficiency of the entire dataset generation process.The diversity of the freely available, virtual 3D object models including the variability of their texture maps enables the automatic generation of a large number of highly variable datasets, which improves the generality of the subsequently developed Spatial AI, SLAM, and deep learning algorithms. If designed well, the dataset properties including the virtual multisensor setup, the sensor trajectory, the scene layout, and the objects’ composition, arrangement, and appearance can be steered at the hand of a simple input script.In recent times, the rapid development of Graphics Processing Unit (GPU) performances and computer graphics algorithms eases the efficient generation of highly realistic virtual datasets. The main advantages of synthetic datasets are:We use the Eevee renderer in Blender 2.8 to generate our datasets on a workstation with GTX 1050 Ti and i7-7700 CPU. We provide a complete and easy-to-use framework for the generation of novel datasets and five pregenerated datasets with certain characteristics for the development of future indoor Spatial AI systems. https://www.CC0textures.com/ and https://www.poliigon.com, and adjust reflection and transparency to guarantee the complexity and authenticity. Our primary output is stereo RGB sequences for research on stereo SLAM systems. Both cameras have a default framerate of 30 fps, a baseline of 0.12 m, a CCD sensor size of 35 mm, a configurable horizontal FOV of 72.7 degrees , an F-stop of F22, and SVGA resolution . The intrinsic parameters are summarised asTo get photo-realistic RGB data, we set up proper irradiation of sun and lamps, map objects with 4K high-quality textures and HDRI from We furthermore add noise to the rendered datasets to simulate the behaviour of real-world cameras. Normal DSLR cameras contain three types of noise: photon shot noise , which is why Blender transformation matrices Next, we discuss the definition of the camera trajectories. Realistic trajectories could be obtained by running a SLAM algorithm over real image streams captured by a handheld or a robot embedded camera and using them for the virtual camera in the synthetic scene. However, the sparse and uneven timestamps between keyframes gained from a SLAM tracker do not serve well for the generation of IMU data and limit the flexibility in customising the framerate or the actual trajectory of the virtual camera. We therefore synthesise smooth trajectories emulating handheld camera motion for all datasets. The camera in Blender follows a convention different from the one used in computer vision . Simulated IMU accelerations and angular velocities are finally obtained by numerical differentiation. Accelerations are derived usingThe effect of gravity is subsequently accounted for by subtracting To conclude, the angular velocities are again rotated into the respective camera frame.The ground truth data of the trajectories are recorded line-by-line in a separate file, where each line contains the pose variables in the order Furthermore, for IMU data is in the order https://www.turbosquid.com/). We reduce the face and vertex resolution of each object to minimise physical size without damaging the quality of the shapes. The transformation matrix  of each semantic object is stored along with its class label in an individual file. We also provide one .obj model for each added type of object and the empty room.Next, we discuss the definition of the environment. An example with individual objects and their arrangement is illustrated in https://github.com/CaoYuchen/SSS-dataset. The purpose is to supersede complex manual operations in Blender and merge our data processing methods with the Blender API and also provide realistic sensor noise processing for RGB and depth data. The scripts also allow users to simply tune input parameters for the generation of customised datasets. We expect it can be furthermore used in-the-loop inside a  Spatial AI system that operates by the analysis-by-synthesis paradigm. We open-source our scripts for the generation and noise addition of datasets under the link The proposed metrics for Spatial AI systems evaluate all aspects of the Hybrid Graphical Representations introduced in Average Trajectory Error (ATE): The Average Trajectory Error directly measures the differences between corresponding points of the ground truth and the calculated trajectories. The evaluation is preceded by a spatial alignment of these two trajectories using a Procrustes alignment step. The latter identifies a Euclidean or similarity transformation that aligns the two trajectories as good as possible under an L2-error criterion. The residual errors are due to drift or gross errors in the sensor motion estimation, which the ATE characterises by either the mean, median, or standard deviation of the distance between the estimated and the ground truth position:Relative Pose Error (RPE): The ATE is not very robust as a large number of absolute locations may easily be affected by only few gross estimation errors along the trajectory. The RPE in turn evaluates relative pose estimates between pairs of frames in the sequence. Particularly if the chosen pairs of frames are drawn from local windows, the RPE is a good measure of average local tracking accuracy:In order to evaluate the localisation accuracy of a Spatial AI system, we reuse two standard measures which have also been used by the TUM-RGBD benchmarking  frameworIn order to evaluate the overall capacity of a Spatial AI algorithm to parse a scene and gain a correct understanding of the composition and semantics of the individual objects, we propose to take the Intersection over Union (IoU) between the ground truth and the inferred label distributions. This measure identifies whether or not certain object classes have been identified, whether all of them have been identified, or whether some of them have a nonunique representation. Note that all class labels that are not present in the ground truth set of labels are simply being replaced by the label “other”. In addition to the overall IoU for the entire label distribution, the analysis is completed by individual IoUs for each class. An example distribution is indicated in For each object, we pick the nearest and the second nearest object judged by their object centers.We first ensure that the distance to the best is below a certain threshold.We then ensure that the ratio between the distance to the second best and the distance to the best is below a certain threshold.The most important part of a modern evaluation of a Spatial AI system consists of assessing the quality of each individual object inferred in the scene. However, before any of the inferred objects can be evaluated, we first need to establish correspondences between each inferred object and the best possible correspondence within the ground truth set of objects. These correspondences are established as follows:z-axis points upward, others where it points forward). More importantly, there may be a difference between the convention for this reference frame between the ground truth and the estimated shape representations.For each identified correspondence, we then evaluate the class, pose, shape, and completeness. The object class is trivial to evaluate as it is simply right or wrong. However, other criteria are more expensive to evaluate and require prior steps to transform and align the representation of the shapes. This is due to the fact that the shape and the pose of an object are discerned by the introduction of an object-specific reference frame, and the location and orientation of this reference frame may vary depending on the employed object representation (e.g., some CAD models may be registered in a frame where the First, we need to express the geometries in a common reference frame. We reuse the similarity transformation given by –If it is a point cloud already, we leave it unchanged.–For a polygon mesh, we simply employ the above-outlined strategy to ensure a sufficiently dense sampling of the surface.–For binary occupancy grids, we employ the marching cubes algorithm to transfer the representation into a mesh, and then employ the above-outlined technique to obtain a sufficiently dense point cloud.–For a signed distance field, we employ the Poisson surface interpolation method, by which we again obtain a mesh that is transformed as above.The next step consists of expressing geometries in comparable representations. We choose point-clouds as there exist many algorithms and techniques to deal with point clouds. For the ground truth data, we generate point-clouds by simply using the vertices of each CAD mesh. However, in order to ensure sufficiently dense surface sampling, larger polygons are recursively subdivided into smaller subtriangles until the surface of each triangle is below a certain threshold. For the reconstructed model, we also transfer the employed representation into a point cloud:Once both the ground truth and the estimated object of each correspondence are expressed as point clouds in the same global reference frame, we proceed to their alignment. We choose the GoICP algorithm  for theiOur shape evaluation procedure starts by transforming the representations into a common reference frame, making their representations comparable, and performing an alignment. The alignment is necessary as the magnitude of the aligning transformation can be used to express the object pose error, and the pose error should be compensated for before we are able to evaluate the shape error. The detailed steps are as follows:Pose: The accuracy of the object pose is simply evaluated by the magnitude of the translation and the angle of rotation of the aligning transformation identified by GoICP .Shape: The accuracy of the shape is assessed by first applying the aligning transformation identified by GoICP and then taking the mean and median of the Euclidean distances between each point in the estimated point cloud and their nearest neighbour in the ground truth point set. Further shape accuracy measured such as the standard deviation and the maximum and minimum of the point-to-point distances are given as well.Completeness: We conclude by assessing the model completeness of each object. This is done by—again—first applying the aligning transformation to the estimated point set. We then mark each point in the ground truth point set as either observed or not. A point is observed if it is within a certain threshold distance of one of the inferred points. The final completeness measure is simply given asHowever, this measure obviously depends on a good choice of the threshold distance, which is why the completeness measure is evaluated for multiple such distances. This leads to a curve that expresses the quality of the shape as a function of this radius. The final measure to evaluate the shape completeness or quality is given by taking the area under this curve.Having completed the alignment of each object’s representation, we may finally complete the evaluation of all other criteria:A toy example for the point-to-point distances as well as the ground truth point marking is illustrated in The final geometric element to assess is the structure of the background. We again start by applying As introduced in , one of We conclude our exposition by applying our proposed benchmark to two available, state-of-the-art Spatial AI systems. The first one is given by the Kimera pipeline , which pThe Kimera pipeline includes a high-performance tracking module that is able to work in either monocular, monocular-inertial, stereo, stereo-inertial, or RGBD modes. We use the stereo-inertial mode as it the most stable one. Kimera achieves highly accurate tracking for which even the ATE measures indicate very low drift errors. The ground truth and the estimated trajectories, as well as their deviations along the trajectory, are illustrated in While Kimera is able to estimate highly accurate trajectories, the geometry representation remains primarily a geometrical one. The pipeline relies on a module denoted the mesher to extract dense surface geometries and furthermore assigns a semantic class for each polygon from the originally image-based semantic annotations. However, the framework does not employ any prior shape knowledge, which therefore leads to generally incomplete and inferior object geometries. We therefore complement the Kimera tracker and the image-based semantic labels by an additional module taken from the Deep-SLAM++ pipeline .We run an object detector over the left images of each estimated stereo frame and use both depth and semantic information in the bounding box to perform a segmentation of each observed object.For each detected object, we find all neighbouring frames in which the object is observable, detect redundancies, and initialise unique object representations with a list of frames in which each object is observable.We use Pix3D  to prediWe refine the latent code of each object shape representation by minimising depth errors in all observing frames. This finds a compromise between measurement fidelity and the application of prior shape knowledge.To conclude, the individual object shape optimisations are alternated by object pose optimisations in which the entire object shape is aligned with their respective observations.In order to overcome the weakness of purely geometric pipelines such as SemanticFusion , Fusion+For further details on the approach, the reader is kindly referred to the work of Lan et al. .The most interesting part of the evaluation is finally given by evaluating the accuracy of the object poses and shapes. Note that the accuracy of the object poses is additionally influenced by the slight drift in the camera pose estimations, and it is hard to discern this influence from actual object pose inaccuracies. The results are summarised in Current Spatial AI systems vary strongly in their approach. While some frameworks start from more traditional SLAM solutions and infer semantics and object-level partitioning in a bottom-up scheme, others aim at an immediate understanding of the topological, object-level composition of the environment thus enabling the representation of each individual object using a tailored, class-specific representation of each object’s pose and shape. The strong variability of these approaches contrasts with a lack of benchmark datasets for which all information such as semantic class distributions, object-level composition, as well as object shapes are known. We have introduced realistic novel synthetic indoor datasets for which such information is readily available. Combined with the proposed evaluation criteria, we introduce a complete framework for benchmarking Spatial AI systems assessing semantic, geometric, and topological aspects of the estimation and complete our toolchain by flexible scripts to create novel datasets.We have combined state-of-the-art solutions and techniques from the communities of visual-inertial tracking, deep learning, and semantic SLAM to achieve a first genuine Spatial AI result in which scenes are represented as a composition of objects with poses and shapes. Our conclusion is that there is still substantial research to be done until both the quality and the computational efficiency of such frameworks will meet the demands of most real-world applications, and it is our hope that this work will be helpful in comparing and driving the future development of Spatial AI systems.


Flatness sensors are required for quality control of metal sheets obtained from steel coils by roller leveling and cutting systems. This article presents an innovative system for real-time robust surface estimation of flattened metal sheets composed of two line lasers and a conventional 2D camera. Laser plane triangulation is used for surface height retrieval along virtual surface fibers. The dual laser allows instantaneous robust and quick estimation of the fiber height derivatives. Hermite cubic interpolation along the fibers allows real-time surface estimation and high frequency noise removal. Noise sources are the vibrations induced in the sheet by its movements during the process and some mechanical events, such as cutting into separate pieces. The system is validated on synthetic surfaces that simulate the most critical noise sources and on real data obtained from the installation of the sensor in an actual steel mill. In the comparison with conventional filtering methods, we achieve at least a 41% of improvement in the accuracy of the surface reconstruction. Laser based optical sensors have been widely used in industrial environments for various applications like online programming, part measurement and quality control, and part identification and localization.The requirements for the surface quality of sheet metal products are continuously increasing. Flatness defects are a major problem in many industrial areas such as architectural panel works ,2 and thH and period L, its Steepness Index value is defined as i, is defined as:i and the reference fiber The flatness of a metal sheet is its levelness when the sheet is free of tension. The main institution providing the definitions of flatness and how to measure it for steel sheet products is the American Society for Testing and Materials (ASTM), allowing purchasers and providers to understand each other using the same characterizations of flatness and the means of the detection of flatness anomalies. The two main standardized measures of flatness are the Steepness Index , and theOptical sensors based on laser line triangulation are becoming the most widespread solutions to acquire a dense 3D mapping of metal surfaces . Using tThe contributions of this paper are the following: (1) the design and implementation in a real industrial operation of a surface flatness sensor that uses two parallel laser lines and an industrial high speed area scan camera providing height maps of the metal sheets moving below the sensor; and (2) a noise removal system based on Hermite polynomial interpolation that effectively removes high frequency noisy wave patterns induced in the sheet by mechanical manipulation. The flowchart in The filtering process is validated on synthetic and real industrial production data. This allows analyzing and filtering the information even when the surface flatness information is extremely hidden under the influence of noise sources whose amplitude exceeds several times the amplitude of the original signal. In particular, a noise source produced by a cutting station near the sensor location was removed successfully from the synthetic and real data.The structure of the paper is as follows: Early metal sheet flatness measurement techniques relied on manual gauging , demandiWith the advent of computer vision technologies, optical flatness sensors were introduced  to measuIn the computer vision field, there is a long tradition of approaches dealing with surface estimation using gradient data, which are often very noisy ,17,18,19We devised an optical measurement system comprised of two laser lines and a matrix camera that allows the computation of both the height and gradient information along each 1D fiber. Using a combination of both pieces of information, we developed a method capable of obtaining a precise surface reconstruction that is robust against all kinds of noise sources ,33. The In this section, we address some fundamental aspects of the implemented sensor, namely the sheet-of-light triangulation principle, optical design considerations, the use of peak image intensity detector techniques, and the calibration of the camera and laser line sensor.The sheet-of-light technique is based on the principle of triangulation , performng plane .The measurement principle of the sheet-of-light technique is illustrated for a single laser line triangulation setup in quations  and 3)  P=X,Y,Z Equation  correspoer plane .Sheet-of-light laser triangulation requires robust image segmentation of the laser line in the image captured by the camera. However, spurious reflections and speckle noise make this computation difficult.Lasers light can be focused in a narrow and bright line. Moreover, the coherent nature of laser light allows using narrow-bandwidth optical filters on the camera, removing image noise from ambient light. However, the downside of coherence is that it creates speckles in the imaging system. Speckles are interference effects of coherent light photons that travel slightly different distances from the source to the sensor, produced by surface roughness at the order of the wavelength of the incident coherent light. It gives a characteristic granular appearance when the surface is imaged under highly coherent light . SpeckleSpurious reflections are another big problem in the 3D surface scanning of shiny metallic surfaces. The reflected light from the targeted area of the surface may illuminate some other areas of the same surface that are detected by the segmentation of the images, causing fake measurements .To minimize speckle and spurious reflections, the proposed sensor features a blue laser emitter at wavelength The accuracy of a 3D reconstruction using laser linear illumination is significantly determined by the accuracy of the line segmentation in the image. Since the pattern of image intensity in the normal direction to the line has a Gaussian profile, finding the center of the line in the image corresponds to detecting the point of maximum intensity in the normal direction, also known as the laser peak, which can be detected by different algorithms using the intensity distribution along a column of the sensor image, e.g., finding the position of maximum intensity, finding thresholding points, or finding the center of gravity ,46,47. IThe purpose of sensor calibration is to identify the camera intrinsic parameters and the mapping between the laser plane and the camera. Since the sensor consists of a laser projector and a camera, camera modeling and calibration become an integral part of the sensor calibration procedure. The conventional approach consists of using first a standard camera calibration technique ,53 to esScanning large 3D objects, such as flattened rolled metal sheets, implies that the object is moving relative to the 3D scanner. Depth measurements computed from laser triangulation are synchronized with the motion of the metal sheet using an incremental encoder located after the roll leveler stage. This ensures a uniform data acquisition and minimizes the effect of the motion jitter.s is an arbitrary scale factor, The standard accuracy requirements in the steel processing industry are as follows: for the height, better than Define Given a set of calibration points and their corresponding image coordinates, the transformation matrix We estimated the intrinsic parameters of the camera before the installation of the camera in the sensor by a previous calibration process at the lab, achieving an average retroprojection error of 0.04 pixels. We carry out the correction of the lens radial distortion so the mapping from the image plane to the laser plane becomes a linear function. The mapping transformation For a set of corresponding points Equation , we get Equation .(11)A=−The proposed flatness sensor is comprised of two illuminating parallel linear laser sources perpendicular to the metal sheet translation axis separated by a distance x is computed as This dual laser line approach allows separating high frequency vertical vibrations from low frequency flatness defects by detecting variations between lasers lines , as showThe proposed optical flatness sensor was installed in a finishing process line just after an industrial roll leveler close to a cutting station, as shown in t. A series of curve sampling intervals The approximation of surface profiles by splines allows the detection of surface defects as regions of the surface that show a high distance relative to the approximating spline functions . Specifi0 and C1 .However, in order to use the piece-wise cubic Hermite interpolation method, several considerations must be taken into account. On the one hand, the surface fiber recovered from the sensor must have a signal-to-noise ratio (SNR) high enough to allow data interpolation. On the other hand, according to the Nyquist–Shannon sampling theorem , the graThere are two different sources of error in the laser-camera triangulation process. One is the error in the estimation of the homography between the laser planes, and the other is the sub-pixel laser detection errors described in Due to the fact that the homography is estimated using several points, the errors due to this estimation may overcome the errors induced by the laser line intensity peak detection in the image. The segmentation errors of the laser points projected in the image plane have a normal distribution, zero mean, and a standard deviation h the step of the central difference in Gradient estimation errors are due to numerical differentiation, truncation, and round-off errors . The errB and period Flatness calculation in rolled metal steel assumes that flatness defects can be approximated by sinusoidal waveforms, so that the maximum absolute derivative To inspect rolled products detecting flatness defects, it is necessary to scan the sheet as it moves along the processing line. The feeding rate of some lines reaches speeds up to n intervals, each one discretized into m samples of its height measurement To compute the noise-free reconstruction of one metal strip fiber In jth sample and its derivative at the ith interval of the fiber The matrix We apply the proposed Hermite polynomial filtering and reconstruction method to both simulated and real data in order to test its ability to remove high frequency noise due to mechanical manipulation of the metal sheet.We created a synthetic surface showing two of the most common defects in a roll leveler processing line, namely center buckles and wavy edges. The period and amplitude of these sinusoidal flatness defects are The first computational experiment is carried out over the synthetic surface described above without additional noise, in order to simulate an ideal production line and check the accuracy of our method. The surface filtering has been made with a sampling interval for the Hermite interpolation of In the second computational experiment, we introduced different kinds of noise to simulate the vibrations induced in the sheet metal as a result of the mechanical elements of the roll leveler line. Additionally, we introduced another noise source controlled by a single parameter that simulates the triangulation error of the 3D sensor. The noise sources that characterize the mechanical vibrations have an amplitude of In the third experiment, we added a localized damped sine wave along the longitudinal axis of the synthetic sheet with an initial amplitude of the envelope of These results show that the Hermite interpolation filtering method is very robust and that it enables an accurate and reliable surface estimation despite the presence of external noise sources, improving over other methods based on FIR techniques, e.g., Butterworth, Savitzky–Golay, etc. These results show that the flatness sensor and the proposed filtering method are both robust and accurate enough against different noises sources, in particular to localized high amplitude noises produced by a simulated cutting station placed after the rolling leveler and the devised 3D sensor.A moving-average filter featuring a sliding window of 33 samples in length.A third order low-pass Chebyshev Type II filter  with 33 A third order Savitzky–Golay FIR smoothing filter  with a fA third order Butterworth IIR digital filter  with a cFor a quantitative evaluation of the improvement achieved by our proposal, we implemented the following approaches for surface denoising:T seconds depending on the cutting length program, and the conveyor belt produces overall high frequency waves. The resulting interference pattern is a rather complex spatial form of noise, hindering the detection of flatness defects.During the experimental data acquisition, we used sheet samples from two kinds of steel S235JR steel coils . Measuring a test specimen of a metal sheet out of the roll leveling machine with a CMM has some inherent bias, because the sheet is released from tensile and traction stresses, while our optical sensor measures the strip under stress. Therefore, the flatness measures yielded by a CMM cannot be fairly compared with those of an online sensor measuring the actual production.The measuring device and the filtering method proposed in the manuscript solves flatness data reconstruction in a real-time framework. In fact, it takes an average of The proposed sensor relies on the motion of the steel sheet for the measurement. Doing the measurements off-line for comparison with contact sensor measurements carried out off-line would require building up a complete setup where the sensor is displaced over the cut steel plate. We carried out some comparisons of the measurements done by the sensor oonline with off-line measurements by shape measurement rolls. The differences were on the order of 0.25 mm, within standard industry resolution requirements. It must be taken into account that the off-line contact sensor measurements do not suffer from the transport tension, cutting, and other vibration sources.Finally, this method can be used to concatenate as many processed sheets as needed. In order to ensure the In this paper, we propose a hardware-software system that carries out in real time the estimation of the flatness of a metal strip moving below the unrolling mill that flattens a roll of steel cutting it into sheets of predefined lengths. Oonline quality inspection of the unrolled metal strip before cutting is achieved by this system. The standard flatness anomalies to be detected are wavy edges, center buckles, and bow defects, which are low frequency variations in the height of the metal surface. The mechanical manipulation of the metal, such as the jerking due to the strip pulling and tugging, and the cutting process, induces high frequency variations in surface height that are successfully removed by our surface reconstruction software system, which is based on a Hermite interpolation approach.The nonlinear combination of the different noise sources on this particular process means that filtering the sensor signals with the desired precision cannot be achieved by conventional linear filtering techniques. We validated the filtering approach on synthetic and real industrial operation data, showing significant improvement over linear conventional approaches. In the comparison with Savitzky–Golay, Chebyshev, and Butterworth filters applied to the reconstruction of the surface after noise addition, we achieved more than a 41% improvement. Furthermore, we were able to assess the accuracy of the sensor against off-line contact sensor measurements, reporting an average error of 0.25 mm within standard industry accuracy requirements.The proposed sensor and surface reconstruction system can pave the way for online closed-loop control systems, low cost real-time flatness quality inspection, and high efficiency and quality manufacture of rolled steel products. More extensive and detailed analysis of the impact of gradient estimation numerical errors should be carried out.


Introduction: In 2014, a newly designed, case-based seminar was successfully implemented in the subjects of health systems, health economics and public health care (GGG). The seminar “The Lonely Patient” is based on a real patient case and deals with the German health care system from the perspective of a patient. In order to create more space for discussion and exchange among students, the seminar was redesigned on the basis of the Inverted Classroom Method (ICM). Project description: Due to the COVID-19 pandemic, new, purely digital teaching formats had to be developed quickly in the sense of Emergency Remote Teaching. Therefore, the Inverted Classroom concept of the seminar was transformed into an online ICM. In order to promote active learning based on the ICAP model , the online face-to-face part was designed as a synchronous interactive learner-centered course using the gamified audience response system Kahoot! Results: Evaluation results to date and feedback rounds with students indicate that the online ICM-version of the seminar leads to at least as good evaluation results as the previous face-to-face course. In particular, the students positively emphasize the use of Kahoot! as an activating digital medium. Discussion: Through the use of the ICM and the gamified audience response system Kahoot!, students could be activated in meaningful ways. The resulting discussions about the patient case and teaching content of the quiz questions in the synchronous online course could be implemented just as well as in the classroom-based course of previous semesters. Conclusion: The application of the online ICM, along with the consideration of the ICAP Model, has led to the successful implementation of a digital course within the context of the increased difficulty surrounding the emergency remote teaching. Additionally, students’ learning success has remained at a similar level as during traditional classroom-based courses of previous semesters. In 2014, a newly designed case-based seminar was successfully implemented in the subjects of health systems, health economics and public health care (GGG) , [4], 55, 6], , 77. ICM haIn the winter semester of 2018/19, a 15-minute, ICM-based online self-learning phase was developed for the seminar “The Lonely Patient”. During this phase, students are first given an overview of the patient's history and are expected to acquire factual knowledge independently by means of defined learning goals. For this purpose, the existing written teaching and learning materials  were transformed into two to three minute instructional videos. In the face-to-face session, the topics of the videos are discussed in greater depth with the help of flipchart summaries, keynote speeches by the lecturers, group discussions, and a quiz. In accordance with the ICAP model , this should lead to increased interaction between students and lecturers . AccordiDue to the COVID-19 pandemic, new, purely digital teaching formats had to be developed quickly in the sense of emergency remote teaching . The relThe questionnaire was developed on the basis of the ICAP model and comprises 28 evaluation questions that, among other things, record the students’ own learning process. The questionnaire was used at the end of the course via QR code or link .Based on the initial evaluation results and feedback rounds with students, it can be cautiously concluded that the use of the online Inverted Classroom Model does not lead to worse results in subjectively perceived learning success and interactivity during classroom instruction compared to previous traditional classroom-based semesters. Thus, in the overall assessment, the seminar is rated slightly better on average than the previous semester. Brief rounds of feedback directly following the seminar confirmed this result (“What did you particularly like about the seminar?”): Students gave positive feedback regarding the course concept and especially emphasized the use of Kahoot! Students also indicated that they preferred the synchronous course format over asynchronous digital teaching. It should also be noted that the response rate of the online evaluation was higher compared to the traditional classroom-based courses of previous semesters or compared to asynchronous teaching.By using the online Inverted Classroom Model and by means of the planned interactive components (Kahoot!), the interaction with the student could be kept at the same high level as in a classroom format of the same course. The disadvantages of pure online teaching without activating elements could thus be reduced. By using the online Inverted Classroom method, digital teaching could be continued successfully during the COVID-19 summer semester. It also seems that the disadvantages of the pure online teaching, like the reduced or missing interaction with the students could be compensated. According to the ICAP model, the learning success of students could be guaranteed or at least not worsened to a large extent. In the winter semester 2020/21, this course will probably be held in digital form again. It is planned to implement the course concept with an ARS that contains reflective elements compared to Kahoot! Further evaluation will show whether these first positive results can be confirmed. The present concept can theoretically be used in a slightly modified version in the future. The authors declare that they have no competing interests.


Geometric model fitting is a fundamental issue in computer vision, and the fitting accuracy is affected by outliers. In order to eliminate the impact of the outliers, the inlier threshold or scale estimator is usually adopted. However, a single inlier threshold cannot satisfy multiple models in the data, and scale estimators with a certain noise distribution model work poorly in geometric model fitting. It can be observed that the residuals of outliers are big for all true models in the data, which makes the consensus of the outliers. Based on this observation, we propose a preference analysis method based on residual histograms to study the outlier consensus for outlier detection in this paper. We have found that the outlier consensus makes the outliers gather away from the inliers on the designed residual histogram preference space, which is quite convenient to separate outliers from inliers through linkage clustering. After the outliers are detected and removed, a linkage clustering with permutation preference is introduced to segment the inliers. In addition, in order to make the linkage clustering process stable and robust, an alternative sampling and clustering framework is proposed in both the outlier detection and inlier segmentation processes. The experimental results also show that the outlier detection scheme based on residual histogram preference can detect most of the outliers in the data sets, and the fitting results are better than most of the state-of-the-art methods in geometric multi-model fitting.  Geometric model fitting attracts a lot of attention in the computer vision field, for example, in image registration and stitching ,2,3, strGeometric transformations are needed when two-view or multi-view images are considered, but these images often contain noise and outliers, thus a robust geometric transformation is needed to deal with these cases. It is well known that random sample consensus (RANSAC)  and its So far, the most popular strategy for fitting multiple geometric models is preference analysis ,30,31,32k-th order scale estimator (KOSE) [k-th order scale estimator (ALKS) [k in KOSE automatically; iterative k-th ordered scale estimator (IKOSE) [k. However, in the geometric model fitting problem, the noise distribution is always extremely complicated, and more or less deviation is caused by using Gaussian distribution, thus the scale estimators perform poorly in geometric model fitting.Because the fitting accuracy is seriously affected by the outliers, a series of scale estimation techniques ,42,43,44r (KOSE)  assumes r (KOSE)  and the r (ALKS)  and modir (ALKS)  try to fIn geometric model fitting, generally the residuals of outliers to all the true models are big, which presents the notion of a consensus of the outliers that will be highlighted when the proportion of good hypotheses is big enough. However, few works have been proposed studying the outlier consensus to exclude the outliers for robust fitting. Preference analysis methods like J-linkage and T-linkage eliminate the impact of outliers by using an inlier threshold. Permutation preference doesn’t need an inlier threshold, but is very sensitive to outliers. The conceptual space in J-linkage is the binarization of the residuals by inlier threshold, which extremely compresses the residual information and decreases the differences of points belonging to the different models and thus leads to under-segmentation, while T-linkage proposes a continuous relaxation of the binary to construct the preference set for linkage clustering, which keeps the continuous residual information and the retains the differences of inliers belonging to the same model, thus it results in over-segmentation.The residual histograms for each data point have peaks corresponding to the true models because hypotheses generated with random sampling tend to cluster around the true model  We propose a new preference analysis method for geometric model fitting and compare the proposed residual histogram preference with some current preferences like the binarized conceptual preference, the soft conceptual preference and the permutation preference. (2) Based on the residual histogram preference, we propose an outlier detection strategy for geometric model fitting. (3) We introduce a framework for alternative sampling and clustering in both the outlier detection and inlier segmentation for improving the stability and accuracy.The rest of this paper is organized as follows: In In multi-model fitting, outliers greatly affect the fitting accuracy, and inliers polluted by a few outliers can make the fitting parameters far from the true result. As illustrated in Most of existing geometric multi-model fitting methods depend on an inlier threshold to exclude the outliers, which make the fitting accuracy greatly reliant on the inlier threshold. Because the scales of different models vary from each other, a single inlier threshold cannot satisfy all the models to exclude the outliers, thus the single inlier threshold often works poorly in geometric multi-model fitting.Some scale estimators  consider the distribution of inliers as a Gaussian model and try to find the inlier scale of each model in the data automatically, which successfully makes the fitting process more adaptive and robust. However, the distribution of the inliers is extremely complicated in geometric multi-model fitting, and the Gaussian model cannot fit the inlier distribution well. Therefore, Gaussian-based scale estimators work poorly in geometric multi-model fitting with non-Gaussian inlier distribution conditions.Because outliers are not assigned to any of the true models, the residuals of outliers are bigger to all the true models, which suggests using a consensus of the outliers. When the proportion of good hypotheses is big enough after the sampling process, the residuals of outliers will share more common items to have big values. However, the consensus of the outliers is not obviously presented in the residual space. Because the scales of different models are different, the extent of the “big” values in different models is also different. On the other hand, the pseudo-outliers will tend to be much closer to the outliers in residual space, which also have many common items with big values. Currently the preference of the data points, designed from the residuals like conceptual space in J-linkage or soft conceptual space in T-linkage, can highlight the consensus of the outliers from the inliers or pseudo-outliers with the help of an inlier threshold. Permutation preference decreases the impact of the inlier scale difference by sorting the residuals and getting a serial number as the preference for data points, which also obviously represents the consensus of the outliers. Similarly, a residual histogram is quite a good way to reduce the influence of inlier scale differences by quantizing the residuals according to each model. Moreover, residual histograms can compress the residual according to the histogram, and when used to present the consensus of the outliers, the difference of outliers will be greatly compressed, thus it will make the outliers gather away from the inliers. We present the detailed preference analysis on outlier consensus for residual histogram preference below.Preference analysis-based method was first used in J-linkage , in whicThe conceptual space in J-linkage can be considered as Tanimoto distance as follows:Unlike the binarized conceptual space in J-linkage, T-linkage  introduck list rather than a full rankings list, i.e., The permutation preferences are also extracted from the hypotheses residuals ,47. WhenWhen comparing two points  dF ,47 betweR, and takes the quantized value The residual histograms for each data point have quite significant features whose peaks correspond to the true models because hypotheses generated with random sampling tend to cluster around the true model, which was used by residual histogram analysis (RHA)  to find ith row of We will introduce the selection principle on the quantization length In the binarized conceptual preference in J-linkage, the residuals bigger than the inlier threshold are set to 0 and the smaller are set to 1, the residuals of outliers are bigger for all the true models, and when the proportion of the good hypotheses is high enough, the binarized conceptual preference of the outliers will have more common items to be 0 with a proper inlier threshold. When projecting to the binarized conceptual preference with the Jaccard distance, the outliers will gather together.Similar to the binarized conceptual preference in J-linkage, the soft conceptual preference in T-linkage cuts off the residuals bigger than the inlier threshold to 0, and the soft conceptual preference of the outliers will have more common items to be 0 with a proper inlier threshold, thus the outliers gather together in the soft conceptual preference space.k items of the outliers’ permutation preferences will share quite few common list with the inliers’, while outliers’ permutation preferences will more likely to have more common items with each other when using a proper permutation length, thus the outliers gather away from the inliers in permutation preference space.As for permutation preference, the residuals of outliers are bigger for all the true models and the residuals of the inliers are small for the hypotheses close to true models, the top-Because the residuals of outliers will be bigger for all the models in the data, the bigger residuals will more likely tend to be close to As shown in As shown in Because the outliers greatly affect the fitting accuracy, the proposed fitting method should detect the outliers first, and then conduct inlier segmentation on the data without outliers. Hence our proposed method consists of two main parts: outlier detection and inlier segmentation. As presented in As presented in The proposed outlier detection method starts with a sampling process to generate enough hypotheses, and then calculates the residuals to construct the preferences for the data points. Similar to the sampling method in J-linkage , which gAfter the hypothesis generation, residuals for each hypothesis will then be calculated and get the residual matrix. The residual histogram preference for the data points can be obtained by method presented in By the linkage clustering, most of the outliers in the data points will be detected, but because of the random sampling process, the detected outliers will change slightly every time. In order to decrease the change and further improve the outlier detection, an alternative sampling and clustering framework is proposed to conduct sampling the hypotheses and then carry out linkage clustering alternately, which make it possible to sampling the hypotheses in inlier clusters to further increase the proportion of the good hypotheses, and in turn improve the preference for better linkage clustering.Algorithm 1 Outlier Detection 1: Divide  outlier cluster 2: Conduct random sampling on each sub-region and generate hypotheses 3: Calculate the residual histogram preference matrix 4: Calculate the distance matrix 5: Conduct linkage clustering with distance and obtain clusters 6: Calculate the outlier index  outlier index as the outlier cluster 7: If As a result, the whole process for outlier detection is summarized here. Initially, we perform the random sampling within the sub-regions, which are divided by the Euclidean space distance of the data points, and the initial outlier cluster is detected by linkage clustering with the residual histogram preference. Then, in turn, the inlier clusters obtained by the linkage clustering are regarded as sub-regions for the random sampling to generate hypotheses and, in the same way, to perform linkage clustering with the residual histogram preference to detect the outlier cluster. Thus, the sampling and clustering processes are iteratively conducted until the outlier cluster is unchanged. The details of the algorithm are listed in Algorithm 1.The outlier detection process usually makes it possible to detect most of the outliers in the data set, and after removing the outliers from the data set, the inlier segmentation is carried out on the remaining data set. The inlier segmentation process is conducted by using linkage clustering integrated with the alternative sampling and clustering framework, similar to the outlier detection process. However, during the linkage clustering, permutation preference ,30,31,47Algorithm 2 Inlier Segmentation1: Divide  inlier cluster 2: Conduct random sampling on each sub-region and generate hypotheses 3: Calculate the permutation preference matrix 4: Calculate the Spearman footrule distance matrix  Equation (5);5: Conduct linkage clustering with distance 6: If The inlier segmentation follows the alternative sampling and clustering framework, as presented in In this section, we describe the experiments undertaken in geometric multi-model fitting, including multi-homography estimation and multi-fundamental matrix estimation, which are fundamental issues in image stitching ,49 and vAs a matter of fact, most of the multi-model fitting methods do not work well when dealing with a data set such as “johnsonb”, which contains quite a lot of models with highly varied inlier scales. This results in the sampling being uneven and the sampling rate on models with a smaller inlier scale being too low for the preferences to present a consensus on the inliers. Although the sub-region random sampling method can overcome the uneven sampling problem, the improvement is quite limited without the consideration of the distribution of the model inliers, which is quite difficult to obtain without sampling and preference analysis or a priori knowledge. The alternative sampling and clustering framework make it possible to sample the inlier clusters, but the performance is greatly dependent on the initial inlier segmentation results.The misclassification results of the state-of-the-art methods and the proposed method for multi-fundamental estimation are presented in Overall, the proposed method works quite well in the multi-fundamental estimation experiment. For the “dinobooks” data set, the proportion of outliers (43%) is very high, and eight points need to be randomly selected every time to generate a fundamental matrix hypothesis, which results in the proportion of good hypotheses being very low and seriously impacts the performance of the preferences.The outlier detection results are presented in During the outlier detection process, the key parameters are quantization level OID change with quantization level ion data a–c and θion data d–f. The OID are very close for the three multi-homography matrix data sets, which are also similar for the three multi-fundamental matrix data sets. The biggest OIDs for the multi-homography matrix data sets are near to the side where OIDs arise along the x-axis, which means that the value of k-nearest inlier to point k-th nearest inlier distance for each outlier. In our experiments, the value of k was set to 20.It is clear that the change trends of In this paper, a robust geometric multi-model fitting method has been proposed. Firstly, the outliers in the data set are detected by means of linkage clustering with residual histogram preferences and then the inlier segmentation process is carried out on the data with outliers removed through linkage clustering, but permutation preferences are used in this stage, which shows superior performance for data sets with outliers removed. In order to obtain a stable clustering result after the linkage clustering, an alternative sampling and clustering framework is proposed in both the outlier detection and inlier segmentation processes. The experimental results also show that the outlier detection method based on residual histogram preference can successfully detect most of the outliers in the data. Overall, the proposed model fitting method can separate the inliers for different models and outperforms the state-of-the-art methods in geometric multi-model fitting.In the future, the accuracy of proposed solution can be further improved by combining some state-of-the-art methods such as those described in ,53,54,55


Stability is a prerequisite for legged robots to execute tasks and traverse rough terrains. To guarantee the stability of quadruped locomotion and improve the terrain adaptability of quadruped robots, a stability-guaranteed and high terrain adaptability static gait for quadruped robots is addressed. Firstly, three chosen stability-guaranteed static gaits: intermittent gait 1&2 and coordinated gait are investigated. In addition, then the static gait: intermittent gait 1, which is with the biggest stability margin, is chosen to do a further research about quadruped robots walking on rough terrains. Secondly, a position/force based impedance control is employed to achieve a compliant behavior of quadruped robots on rough terrains. Thirdly, an exploratory gait planning method on uneven terrains with touch sensing and an attitude-position adjustment strategy with terrain estimation are proposed to improve the terrain adaptability of quadruped robots. Finally, the proposed methods are validated by simulations. Nowadays, legged robots have been a research hotspot and gained various attention ,2,3,4. QQuadruped robot gait consists of static gait  and dynaDynamic quadruped robots are hard to control because they are nonlinear, unstable, time-variant, and multi-input multi-output (MIMO) systems. They also interact with environment and behave switch dynamics, especially during support exchange . ResearcAs for static gait, Nie et al. employed some optimization methods to do the motion planning algorithm of static walking gait for a quadruped robot . Focchi Stability is a prerequisite for legged robots to execute tasks and traverse rough terrains. The legged robot is expected to be an environment-accessible platform because of its environmental adaptability. With the consideration of stability, walking speed, and energy consumption or Cost of Transport (CoT) , the quaMeanwhile, compliance control is an effective way to reduce body oscillations and maintain the stability of robot. In order to improve the environmental compliance of a quadruped robot on rough terrain, impedance control ,37 has bIn this paper, the overall aim is to look for a static gait of quadruped robots with the biggest stability margin and its control strategy for high terrain adaptability. Firstly, the static gait of quadruped robots with the biggest stability margin is chosen by comparing three classical stability-guaranteed static gaits: intermittent gait 1&2 and coordinated gait. Secondly, a position/force based impedance control is employed to achieve the compliant behavior of quadruped robots on rough terrain, which will maintain the stability of robot well. Thirdly, an exploratory gait planning method on uneven terrains with touch sensing and an attitude-position adjustment strategy with terrain estimation are investigated to further improve the terrain adaptability of quadruped robots. This research is meaningful for quadruped robots, especially when considering the carriage of heavy goods .The rest of this paper is organized as follows: The simulation model setup by WEBOTS software is shown in The DH (Denavit–Hartenberg) method ,45,46 caThe Lagrange method can be applied to derive robot dynamics as follows:The workspace of foot end-effector of robot can be defined as the collection of all positions that can be reached in space by the foot end-effector under the hip joint coordinate system {H}. It is mainly determined by the motion range of joint angle. The workspace of the right fore leg is shown in It can be seen that the workspace is an irregular three-dimensional space, and it is difficult to get its analytic expression. However, gait planning must be carried out in the workspace, and the point outside the workspace is where the robot can not reach. In order to simplify the analysis, a simpler method based on the restriction of workspace boundary is adopted since the workspace boundary of the robot is determined by the angle range of each joint. It is decided that, once the difference between the planned joint angle and the corresponding upper or lower limit is less than a certain threshold Zero moment point (ZMP) is a popular stability criterion for legged robots . For simm is the mass of robot, The formula for calculating CoP isRemark 1.In static walking, the projections of robot base origin (RBO), CoM, CoP, and ZMP could be equivalent to the same point on the ground by setting proper counterweights, even though there is a little bit difference among them. As such, through planning and controlling the projection of RBO inside the support polygon area formed by the end-effectors of supporting legs, the static walking stability can be guaranteed. If the projection of RBO falls in the inner center of support triangle, the static walking has the highest stability.Static gait of the quadruped robot can be divided into two classical gaits: intermittent gait and coordinated gait. Intermittent gait refers to the separation of leg swinging and CoM movement; coordinated gait refers to the simultaneous movement of swinging leg and CoM.Referring to the static walking of quadruped animals and based on the stability margin, gait naturalness and movement space requirements, the basic static walking rules are given as: (1) Hind leg moves firstly between legs on the left or right side in the same row; (2) Backward leg moves firstly between legs on the fore or hind side in the same column; (3) Each movement of CoM follows the walking forward direction as far as possible. Under these rules, three typical stability-guaranteed static gaits investigated in this paper are given: (1) In intermittent gait 1, as shown in S and the time of each action (take a step or move CoM once) is T, the gait cycles of three static gaits are S is the same in a gait cycle. Thus, the walking speeds of three typical static gaits are The three typical stability-guaranteed static gaits are shown as rotate clockwise for one step cycle in In summary, the increase of walking speed will inevitably lead to the decrease, even the disappearance of stability margin. In order to seek a greater increase in walking speed, robot is usually forced to switch from static gait to dynamic gait. In addition, the stability margin is replaced by dynamic stability in dynamic balance control, which is not discussed in this paper. Taking stability as a prerequisite, the following research focuses on intermittent gait 1. In High terrain adaptability of quadruped robot can be achieved by two ways: adaptive gait planning on uneven terrain and attitude-position adjustment.Reasonable gait planning will reduce the contact impact when foot end-effectors of robot touch down on the ground. Gait planning includes support phase and swing phase. In this paper, the cycloid trajectory planning method is used to plan support phase gait asThe swing phase gait is planned by the polynomial interpolation trajectory method , and it o impact , there aBecause of the oscillations of robot trunk, the contact impact of foot end-effector still will occur on the even terrain with the gait planning under principle of zero impact. Even through impedance control can reduce the impact efficiently, the impact is not small enough to ensure the stable walking on uneven terrain, such as stairs, slopes, gravel and high platforms, etc. The walking stability is always at the first place for legged robots. Therefore, intermittent gait 1 is adopted to do the gait planning on complex terrains. In addition, an exploratory gait planning on uneven terrain with touch sensing is proposed as shown in Exploratory gait is divided into four parts: support phase and swing phase 1,2,&3. The cycloid trajectory planning method Equation  is emploH. The swing phase 2 can explore high platforms as shown in x axis and the force is greater than the given threshold, it is determined that there is a high platform in the front. At the moment, the swing leg retracts and returns to its original position in the horizontal direction and raises upward by H  Touching down sensing is added, and it can deal with complex terrain calmly and enhance mobility; (2) Gait planning is relatively simple and parameters To handle the problem of friendly environmental interaction, a position/force based active compliance controller in our previous work , as showGenerally, a second-order linear system (spring-damping-inertia system) is adopted as the desired impedance ,43(11)ZThe detailed model and stability analysis can be found in .The environmental perception sensors, such as lidar and camera, are not reliable and do not work well in complex environments. They also may be broken sometimes. Therefore, the attitude–position adjustment strategy without the environmental perception system should be addressed .S step length in the support phase. On the premise of walking stability of a robot, it is necessary to adjust the position and pose of robot according to the terrain for reserving more workspace for the next leg stride. The environment perception system composed of lidar and a camera is a direct way to detect the front terrain, but it can not accurately detect the terrain under the foot and body in real time. There is a big deviation in the estimation by combining moving speed of robot and environmental sensing data. Sometimes, the robot is without an environment perception system or its environment perception system is broken. Therefore, it is necessary to estimate the current terrain by combining robot attitude sensor and foot position.After the swing phase, the CoM moves to continue completing O is the CoM position, Equation  is the eThe terrain can be estimated by the direction angle of normal vector of support surface in the next leg stride of robot. The formula for calculating the normal vector Then, its direction angle can be written asEquation  is the eDefine rmula isPiB=TαTβTAssume the attitude angles of a desired coordinate system {Equation  without T is the time of attitude adjustment. When Getting the foot end-effectors Equation , it yielKeeping the yaw angle unchanged  solver is chosen and the fixed-step size is ic gaits , walkingic gaits  and walkic gaits  are anals is the walking distance, Walking speed, CoT , and staStability margin could be utilized to measure the static walking stability . The staThe simulations of quadruped robot walking with three static gaits: intermittent gait 1 (bottom), intermittent gait 2 (middle), and coordinated gait (upper) are shown in x-axis, and its walking distance is the farthest in the same 50 s time, but its stability margin is sacrificed. In the direction of the y-axis, in order to obtain a good stability margin, the left and right swing of three static gaits are almost the same. The fluctuation of coordinated gait at a zero crossing point is due to the stability margin having been close to zero. Note that the pendulum-like motion of robot upper body is to improve the walking stability of quadruped robot [z-axis, the intermittent gait 1 has more oscillates because it makes more CoM shifts, including backward shifts. The CoM trajectory and attitude of three static gaits are compared in ed robot ,30. In tz-axis follows the leg shank. Comparing the contact force of The foot end-effector trajectory and contact force of three static gaits are compared in The energy consumptions of three static gaits are compared in The simulation of the quadruped robot walking on a high platform with 0.2 m height is shown in Especially due to the exploratory gait, the robot can sense the high platform, and then raise its CoM and foot to climb up the high platform. The four foot end-effector trajectories of quadruped robot walking on high platform are shown in The simulation of quadruped robot walking on uneven terrain is shown in Further research is implemented to calculate the CoT of quadruped robot walking on different terrains by using intermittent gait 1, as shown in Stability margin is taken in the first place. In addition, three static gaits are then chosen. The three static gaits can eliminate unnecessary motions, consume less energy, reduce gait cycle, and are more bionic. Among the 24 walk gait combinations in , the thrIn order to further analyze the relationship among walking speed, CoT, and stability of static gait, three static gaits with four different speeds are compared by changing the gait and gait cycle, as shown in It can be seen that the increase of walking speed among three static gaits directly leads to the decrease of stability margin, but the CoT is reduced as well. In the same static gait, the increase of walking speed can also effectively reduce CoT, which results in a certain degree of the attenuation effect on the stability margin. Therefore, the factors that need to be considered in static gait selection contain terrain complexity, stability, walking speed, and CoT.Three stability-guaranteed static gaits: intermittent gait 1&2 and coordinated gait, are investigated, and a novel exploratory gait planning on uneven terrain with touch sensing is proposed to improve the terrain adaptability.A position/force based active compliance controller is introduced into static gaits to improve compliant behavior and energy efficiency of quadruped robots.A novel attitude-position adjustment strategy with terrain estimation is proposed to improve the walking stability.Proposed methods are validated by simulations, and the relationships among four factors for static gait selection: terrain complexity, stability, walking speed, and CoT are discussed.This paper proposed a stability-guaranteed and high terrain adaptability static gait for quadruped robots. The main contributions can be concluded as follows:Future works focus on multi-body dynamics, anti-disturbance, AI adaptivity, etc.