The recent development of the Internet of Things (IoT) in healthcare and indoor air quality monitoring expands the market for miniaturized gas sensors. Metal oxide gas sensors based on microhotplates fabricated with micro-electro-mechanical system (MEMS) technology dominate the market due to their balance in performance and cost. Integrating sensors with signal conditioning circuits on a single chip can significantly reduce the noise and package size. However, the fabrication process of MEMS sensors must be compatible with the complementary metal oxide semiconductor (CMOS) circuits, which imposes restrictions on the materials and design. In this paper, the sensing mechanism, design and operation of these sensors are reviewed, with focuses on the approaches towards performance improvement and CMOS compatibility. Gas sensors have been widely applied in various fields, such as agriculture , automotThe gas sensors can be categorized into four types according to Janata : (1) masThe mass sensors measure the frequency shift of the acoustic wave when foreign particles or molecules are absorbed onto the surface of an oscillating structure. Gravimetric sensors, such as quartz crystal microbalances (QCM), surface acoustic wave (SAW) sensors, surface transverse wave (STW) sensors and shear-horizontal acoustic plate mode (SH-APM) sensors, use quartz crystals with different cuts as the oscillating structure. They are extremely sensitive, with resolutions down to nanogram level. The frequency shift was observed to have an approximately linear relation with the concentration of the gas by Öztürk et al.; additionally, the sensitivity of their QCM sensor can reach a few hertz per ppm . HoweverThermal sensors detect the gas by the temperature-induced resistance change when combustion reactions take place in catalytic materials. The sensors are low cost and reliable in dusty environments; hence, they are widely used in industrial applications. However, catalytic based reactions restrict the applications of these sensors to flammable gases such as hydrogen, methane, propane and butane. Moreover, catalytic poisoning may deteriorate the long-term stability of the sensors.Optical sensors recognize the gas by absorption of light with certain frequencies. These frequencies are closely related to the oscillation of molecules; the spectrum suggests that the bonds exist in the molecules. Optical sensors are highly selective and have much higher cost than the other three types because of the complex interferometer mechanisms and the light source. The interferometer and light source also impose challenges on their miniaturization and system integration. Therefore, the high cost and power consumption limit their on-site applications, and most optical sensors still remain as laboratory apparatus .Electrochemical sensors are based on the change in electrical properties when target gas diffuses and reacts with the sensing material. Electrochemical sensors are relatively easy to be miniaturized and manufactured with microfabrication technologies. Successful miniaturizations include field effect transistor (FET) gas sensors and metal oxide semiconductor (MOX). FET sensors are based on the change of the threshold voltage when gas molecules reach the gate material. It is fully compatible with CMOS processes and has a sub-milliwatt level power consumption. However, metal gate materials, such as palladium and platinum, are only sensitive to hydrogen. Recently, many works have been done on the gate design to improve the performance of the FET sensors by applying nanomaterials, such as carbon nanotubes  and grapCompanies such as ams AG , Bosch SThe demand for continuous miniaturization and power consumption reduction drive the research of MOX gas sensors towards direct integration of the MEMS sensing structure with the integrated circuits for signal conditioning circuits . This inDifferent aspects of the CMOS-MEMS integration technology have been reviewed by many researchers. For example, Qu focused on the fabrication technologies , Li et aThe reaction between the target gas and the metal oxide film is composed of the reaction of the target gas with metal oxide molecules, as well as the preabsorbed oxygen species. The sensing process consists of three steps: diffusion of the target gas molecules onto the surface of the metal oxide, adsorption of the gas molecules into the metal oxide and reaction between the gas and metal oxide. In the case of n-type semiconductor metal oxides, the sensing mechanism can be explained by the double Schottky barrier model . The int2, competes with the oxygen species for the electrons and further increases the resistance of the film.The released electrons are trapped in the conduction band, forming depletion layers on the surface of the grains. The double Schottky barrier and upward bending of the conduction band at the grain boundaries lead to an increase in the resistance of metal oxide film. When a reducing agent, such as hydrogen, carbon monoxide or ethanol, is brought into contact with the metal oxide film, the resistance drops because of the neutralization of the oxygen species and mitigation of potential barriers. Conversely, an oxidizing agent, such as NOa and Rg are the resistance of the thin film before and after exposure to the target gas, respectively. Sometimes, the net change in resistance replaces the numerator.The response of the sensor is defined by the ratio of the resistance of the sensing material before and after exposure to the target gas (Equations (4) and (5)) , where RThe reaction rate between the chemical species and metal oxide film governs the response of the gas sensor. The α and β species dominate the temperature below and above 300 °C, respectively, while the γ type starts to appear from 550 °C onwards [g and m are constants and Pg is the partial pressure of the gas, which is proportional to the concentration of the gas. The value of m depends on the concentration of oxygen species when the metal oxide is exposed to air and is specific to each pair of target gas and metal oxide. The value of m can be approximated by theoretical approaches. Recently, Hua et al. proposed reduced receptor and transducer functions to calculate the value of m [3 gas sensor to NO2 [The response has to be closely related to the centration of gas for effective detection. At a fixed temperature, the sensor response increases with respect to the concentration of the gas due to an increase in carrier concentration and mobility, which can empirically be represented by the power law (Equation (6)) :S = Ag∙Plue of m . Figure r to NO2 . The eff2 gas sensor in response to reducing gas would decrease sharply when the grain size fell under two times of the width of the depletion layer, since electron transport was suppressed due to fully depleted grain. Rothschild and Komen showed from a numerical simulation that the sensitivity is proportional to the inverse of the grain size [Additionally, the sensor response has a strong dependency on the grain size of the metal oxide crystal. Xu et al.  found thain size . Yamazoeain size . The substrate.The bottom silicon oxide/nitride layer, which insulates the heating element from the substrate.The heating element layer and an adhesion layer, if necessary. Sometimes, temperature sensing elements, such as resistance temperature detectors (RTD), are also fabricated on this layer to monitor the temperature of the microhotplate and provide a reference in the temperature control loop.The top silicon nitride/oxide layer, which serves as the insulation between the heater and the sensing material or electrode and passivation layer to prevent catalytic interaction between the target gas and the heater material .The electrode layer.The sensing material layer.The device structures of a traditional metal oxide gas sensor and a MEMS-based metal oxide gas sensor are shown in The performance of a sensor is usually evaluated by its 4S, i.e., sensitivity, speed, stability and selectivity. In gas sensor applications, the sensitivity is expressed by the response and detection range. The speed is evaluated by the response and recovery time, which is defined as the time taken to achieve 90% of the change in the resistance. In addition to the 4S, the power consumption of the sensor needs to be minimized due to the voltage and current limit for miniaturized sensors. Extensive studies have been conducted to improve the design of each component for the optimized performance; these will be reviewed accordingly in this section.The function of the microhotplate is to raise the temperature of the sensitive material to its optimum operating temperature and it is the main source of power consumption in the gas sensor. Bhattacharyya  and SpruSilicon is the mainstream substrate material for micro gas sensors due to its capability with IC fabrication processes and the potential for CMOS integration as a monolithic system for in-situ sensing and processing. Non-silicon materials, such as ceramics , borosilicate glass , siliconcond, Aconv and Arad are the areas of the surfaces where each heat transfer process takes place; s is the surface temperature; Ta is the ambient temperature; and Tb is the temperature of the surrounding material, such as the substrate or the package. From Equations (7)–(9), the power consumption can only be reduced by reducing the operating temperature or the area of heat loss. Reduction in the operating temperature can be achieved by depositing metal oxide films with nanostructures, whose response can be compensated by the surface area to volume ratio. Nanostructures and their synthesis method will be discussed in The majority of the power consumed by the microhotplate is converted into thermal energy and dissipated into the surroundings. Therefore, it is essential to suppress the heat loss from the various heat transfer processes, i.e., the conduction from the sensing area to the substrate, the conduction from sensing area to the air, the convection from the sensing area to air and the radiation to the environment. The amount of heat transfer can be obtained from the Equations (7)–(9) :(7)Qcond−1 K−1 compared to 145 Wm−1 K−1 for silicon). Conduction and convection are the main heat loss mechanisms. Although some studies treated conduction as the primary mechanism ,45,46, iAt the expense of reduced power consumption, the mechanical structure of the microhotplate becomes more fragile due to less supporting materials; thus, the configuration must be designed according to the applications. Closed membranes are preferred for commercialized products or on-site applications since the sensors need to withstand vibrations. In research on the other hand, suspended membranes are more popular, since it is able to push the power consumption performance to a limit. In addition, its low thermal inertia contributes to fast response and recovery time and enables reliable temperature modulation. The suspended membrane can either be released from the front side a or the Both of the areas in Equations (8) and (9) refer to contact area between the microhotplate and air. This area is usually predetermined by the overall size of the device; hence, not much improvement can be made during the design. The convection heat loss to the air can be obtained by fluid simulations . This isA Finite Element Method (FEM) electro-thermal simulation is commonly implemented during the microhotplate design process for optimization of the temperature distribution and power consumption. A large variety of commercially available software such as ANSYS ,54,55,56The temperature profile of the optimized geometry has to be validated by the measurements of an infrared radiation (IR) camera or IR pyrometer. The relationship between the power consumption and the temperature of the hotplate can be obtained by the thermal image together with the measured voltage and current.xNy) or oxide/nitride/oxide (ONO) layer [The microhotplate is a sandwiched structure formed by the insulation layer, the heater layer and the passivation layer to support the sensing material. Concerns in high-temperature stability determine the material and processes of the sandwiched layers. Residual stress and mismatch in thermal expansion coefficient induce a vertical displacement of the microhotplate at high temperatures and may lead to delamination . Large dO) layer ,72,73,74O) layer . A reduc0 is the baseline temperature, R0 is the resistance at the baseline temperature, α is the temperature coefficient of resistance T is the operating temperature and R is the resistance at the operating temperature. The linear behavior assisted the measurement and control of temperature and simplified the signal processing steps. The temperature measurement was either performed by the heater itself or a thermometer on the same plane.The heating element material has to be stable at high temperature and is preferred to have a linear relationship (Equation (10)) between its resistance and temperature within the operation range: 2 and Al2O3 show much better adhesion performance, their applications are restricted by the feasibility of integration into the CMOS process due to their high-temperature deposition conditions. In addition to single layer adhesion films, a Cr/CrN/Pt/CrN/Cr multilayer approach was demonstrated by Chang and Hsihe [Chemical and mechanical stability at high temperatures were the main concerns for heater material selection. Platinum heaters are the most extensively studied material due to its high thermal conductivity and stability up to 600 °C, which is above the optimum operating temperature of most metal oxides. A layer of titanium or tantalum with tens of nanometers thickness is usually sandwiched between the platinum and insulation layer for better adhesion. For operations above 500 °C, titanium is not recommended, because it will diffuse into the platinum layer and form precipitates on the grain boundaries , whereasnd Hsihe , which sDoped polysilicon is another widely adopted heater material with linear resistance-temperature relations. It is a fully CMOS-compatible material and the adhesion problem no long exists. However, polysilicon is only suitable for operating temperatures below 500 °C, beyond which the recrystallization of polysilicon will cause drift in resistance . SpecialFor microhotplates operating above the stability point of platinum and polysilicon, molybdenum ,71 and tBoth electro- and thermo-migration of the material could cause the degradation of the heater . The forTemperature uniformity can be improved by attaching a layer of material with high thermal conductivity to the microhotplate. The high thermal conductivity material can either be a silicon island  left aftAlternatively, optimization of heater geometry with simulation and validation experiments has a significant impact on the temperature uniformity across the microhotplate. Meander, double spiral and drive-wheel are the main geometries reported . MeanderExcept for the sandwiched structure of the microheater, insulation and electrode layers, the heater and electrode can be fabricated on the same plane provided the same material is used a,b. ThisBesides heater electrode deposited on a planar surface, a microhotplate with three-dimensional heater deposited on a three-dimensional V-shape groove was developed for a catalytic type gas sensor . The heaThe electrode transmits the resistance of the metal oxide film to the signal conditioning circuit during operation: its material is required to have high electrical conductivity, high-temperature stability, good adhesion to the passivation layer and low contact resistance . In addi3, SnO2, ZnO, TiO2, V2O5) and p-type  metal oxide semiconductors have been investigated extensively as the sensing materials. However, the n-types takes up more than 90% of the work [Eranna has reviewed various types of bulk metal oxide materials and their gas sensing applications . Both n-the work , since tthe work . Table 4a/Rg).As the gas sensing reactions occur through gas adsorption, charge transfer and desorption on the surface of the sensing materials, the sensing response is highly dependent on the amount of available surface active sites. In this regard, zero-dimensional nanoparticles with abundant active sites, due to increased specific surface area, are highly desirable for promoting improved gas adsorption and prompting more target gas molecules to participate in the oxidation and reduction reactions. Uniform distribution of nanoparticles with minimum agglomeration is essential to ensure good gas-sensing reactions for this type of material. One-dimensional (1D) nanostructured materials, such as nanowire ,90,97, nThe concentration of noble metal has to be carefully controlled to avoid coagulation. A high percentage of doping may suppress the functionality of the sensor, since agglomeration may decrease the total area of the catalyst surface. Adding catalytic layer results in loss of sensitivity over a long period due to fouling or coking of catalytic metal when exposed to certain gas or vapors. Additionally, the stability of the sensor is affected by the degradation of the dopant metal. The high-temperature degradation of metal oxide materials is not limited by electromigration, but rather by outdiffusion and evaporation of dopants from bulk materials .3/Cu2O nanoneedles directly on a MEMS hotplate by a single step aerosol-assisted CVD process [The metal oxide sensing layer is deposited via a post-processing process after the fabrication of the CMOS circuits and the MEMS microhotplate. Guo et al. have given a comprehensive review of the synthesis methods of metal oxide nanostructures . The sen process . The met process .Thermally grown/deposition of the insulation layer.Lift-off/sputtering, patterning and etching of the heater material.Deposition of the passivation layer; patterning and etching for electrical contact.Lift-off/sputtering, patterning and etching of electrode material.Patterning the oxide/nitride layer to define the geometry of the membrane and bridge.Backside/frontside etching to release the suspended membrane.Deposition of the metal oxide layer.Monolithic gas sensors were fabricated by a CMOS-first approach. The microhotplate materials are deposited and released after the CMOS circuits have been deposited into the wafer. The typical fabrication processes of a suspended membrane microhotplate with a metal heating element are listed as follows and shown in Both dry etching and wet etching have been used in the membrane step. Tetramethylammonium hydroxide (TMAH) or potassium hydroxide (KOH) can be used as the wet etchant, with the former preferred due to full compatibility with CMOS processes. Dry etching processes, such as deep reactive ion etching (DRIE), is, in general, more preferred because the vertical sidewall makes the footprint of the sensor smaller. Despite the traditional layer-by-layer deposition approach, the silicon on insulator (SOI) CMOS technique was adopted to achieve simple fabrication, the high operating temperature of MOSFETs (400~600 °C), effective isolation and reduction in leakage current ,88,109. 2/3% Pd CO sensor [The performance of the sensors listed in O sensor . EnhanceO sensor . They atAlthough most microhotplates mentioned above are fabricated with CMOS compatible processes, few of them has actual CMOS circuits integrated for temperature control and signal conditioning. The main challenge for CMOS integration is that the operating temperature of the metal oxides  is usually higher than the maximum operating temperature of IC (<300 °C). Therefore, thermal isolation is crucial for monolithic gas sensor design. Suspended membrane geometry is preferred because of their excellent heat isolation performance. Less than 3 °C temperature difference between the circuit and ambient can be achieved by this geometry when the hotplate works at 400 °C .2C) serial protocol is commonly adopted for external communication in both commercialized sensors and the research works because it can reduce the number of pins required. The signal conditioning circuit consists of the temperature control circuit, the resistance readout circuit and the serial bus interface for external communication. In the temperature control circuit, the heater is driven by the MOSFET switches  Figure a with thIt can be observed from The excellent performance of the MEMS metal oxide semiconductor gas sensor makes it dominant in industrial applications and research projects. As the main structure of a gas sensor to operate the sensing material at an elevated temperature, the microhotplate is fabricated into a membrane or bridge for the sake of thermal isolation to achieve low power consumption. Doped polysilicon and tungsten are recommended as the heater material for operating temperatures below 500 °C and 700 °C, respectively, because of their stability in the temperature range and excellent compatibility with the CMOS process. The temperature uniformity can be improved by optimizing the heater geometry through electro-thermal simulations and validation experiments. Besides the power consumption and stability, other performance parameters such as sensitivity, response/recovery time and selectivity are determined by the sensing material deposited with a post processing approach. Recent research in nanostructured materials shows their enhanced gas sensing performance due to the increase in surface area to volume ratio. CMOS-compatible nanomaterial deposition techniques are still being explored.Monolithic CMOS-MEMS integration has advantages in size, noise level and power consumption compared to the multi-chip integration. However, temperature constraint of the CMOS circuit must always be kept in mind when designing the MEMS process. The CMOS circuits enable precise control of the on/off state of the microhotplates, which is suitable for sensor arrays with different sensing materials on each unit. In addition, the monolithic integration of the MEMS sensor and CMOS circuit make it possible to develop systems in a package (SiP) or system on a chip (SoC) gas sensing in the future. With a microcontroller unit encapsulated in the same package as the sensor, temperature modulation and signal processing become more efficient because of the reduction in noise during signal transmission.


Visual inertial odometers (VIOs) have received increasing attention in the area of indoor positioning due to the universality and convenience of the camera. However, the visual observation of VIO is more susceptible to the environment, and the error of observation affects the final positioning accuracy. To address this issue, we analyzed the causes of visual observation error that occur under different scenarios and their impact on positioning accuracy. We propose a new method of using the short-time reliability of pedestrian dead reckoning (PDR) to aid in visual integrity monitoring and to reduce positioning error. The proposed method selects optimized positioning by automatically switching between outputs from VIO and PDR. Experiments were carried out to test and evaluate the proposed PDR-assisted visual integrity monitoring. The sensor suite of experiments consisted of a stereo camera and an inertial measurement unit (IMU). Results were analyzed in detailed and indicated that the proposed system performs better for indoor positioning within an environment that contains low illumination, little background texture information, or few moving objects. In the modern world, people are becoming increasingly dependent on location services. The need to provide accurate indoor location services is becoming more and more urgent ,2,3,4,5.Vision-based indoor positioning technology can be divided into visual odometry (VO)  and visuVision-based positioning systems are very susceptible to environmental influences. The ability to extract stable and accurate visual observations is a key factor affecting positioning accuracy. When visual observations (image features) are rare or unevenly distributed, the positioning error of a vision-based positioning system can be very large and can even lead to system collapse. Alexander  proposedPedestrian dead reckoning (PDR) systems ,29, whicWe analyzed the error source and divided it into four error situations when the vision-based positioning system had a large positioning error under special indoor environments that had fewer textures, dynamic obstacles or low lightings.We proposed autonomous integrity monitoring of a visual observation-based pedestrian dead reckoning system. According to the characteristic of short-term reliability of PDR, the proposed PDR-assisted visual integrity monitoring system switches states between VIO (or VO) and PDR automatically to provide more accurate positions in an indoor environment.In this paper, we analyzed the error source and divided it into four error situations when the vision-based positioning system had a large positioning error under special environments. We proposed autonomous integrity monitoring of a visual observation-based pedestrian dead reckoning system. According to the characteristic of short-term reliability of PDR , PDR canVisual inertial odometers generally consist of two parts, namely, front-end and back-end. The front-end mainly deals with a sensor’s observations such as feature extraction, feature tracking, feature screening, IMU pre-integration processing, and integration of images with IMU data. The back-end mainly performs estimations of position on the abstracted data produced by the front-end by minimizing the residual which is caused by observations through a filter or an optimization scheme. The structural block diagram of the system is shown in C} for a global frame of reference {G}. Since a stereo camera consists of two cameras, the camera frames are represented by {G} to frame {IMU}, G}, The goal of the back-end is to estimate the 3D pose of the camera frame {k-th measurement of the camera contains a series of feature points that are observed by the k-th camera pose  , Speed Uht pairs .(7)[δX^When the number of feature points is sufficient, Illumination changes often occur in indoor environments, and we used the Lambertian model as the lighting model.It can be seen in the observation equation of the image that the presence of noise causes positional errors in the feature points in the image. The position error of the feature points will affect the state estimation of the camera when calculating the re-projection error. To better represent the role of the geometric relationship between image feature points and camera poses, our line used a simple two-dimensional example to describe the geometric relationship. As shown in We describe the quality of the position estimate based on the camera state’s Jacobian matrix of feature points z directions is represented by The amount of change in the position error in the All moving objects, such as pedestrians or vehicles, will affect the positioning result during positioning. When the feature points are concentrated on the moving object, the relative movement of the feature points results in a larger calculated camera movement. This situation can be expressed in the world coordinates of the feature points as having an additional motion shift, Let us analyze the residuals Although PDR has a problem of cumulative error, the error over a short time is very small. The rotation matrix of the IMU relative to the world coordinate system can be constructed by the three-axis gyroscope. After the three-axis acceleration is rotated, the relative position information Now assume that there are two sampling points In this paper, we propose an autonomous PDR-assisted visual integrity monitoring approach to improve positioning accuracy. According to the characteristic of short-term reliability of PDR, the proposed PDR-assisted visual integrity monitoring system switches states between VIO (or VO) and PDR automatically to provide a more accurate position in an indoor environment. The specific switching situation is shown in Hypothesis deviation obeys Gaussian distribution It is a normal distribution of multidimensional standards. It can be thought of as the sum of the squares of two independent random variables subject to the standard normal distribution which obeys the chi-square distribution of three degrees of freedom. The probability distribution (cumulative distribution function) is Our indoor positioning system is based on the Multi-State constrained Kalman Filter (MSCKF) positioning algorithm and PDR. It is very important to switch states between MSCKF and PDR automatically to provide an accurate positioning result for our indoor positioning system. The problem that visual observations cause on MSCKF can be attributed to the fact that visual observations are not updated or are updated incorrectly. So, the update frequency of visual observations and the estimation of the gyroscope bias will generate a large abnormal fluctuation when the abnormality is detected by PDR-assisted visual integrity monitoring . We used the update frequency of visual observation The sensor suite we used is shown in The following experiments were designed to evaluate the above four causes of errors identified in the previous session.x-axis and the y-axis.As shown in When the lighting is different from the left and the right cameras, the average gray value of the images acquired by the left and right cameras was different, and the matching rate was low. We used the distribution of feature points as variables and compared the trajectory with the original feature distribution as shown in Pedestrians walked in front of the camera, and the contrast tracking is plotted in This experiment was carried out in a large office building with a length of 100 m and a height of 20 m which lasted for 20 min and spanned three floors. The abnormalities of visual observations led to the update frequency of visual observation, and the estimation of the gyroscope bias generated a large abnormal fluctuation. According to the update frequency of visual observation and the estimation of the gyroscope bias, we divided the path into three parts to introduce the effect of PDR-assistance based on visual observations as shown in The length of walking distance of “Section A” was approximately 340 m. The specific action track was to walk straight in the corridor, then go downstairs to the next floor and walk three times in the hall. It can be seen from The length of walking distance of “Section B” was approximately 214 m which went upstairs to the rooftop area and involved two laps of the rooftop space. As shown in The length of walking distance of “Section C” was approximately 166 m, and the route was looped two times in an empty room which mostly contained turns. Due to the existence of a large number of white wall scenes, the visual observation of the MSCKF was relatively poor in quantity and quality. The filter estimated the deviation of a wrong gyroscope, resulting in a deviation of the overall trajectory direction. With the short-term reliability of the PDR, the direction deviation of the positioning can be reduced. It can be seen from To describe the positioning results more clearly, we labeled 32 landmarks and recorded the location information of those landmarks. Then we calculated the positioning errors of MSCKF, PDR, and our positioning system based on the experimental data. As shown in At this part, we performed a real-time positioning experiment, such as IPIN  (The IntTo better display the experimental results, the positioning trajectory of each floor is shown in To solve the problem of vision-based positioning systems being very susceptible to environmental influences, we analyzed the error source of visual observations when vision-based positioning systems had a large positioning error under special indoor environments having fewer textures, dynamic obstacles or dim lighting. We divided the error sources of visual measurement into four error situations and performed detailed analysis and explanation. The first part of the experiment assessed the scenarios’ impact on the positioning results to display intuitively the effect of feature observation. To address this issue, we proposed autonomous integrity monitoring of visual observation based on a pedestrian dead reckoning system. Through the error analysis of PDR, it was found that the error of PDR in a short time was small and bounded. According to the characteristic of short-term reliability of PDR, the proposed PDR-assisted visual integrity monitoring switches states between MSCKF and PDR automatically to provide a more accurate position in an indoor environment. The second part of the experiment tested and evaluated the proposed PDR-assisted visual integrity monitoring. In conclusion, we proved that our positioning system can effectively provide more reliable and accurate positioning results. Future research should consider the potential effects of visual observation more carefully. Also, future investigations are necessary to improve the accuracy of the step size of the PDR to improve the positioning accuracy of the system.


As a cutting-edge research topic in computer vision and graphics for decades, human skeleton extraction from single-depth camera remains challenging due to possibly occurring occlusions of different body parts, huge appearance variations, and sensor noise. In this paper, we propose to incorporate human skeleton length conservation and symmetry priors as well as temporal constraints to enhance the consistency and continuity for the estimated skeleton of a moving human body. Given an initial estimation of the skeleton joint positions provided per frame by the Kinect SDK or Nuitrack SDK, which do not follow the aforementioned priors and can prone to errors, our framework improves the accuracy of these pose estimates based on the length and symmetry constraints. In addition, our method is device-independent and can be integrated into skeleton extraction SDKs for refinement, allowing the detection of outliers within the initial joint location estimates and predicting new joint location estimates following the temporal observations. The experimental results demonstrate the effectiveness and robustness of our approach in several cases. Due to its high relevance for a variety of applications including surveillance, smart environment, gaming, human-robot interaction, or medical rehabilitation, the anticipation of human behavior has gained widespread attentions in recent years. The fundamental prerequisition to this task is the extraction of reliable skeleton regarding human poses that form the basic element within the variety of gestures and can then be analyzed in the context of the respective applications.However, accurate skeleton extraction is a highly challenging task due to the significant variation encountered in appearance, articulation, as well as the possibly occurring strong occlusions of several body parts and noise in the sensor data. Parameters such as height or bone length exhibit significant variations between individuals and, therefore, cannot be provided in advance but have to be estimated. Especially in marker-less motion capture, this becomes particularly challenging due to the highly dimensional representation of human poses conventionally followed in terms of joint models (joints with their positions and relations to other joints) as well as the large number of underlying degrees of freedom. So far, several approaches have been proposed and successfully applied in commercial entertainment applications such as Microsoft’s Kinect  or IntelAs shown in The same skeletal constraints as well as temporal characteristics of joint movements are used to detect erroneous estimates by the standard SDKs for depth cameras, as for example, occurring for occlusions of body parts that cannot be compensated by the optimization framework. Erroneous joint location estimates are counteracted by predictions from the observed motion data, i.e., we propagate the reliably extracted information over the temporal domain in order to handle occlusions of body parts possibly occurring in some of the video frames. To counteract the noise in the joint location hypotheses, we use the combination of forward and backward predictions to obtain robust initial joint position estimates, however, at the cost of accepting a delay of three frames. Nevertheless, we favor the accuracy of the estimated skeleton over the true live characteristics. We demonstrate the benefit of this approach on several examples.We present a novel approach for accurate human skeleton extraction from depth images where the initial per-frame pose estimate delivered by the motion estimation SDKs is refined based on additional symmetry constraints between left and right body parts as well as the constraint for constant lengths of skeletal parts.Our framework allows for the detection of errors in the initial joint location estimates and compensates for them based on the joints’ motion characteristics. For this purpose, we use a combination of forward and backward predictions which provide more robust predictions than naive interpolation between data from neighboring frames or simple forward prediction.In summary, the key contributions of our work include the following:Human pose estimation has been extensively studied in literature, and there are several comprehensive surveys on the respective developments ,3,4.The most common way for pose representations for human motion analysis is by using skeletons. A skeleton is a tree system of rigid bodies connected by bone joints. Uzun et al. developed a system adapting 3-D models built by an IClone to a human skeleton generated by Kinect . An L1 mNot only does emerging consumer hardware such as Microsoft’s Kinect v2.0 provide the means for low-cost RGB-D measurements but also the various applications in the context of games involve the estimation of human motion. In more detail, the Kinect SDK follows the developments discussed in several publications ,9,10 andHowever, the absence of kinematic constraints such as constant bone lengths, collision constraints , joint aFurther improvements have been achieved by additionally enforcing kinematic constraints , using aFurthermore, the refinement of MoCap data has been addressed based on a robust matrix completion approach that takes the low-rank structure and temporal smoothness of motion data into account and is solved using the augmented Lagrange multiplier method . Other aAnother work directly addresses the inference of human poses solely from RGB information e.g., ,27). The. The27])Our approach directly follows the avenue of human pose estimation from RGB-D images captured from a single view. We additionally exploit skeletal constraints such as left/right symmetry besides the bone length constancy as well as temporal motion constraints to refine per-frame pose predictions ,9,10. NoBefore discussing the steps involved in our pipeline, we provide a discussion of the underlying pose representation and the constraints derived from the skeleton which we use to refine the pose estimates. In the following section, we describe the skeleton pose representation and the proposed constraints.N individual joints, where each joint has certain attached attributes including its position and its tracking state. In comparison to the Kinect SDK [We represent human poses in terms of a skeletal graph ture see . Our skenect SDK , we discWe exploit constraints that are inherent to the underlying definition of the pose representation in terms of a skeleton. In particular, we exploit the symmetry of the left and right parts of the skeleton including the length of upper and lower arms, the length of upper and lower legs, as well as the distances from the shoulders to the neck that have the same length respectively see a. FurtheAs illustrated in Algorithm 1 A proposed refinement of initial human pose estimatesInput: initial pose estimate with initial joint positions s {ji,k} Output: refined pose estimate1:Bone Lengths Initialization . Furthermore, we use the state of being halfTracked when only one of the joints is tracked. The value of T is set to The latter We analyze the tracking state of the respective bone, i.e., whether it is marked as untracked by the Kinect SDK. If the involved joints are not tracked according to the Kinect SDK, we mark the respective joints as unreliable.i and j for the current frame t obtained from the initial per-frame pose estimates delivered by the Kinect SDK [i and j for the current frame as unreliable.We analyze whether the deviation of bone lengths is too large. After the initialization phase, we have approximate bone length references that allow for the detection of severely incorrect skeleton pose hypotheses provided by the initial per-frame skeleton estimation. In more detail, we compare the length nect SDK  to the cWe analyze whether the joint position is too far away from the predicted position  to consiThe joint position estimation provided by the initialization is susceptible to errors due to the huge variations in appearance, possibly occurring occlusions of individual body parts or sensor noise. To detect unreliable initial pose estimates, we analyze the following aspects:As joint locations of Here, in Equations  and 7),,7), the If the distance between the forward and backward predicted joint locations are further than a threshold θp θp=, the preTherefore, computing the estimates for vertices of the sequence will give hints per vertex about its reliability. For each frame, we gave the joint a score t. We apply a quasi-Newton solver for this energy minimization problem.We formulate the refinement of the initial pose estimation Equation ). The sei in consecutive frames into account. As explained in The weight In the scope of the evaluation, we analyze the effect of our refinement on the initial estimates provided by the Kinect SDK and the stability of the resulting bone lengths. Furthermore, we provide a discussion of the effects of our trust detection and motion prediction steps.The evaluation is performed based on RGB-D video inputs provided by a Kinect v2.0, where the initial skeleton and the respective point cloud are provided by the Kinect SDK . As demoThe skeleton captured by Kinect SDK  is very shown in  is changAs shown in Our method formulates the problem of skeleton pose estimation as a constrained energy minimization problem. During the update over time, the value of the energy function in Equation  decreaseWhen human movement is occluded by objects or the self, the Kinect SDK  may deteThe upper diagram in Furthermore, when we evaluate a “good” motion, one of the standers is temporal symmetry. This means that the joint trajectory in a smooth motion should be predictable from both the previous and future side. For each time frame, the prediction of the joint position from the previous frame and the future frame has huge a difference in the skeleton captured by the Kinect SDK , while wIn order to find the best value for Especially in the case when the moving speed of a joint is changing, our method with forward and backward predictions allows for the correction of erroneous estimates. The bone length constraint and the skeleton symmetry constraint together preserve the consistent skeleton along the sequence. When we consider body joints as moving points in the space, their movement should always follow the physical law of kinematics. Therefore, we address the method of trust data detection to check the erroneous cases . Thus, tAs a result, the length of the skeleton bones are stable and consistent. In addition, our framework not only allows for the detection of outliers within the initial joint location estimates, but also, in such cases, allows for the prediction of new joint location estimates following the temporal observations. As shown in However, our optimization scheme relies on the absence of longer temporal sequences of unreliable estimates in the original captured data. A special case is turning around, the optimization does not work as there is a left-right flip in the captured skeleton (examples shown in This paper presented an approach for optimizing human skeleton data acquired by a single depth camera. The skeleton refinement is based on an energy optimization framework that incorporates skeleton characteristics (such as stabilization and symmetry characteristics) as well as the temporal constraints which represent the physical movement of body. In addition, the proposed refinement can be applied to different skeleton representations used in different frameworks/SDKs.Our method can also be employed to detect erroneous estimates in motion skeleton sequences. Current limitation are that our technique only supports a recovery from errors for up to four erroneous frames in a pose sequence. As modern depth cameras usually have a high frame rates , a self-collision motion like turning around may easily take more than 4 frames in the motion sequence.