The new algorithm’s performance is validated against a manual fixation-by-fixation mapping, which is considered as ground truth, in terms of true positive rate (TPR), true negative rate (TNR) and efficiency. Using only 72 training images with 264 labelled object representations, cGOM is able to reach a TPR of approx. 80% and a TNR of 85% compared to the manual mapping. The break-even point is reached at 2 hours of eye tracking recording for the total procedure, respectively 1 hour considering human working time only. Together with a real-time capability of the mapping process after completed train-ing, even hours of eye tracking recording can be evaluated efficiently.  Areas of Interest (AOIs) are widely used for stimuli-driven,quantitative analysis of eye tracking data and allow the determinationof important metrics such as dwell time or transitions . DespiteVarious methods exist for assigning gaze data to respective AOIs suchas manual frame-by-frame or fixation-by-fixation analysis and dynamicAOIs using either key frames or different types of markers. Ooms et al. state thTo overcome these limitations, object detection algorithms could beapplied directly to the objects of interest and not to markers . InreceIn 2017, mask R-CNN was introduced  as one oSemantic Gaze Mapping. SGM is a manualfixation-by-fixation analysis method used to connect the gaze point ofeach fixation to the underlying AOI in a static reference view . cGOM isbased on a loop function that iterates through all fixations’ middleframes and always performs the same routine of (i) object detectionusing mask R-CNN and (ii) comparison of object and gaze coordinates. Indetail, each frame consists of a number of pixels that can be preciselydescribed by x and y coordinates in the two-dimensional plane with theorigin in the top left corner of the image. Mask R-CNN uses plain videoframes as input and outputs the frame with a suggested set ofcorresponding pixels for each object of interest. If the gaze coordinatematches with the coordinate of an object of interest, cGOM automaticallyassigns the gaze to the respective AOI.RQ1 and RQ2. Thegoal of the study is to investigate whether the new algorithm offers thepotential of replacing conventional, manual evaluation for study designswith tangible objects. Mask R-CNN, which is the core element of the cGOMalgorithm, has already surpassed other state-of-the-art networks inobject detection and segmentation tasks when trained on huge online datasets  How effective is cGOM in assigning fixations to respective AOIsin comparison with the ground truth?(RQ2) At which recording duration does the efficiency of thecomputer-based evaluation exceed that of the manual evaluation?SGM (Semantic Gaze Mapping) and cGOM.The study presented in this article consisted of two parts. Firstly,an observation of a handling task was performed for creating ahomogeneous data set in a fully controlled test environment. Secondly,the main study was conducted by analysing the data sets of the handlingtask and varying the evaluation method in the two factor levelsParticipants. 10 participants  conducted the handling taskwearing the eye tracking glasses. All participants had normal orcorrected to normal vision and were either mechanical engineeringstudents or PhD students.Material. The data was collected using the eyetracking glasses SMI ETG 2 with a scene resolution of 1280 x 960 px of the front cameraoffering a sampling frequency of 24 Hz with the gaze point measurementhaving an accuracy of 0.5° over all distances.Stimuli. The stimuli  were placed on atable covered in a green tablecloth and consisted of five transparentsyringes, two beige bowls and one disinfectant dispenser. Four of thesyringes had a green piston and maximum filling capacities of 2, 5, 10and 20 ml and one was fully transparent with a filling capacity of 50ml.The bowl on the left side was almost completely filled with water andthe other one was empty. Moreover, there was one long piece of adhesivetape attached to the tablecloth with five filling quantities written onit .Task. The participants were asked to decant thefilling quantities specified on the adhesive tape from the left bowl tothe right bowl. The two bowls and the disinfectant dispenser should notbe moved and the participants were only allowed to use the maximumfilling quantity for each syringe. After each completed decanting of oneof the five preset values, the participants were instructed to disinfecttheir hands.SGM andcGOM. Both evaluation methods were compared in terms ofconformance with the ground truth and efficiency, quantified through thetwo dependent variables (i) fixation-count per AOI and (ii) requiredtime for each evaluation step. For calculation of thefixation-count per AOI, three AOIs were defined. Allsyringes were equally labelled as syringe withoutfurther differentiation. The disinfectant dispenser was referred to asbottle. All gaze points that did not affect either AOIwere to be assigned to the background (“BG”). Truepositive rates (TPR) and true negative rates (TNR) were calculated forthe AOIs syringe and bottle toevaluate the effectivity of the algorithm. While TPR describes how manyof the true positive assignments of the ground truth were found by thecGOM tool, TNR describes the same comparison for thetrue negative assignments or non-assignments.For the main study, the data set of the handling task was analyzed bythe two evaluation methods cGOM is able to assign the gaze point ofeach frame to the corresponding AOI, for reasons of comparability theassignment was also performed using only the fixations’ middle frame.For comparison of efficiency, the required time for eachevaluation step of SGM andcGOM was measured and summed up. For all manual worksteps, the times were averaged over all analysts, whereas allcomputational steps were measured in one representative run. Finally,the relation of data size and required time for evaluation was plottedand extended by a linear trend line to allow the determination andvisualization of the break-even point of both evaluation methods.Even though Semantic Gaze Mapping and the manualsteps of computational Gaze-Object Mapping . For the latter, they received a training prior to execution.All operations concerning mask R-CNN were performed using an AWS GPU 1NVIDEA Tesla V100 (Graphics Processing Unit) via Amazon Web Servicescloud computing. Both Semantic Gaze Mapping and the export of gaze datawere performed using SMI’s BeGaze 3.6.Five professional analysts , experienced in eye tracking data analysis, performed both theevaluation using SGM and five steps for cGOM with thelatter consisting of two computational operations and three that demandmanual execution by the analyst. The respective steps of both methodsare explained in the following.The evaluation process is divided into three purely manual steps forSemantic Gaze Mapping. Initially, the evaluation wasprepared once by loading a reference image with all objects of interestinto the software and drawing AOIs accordingly. This reference image andall respective AOIs can be reused over all trials. Subsequently, themanual mapping for the recordings of the handling tasks was performedfor all fixations until at last, the data was exported.Computational Gaze-Object Mapping. First, trainingimages were collected to train mask R-CNN on the handling task setup,using only 72 frames from the recording of a pilot study. All imageswere taken from the front camera of the eye tracking glasses, resultingin a corresponding resolution of the training and the test images. Dueto the small amount of training images, it was of great importance tokeep environmental conditions constant throughout all studies. MaskR-CNN needs labelled images as input for training, just as they shouldbe outputted later. To do this, close contour masks were manually drawnon all objects of interest in the training images. Once all images werelabelled, the training of the neural network was started. This operationis purely computer-based and thus did not require the analyst's workingtime, allowing the analyst to export the gaze data in the meantime. ThecGOM algorithm requires start time, duration and endtime of all fixations, the x and y coordinates of the gaze point as wellas the raw video recording of the front camera. Once all data wasprepared, in a final step, the algorithm performed the gaze-objectmapping of all trials in a single run.Methods, the algorithm in its coreis an already established state-of-the-art convolutional neuronalnetwork for object detection including the prediction of masks. Figure 3shows that the algorithm is able to perform the mapping of specific gazepoints to gazed-at objects when comparing the position of the objectmasks with the gaze point coordinates. On the one hand, this chaptershall evaluate the mapping performance of the algorithm in comparison tothe manual mapping, which is considered as ground truth. On the otherhand, it shall provide an overview of the time needed to operate thealgorithm and present the break-even point from which on the algorithmapproach is faster than the manual mapping.As presented in Effectivity evaluation. Figure 4 shows the resultsfor TPR and TNR of the two AOIs syringe andbottle. For the AOI syringe, thecGOM tool achieves a TPR of 79% and a TNR of 85%. Forthe AOI bottle, the TPR of the cGOMtool is 58% and the TNR is 98%. Table 1 shows the overview statistic ofthe fixation mapping both for SGM and for cGOM, including the number offixations mapped [-], the average fixation duration [ms] and thestandard deviation of the fixation durations [ms] for the two examinedobjects syringe and bottle.Efficiency of the manual mapping. The analystsneeded a total of 128 minutes on average for the mapping, with astandard derivation of 14 minutes. The subtasks were preparation,mapping and export of the data sample. The main time was needed to mapall 4356 fixations. The exact sub-times for the total mapping processare presented in Table 2.Efficiency of the computational mapping. The wholeprocess using the algorithm required 236 minutes in total. The exacttimes of the subtasks of the mapping process are shown in Table 3. Themapping aided by the algorithm requires manual and computational steps.The cGOM tool spends most of the time for the steps oftraining and mapping that are solely conducted by a computer. The stepsfor collecting training images, manual labelling of the training imagesand export of the main study gaze data have to be performed by anoperator. The labelled training set of this study included 72 imagesshowing in total 264 representations of the objectsyringe and 32 representations of the objectbottle. These three manual steps required 129 minutesin total.Break-even point. Figure 5 shows the approximatedbreak-even point for the average time required for the manual mapping.The measurement points for the manualcGOM tool. For the manualSGM tool, the ratio between to-be-assigned gaze datasample and the required time for the mapping is 2.48 on average. For thecGOM tool, the ratio is 0.77 on average. The break-evenpoint of the whole manual mapping procedure and the whole computationalmapping procedure lies approximately at a data sample size of 02:00h.When comparing only the manual steps of both procedures, the break-evenpoint reduces to a data sample size of approximately 01:00h .mapping are the averaged interims for the single participant datasets. The completely analysed data sample (video time) totals up to 52minutes. The start time for the data mapping was dependent on therequired preparation for the manual and for the computational mapping.The average of the measured mapping times is extended by a linear trendline . The grey cone shows the extendedlinearized times of the fastest and slowest expert. All linearization isbased on the assumption that the experts take sufficient breaks. Thehorizontal dashed line in Figure 5 indicates the time of the threemanual steps required for the cGOM offersthe potential of replacing conventional, manual AOI evaluation inexperimental setups with tangible objects. Therefore, manual gazemapping using SGM, which was considered as groundtruth, and cGOM were compared in regards to theirperformance. In the process, it was quantified whether the new algorithmis able to effectively map gaze data to AOIs (RQ1) and from whichrecording duration on the algorithm works more efficiently than themanual mapping (RQ2). Based on the results of this study we evaluateboth research questions.The goal of the main study was to investigate whether the newlyintroduced machine learning-based algorithm (RQ1) How effective is cGOM in assigning fixations torespective AOIs in comparison with the ground truth?syringe and 58% for the AOIbottle. For training the neural network, not only thenumber of training images but also the total number of objectrepresentations in these images is important. Since the stimuliconsisted of five syringes and only onebottle, the 72 training images included 264representations of the AOI syringe on which the neuralnetwork could learn, but only 32 representations of the AOIbottle. Due to the small learning basis, the masksproduced by the algorithm sometimes did not include crucial featureslike the tip of the bottle .The two objects used during the handling task were deliberatelyselected because they represent potential challenges for machinelearning. On the one hand, the syringes are partially transparent andconstantly change their length during decanting. On the other hand, boththe syringes and the bottle have partly tapered contours, which wereassumed difficult to reproduce by close contour masks, in particularwhen working with small training data sets. According to the resultspresented in Figure 4, the assignment by the computational mapping has aTPR of 79% for the AOI syringe, the TNR is slightly better thanthe TPR, whereas for the AOI bottle the TNR greatlyexceeds the TPR. The relation between TPR and TNR can be well explainedby the quality of the created masks. The masks tend to rather fill toolittle of the object than too much. The more the masks are directedinwards from the outer contour or exclude crucial features like in caseof the bottle, the less true positives are registered, but the higherthe probability of true negatives being recorded.For the AOI syringe the conformance with the ground truth isalready promising, but can still be further increased. For the AOIbottle, there is no sufficient true positive rate yet,but with almost 60%, it is surprisingly high given the fact that theneural network was shown only 32 object representations of thebottle during training. In comparison, large onlinedatabases such as MS COCO work with hundreds to thousands of labelledimages for training one single object. Objects already represented inthe COCO data set may even be used in a plug-and-play approach withouttraining, using only the gaze-object mapping feature of the presentedalgorithm. In contrast to the 72 images needed for the tailored approachof this study, the COCO dataset includes more than 320k labelled imagesin total  At which recording duration does the efficiency ofthe computer-based evaluation exceed that of the manualevaluation?cGOM tool exceeds the manual evaluation when therespective procedure in total needs less time. The amount of data fromwhich onwards the cGOM tool is faster than the manualevaluation is called the break-even point. For the break-even point onehas to distinguish between the time for the total computational mappingprocedure and the time, a person has to invest . The main part of thetime is needed for training the algorithm on the training images, whichis solely performed by the computer, and for the labelling of thetraining images, which has to be performed once by the analyst. For thetotal procedure and the consideration of the average speed for manualmapping by the experts, the break-even point lies at 2 hours of eyetracking recording. When focussing only on the time a person has toinvest, this break-even point reduces to just 60 minutes of eye trackingrecording.The cGOM algorithm, thealgorithm needs less than 8 minutes for every 10 minutes of eye trackingrecording and thus is able to work in real time with a factor of 0.77.This is by far faster than the manual mapping, which has on average aratio of 2.48 between recording length and time needed for the manualmapping. The difference in evaluation speed does not take into accountsymptoms of fatigue on the part of the analyst that increaseconsiderably with longer evaluation times of SGM. Withthe given break-even points of only 2 hours of eye tracking recording orrather 1 hour, considering only the human working time, and thereal-time capability of the mapping, the authors evaluate the efficiencyof the cGOM tool exceeds the manual mapping for themajority of mobile eye tracking studies.After the preparation of the Due to the novelty of the algorithm presented in this study, thereare several limitations regarding the results. First and foremost, toachieve the best possible results with a minimum amount of trainingimages, the results presented are only valid in a laboratory environmentwith constant conditions. The amount of training data has to be higherto cover the possible variations in field studies. Furtherinvestigations are also required to determine which other objects aresuitable for the presented algorithm and how the characteristics andnumber of objects influence the evaluation time. Although the comparisonwith SGM as a ground truth allows for a good assessment of thealgorithm’s potential, it is questionable that the results of the manualmapping are always error-free due to the subjective evaluations by theanalysts. The approach using five independent professional analyststries to compensate this limitation.Moreover, the gaze data set consisted of only 52 minutes videomaterial and the derived linearization may not be true, as the humananalysts cannot work uninterruptedly and the mapping speed decreases.The results for the measured times for the computational and the manualmapping may not be achieved when using a different computer system ormapping a different set of gaze data. The computational system used forthis study has a state-of-the-art graphics processing unit (GPU), whichis not comparable to the one of a standard computer regarding the speedand hence the time needed for training the algorithm and mapping thegaze data. Cloud computing, which was used in this study, lowers theburden as it continuously decreases the costs and democratises theaccess to the amount of computational performance needed.cGOM algorithm is afirst step to address this gap and complement state-of-the-art methodsof automated AOI analysis (e.g. markers). For objects that are trainedwith a sufficient amount of training data like the AOIsyringe, the algorithm already shows a promising TPRand TNR. Due to the early break-even point, both for the total procedureand in particular considering human working time, as well as thereal-time capability of the mapping process, even hours of eye trackingrecording can be evaluated. Currently, this would require an amount oftime for the manual mapping that is not or difficult to realise, seenfrom an economical and a humane point of view. Consequently, thisapproach using machine learning for the mapping task promises to enablethe mapping of a great amount of gaze data in a reliable, standardizedway and within a short period. It lays the foundation for profoundresearch on AOI metrics and reduces the obstacles many researchers stillhave when thinking about applying mobile eye tracking in their ownresearch.As described in the introduction, AOI analysis of mobile, interactiveeye tracking studies with tangible objects is still a time-consuming andchallenging task. The presented http://biblio.unibe.ch/portale/elibrary/BOP/jemr/ethics.html.The ethics committee Zurich confirms that this research project does notfall within the scope of the Human Research Act (HRA) and therefore noauthorization from the ethics committee is required. . All participants were asked toread and sign a consent form, which describes the type of recorded dataand how this data will be used for publication. The authors declare tonot have a conflict of interest regarding the publication of thispaper.The authors declare that the contents of the article are in agreementwith the ethics described inThe authors Julian Wolf and Stephan Hess contributed equally to thepublication of this paper and are both considered first author. We wishto thank all participants of this study for their time.


Pectus excavatum (PE) is one of the most common chest wall defects. Accurate assessment of PE deformities is critical for effective surgical intervention. Index-based evaluations have become the standard for objectively estimating PE, however, these indexes cannot represent the whole information of chest CT images and may associated with significant error due to the individual differences. To overcome these limitations, this paper developed a computer-aided diagnosis (CAD) system based on the convolutional neural network (CNN) to automatically learn discriminative features and classify PE images. We also adopted block-wise fine-tuning methods based on the transfer learning strategy to reduce the potential risk of overfitting caused by limited data and experimentally explored the best fine-tuning degree. Our method achieved a high level of classification accuracy with 94.76% for PE diagnosis. Furthermore, we proposed a majority rule-based voting method to provide a comprehensively diagnostic results for each patient, which integrated the classification results of the whole thorax. The promising results support the feasibility of our proposed CNN-based CAD system for automatic PE diagnosis, which paves a way for comprehensive assessments of PE in clinics. PE is a type of congenital malformation in which the sternum and thorax of the anterior thoracic wall are recessed inward relative to other anterior chest walls2. The deformity causes a funnel-like depression in the chest, which not only affects normal cardiovascular and respiratory functions, but also leads to exercise intolerance and physical limitations3. To repair the PE, surgical techniques are commonly used4.Pectusexcavatum (PE) is one of the most common chest wall defects occurring once in every 400 newborns5 firstly used medical images to quantify the degree of a chest wall deformity, which is referred as the Haller index (HI). The HI, which is defined as the widest transverse diameter of the internal chest divided by the distance between the anterior spine and posterior sternum, has been widely employed as the decision criterion for surgical correction6. Additionally, some approaches have been proposed to modify HI and improve the accuracy of PE diagnosis10. However, these assessment methods proposed in previous studies focused on some indexes which were selected based on clinic experiences to reflect the chest shape. Neither HI nor other modified indexes can represent the whole information of chest CT images. Due to individual differences in chest shape, existing indexes could hardly be applied as a universal diagnostic and assessment approach, especially in non-standard chest cases, such as narrow chest and asymmetric chest wall. To overcome this limitation, we proposed to use convolutional neural networks (CNN) to automatically learn representations from CT images and evaluate the PE deformities. To our knowledge, no previous study has evaluated the feasibility of using a CNN-based computer-aided diagnosis (CAD) system to assess the severity of PE deformities.Accurate measures of PE are critical for effective surgical interventions and many methods have been proposed to assess the severity of PE. To the best of our knowledge, Haller et al.11. CNN is a deep learning method which has a natural advantage in utilizing the 2D structure of an image12. CNN-based CAD systems have been applied in many medical fields and have achieved promising results in the detection of lung nodules, breast lesions and colonic polyps15. Inspired by this, we investigate whether a CNN-based CAD system can be employed as an end-to-end classifier to automatically diagnose PE. Without any requirement for subjective and manual feature extraction processes, the proposed CNN model can automatically learn discriminative features from CT images and classify PE cases. In addition, it can overcome the diagnostic difficulty caused by the individual differences in PE and has excellent diagnostic performance in cases never seen.CAD is a computerized process that can assist radiologists in interpreting and diagnosing medical images and provide a second objective opinion to improve the clinical diagnostic results16. For small datasets, training the CNN models from scratch may require excellent computational power, sufficient memory resources and considerable time17. To address these issues, many researchers proposed the transfer learning strategy, that is, to utilize the CNN weights that are pretrained on large-scale public datasets to initialize the desired models18. However, the transferability from natural images to CT images with PE symptoms has not yet been explored in previous studies. In this study, we investigated whether transfer learning can be applied to a CT dataset of PE and explored the impact of transfer learning on the model’s performance.In general, training CNN models requires large-scale annotated datasets, which are usually unavailable in medical fields due to the high manual labeling costsBesides automatic assessment of PE using CT images, we have developed a majority rule-based voting method to provide comprehensive diagnoses for patients with PE. Existing approaches for PE diagnosis depend on one or a few images of patients, which may lead to incomplete understanding of the patient’s condition. Though images of the most serious deformation part of the thoracic wall usually can reflect the severity of PE, partial images cannot carry out the whole information. Based on the proposed automatic assessment method, we detected all images of the patient with high efficiency and took all images of the chest into consideration for a more comprehensive diagnosis of PE. In our CAD system, we received and predicted all images of each patient, and output the corresponding proportions of the three labels as a reference to assisting radiologists in diagnosis. We also adopted the majority rule-based voting strategy to draw a final comprehensive conclusion for the severity of each patient’s PE deformities.There are threefold of this study. First, we introduced the CNN method to automatically and intelligently extract discriminative features from the CT images for assessing the severity of patients with PE. It is the first study to develop and evaluate a CNN-based CAD system for PE diagnosis. Second, we adopted the transfer learning strategy using the pretrained weights of the VGG16 and VGG19 networks and fine-tuning the top few layers block-wise to improve the classification performances on our dataset. Our proposed method achieved a high level of classification accuracy with 94.76% for PE diagnosis, which verified the feasibility of CNN models in PE diagnosis. Finally, we proposed a CAD system to integrate the classification results of various parts of the thorax and draw comprehensive diagnostic conclusions from the perspective of individual patients.The remainder of this paper is organized as follows. “Because there are no publicly available datasets on PE, we collected the dataset from 42 subjects at Guangdong General Hospital from 2010 to 2019 in this study. All participants received a traumatic chest CT routine scan on a Philips iCT 256 scanner . The scanned breast contour is saved in the form of CT images. Among these 42 subjects, there were 27 subjects with PE  and min(x) are the maximum and minimum intensity value of the entire image, respectively. In addition, the original images have a size of  Data preprocessing is performed to facilitate the subsequent training of the neural network. Neural networks are not very good at processing data with relatively large values or large differences because these issues are not conductive to the training of the neural network. They may cause a large gradient update of the neural network and thus the neural network would not converge. Therefore, it is necessary to normalize the data before neural network training. The normalization is usually performed to transform the input values of the CNN model and confine them to the same range. In this paper, we performed min-max normalization on each channel of the image, that is, in each channel we subtracted the minimum intensity value of the entire image from each pixel, and then divided the result by the difference between the maximum intensity value and the minimum intensity value so that any intensity value of the input image is in the range of [0–1]17. Geometric transformation is based on image manipulation processes, including flipping, cropping, rotation, color space transformation and so on. In this research, we performed a random rotation ranging from Theoretically, the excellent performance of a deep neural network greatly relies on the scale of the labeled dataset. Unfortunately, large-scale labeled datasets are unavailable in some medical domains. Therefore, some techniques have been performed to prevent overfitting when training a neural network on small dataset, such as data augmentation. Data augmentation is a technique that adds noise or applies geometric transformations to existing picturesentclass1pt{minimaA CNN is a type of deep learning network that evolved from the multilayer perceptron. It has been proven that a CNN can be effectively applied in areas such as image recognition and classification, object detection and natural language processing. Unlike the traditional multilayer perceptron that usually uses a fully connected network, a CNN performs well in the field of image processing due to its structural characteristics of local connectivity, shared weights and down sampling.In this research, we introduced two state-of-art CNN architectures: VGG16 and VGG19. We utilized these CNN architectures and transferred the weights that are pretrained on ImageNet to achieve excellent classification performance on our own dataset. VGG16 is composed of 13 convolutional layers and 3 fully connected layers  and support vector machine (SVM). (2) The fully connected layers of the pretrained model can be replaced with new fully connected layers and the pretrained weights can be fine-tuned to perform classification tasks on our dataset. We will introduce these two methods separately in the next two sections.There are two main types of transfer learning related to deep learning. (1) A CNN pretrained on large datasets can be used as a feature extractorA CNN is usually used as an end-to-end image classifier, that is, when the model receives input images, forward propagation is performed throughout the network and the classification results are output at the end. In fact, any intermediate layer (such as an activation layer or a pooling layer) can be treated as the output end, and its output can be extracted as feature vectors. Subsequently, we can use these feature vectors to train a traditional machine learning model.24. Finally, we trained a new classifier, such as the SVM, LR and decision tree (DT), using the normalized and selected feature vector. It is noted that this way can take advantage of the pretrained networks to easily extract features and overcome the complex and time-consuming defects of the traditional feature extraction method. In addition, the training process has no high requirements on computing power and is relatively simple compared to full deep learning training.A simple way to implement transfer learning is to use a pretrained network as a feature extractor. First, we chose a CNN pretrained on ImageNet and removed the top layers, including the fully connected layers and SoftMax classifier. The rest of the pretrained network was treated as a feature extractor for a new dataset and could generate a feature vector when images are input. Second, we normalized the extracted features, and removed the unnecessary features via principal component analysis (PCA). PCA algorithm is a method that can reduce the data dimension while minimizing the loss of information. It is usually used for data preprocessing and data compressionIn this research, we build our models using transfer learning and a fine-tuning strategy in this way.25. Therefore, we modified the fully connected layers and built a new SoftMax classifier with three outputs for three categories: normal, mild and severe.First, we kept the original structure except the fully connected layers and inserted the new fully connected layers behind it. The original architecture of the pretrained CNN was designed for the ImageNet dataset that includes 1000 categoriesSecond, we froze the weights of all layers except the fully connected layers, and randomly initialized the weights of the new fully connected layers. Then, we trained the model on our dataset with a small learning rate. The initial parameters of the convolution layers transferred from the pretrained network contain a large number of useful convolution filters with powerful feature discrimination capabilities. However, the weights of the newly added fully connected layers are completely new and random, and the weight update may damage the feature extraction capability of the aforementioned pretrained convolution layers due to the back propagation in the whole network. Therefore, it is necessary to freeze the pretrained convolution layers. This enables back propagation to be restricted among the fully connected layers when the fully connected layers are trained and forward propagation is performed throughout the whole network.Finally, after the fully connected layers were trained, we adopt a fine-tune strategy, that is, we unfroze the convolutional base block-wise, trained the model with a very small learning rate and continuously monitored the performance. In a CNN, the closer the network layer is to the top, the more abstract the features extracted by the CNN. From the bottom layer to the top layer, the features extracted by the neural network undergo an in-depth abstraction process from edges, shapes, and patterns to targets. Lower layers refer to the generic features that are independent of the task while higher layers extract the problem-specific features that are closely dependent on the task. To obtain better classification results and explore the impact of block-by-block fine-tuning on the model’s performance, we unfroze the convolutional base block-wise and resumed the backpropagation on the unfrozen layers. The block-wise fine-tuning process is shown in Fig. Based on the automatic assessment of each image, we further investigated the feasibility of comprehensive diagnosis for patients with PE. As shown in Fig. To address the limitation of 2D image-based diagnosis, we built a CAD system based on the proposed CNN model to provide an overall assessment of PE for each patient. First, the CAD system received and processed all images of each patient, and then automatically classify each image using the proposed CNN model. Second, the CAD system counted the number of images of each type, including normal, mild PE and sever PE, and output the proportion, respectively. The distribution of all CT images can reflect 3D components of the deformity of chest with PE. Finally, we adopted the majority rule-based voting strategy to draw a final comprehensive conclusion for the severity of each patient’s PE deformities. In brief, the proposed CNN classifier predicts the category label of each image, and then the majority rule-based voting method outputs the final predicted label as the one that receives most of the votes across all categories. For instance, if a patient has the largest proportion of its CT images in category A, the CAD system will make a prediction that category A is the overall category label. This outcome can be used as a comparative assessment to assist radiologists to get a better understanding of the condition of the entire chest instead of a certain region.Considering the size of our dataset and the model selection issues, we adopted 10-fold cross validation to overcome the shortcomings of the data scale and make full use of the dataset to test the pros and cons of the algorithm and the performance of the model. As listed in Table Before training the newly added fully connected layers and conducting fine-tuning, we added a dropout layer to the dense fully connected layers so that some neurons in the hidden layer are randomly inactivated with a probability of 0.5, and the weights of these neurons are no longer updated during back propagation. The batch size we set is 64 and the SGD optimizer with a small learning rate is used when fine-tuning the network. We also adopted the early stopping strategy to continuously monitor the performance and reduce the risk of overfitting. The learning rate was initially set to 0.001 and would automatically decrease when the model training reaches a plateau. The training would end when the performance no longer improved. We implemented the experiments with PyCharm (using the Keras library with TensorFlow as a backend) and an NVIDIA TITAN Xp GPU with 12 GB of onboard memory.Ethics approval and consent to participate  was obtained from all the participants to complete the protocol approved by the Guangzhou First People’s Hospital Department of Ethics Committee. Informed consent was obtained from the subject’s parent, if subjects are under 18. All the research was performed in accordance with the Declaration of Helsinki.This section provides the validation results. First, we performed a comparison between the random initialization of the network and two transfer learning methods. Second, the effect of block-wise fine-tuning on the model’s performance were presented. Third, we visualized the areas activated by VGG network for each category. Finally, we showed the comprehensive diagnosis results obtained by our CAD system with the optimal configuration.In this part, we compared the random initialization of the network with two transfer learning methods to explore the impacts of different strategies on the model’s performance. Random initialization means that the neural network is trained from scratch with our dataset. We adopted the Gaussian distribution with a low deviation to initialize our weights to help the model converge to a meaningful optimal solution and overcome the negative effects of zero initialization. As an alternative, we used the pretrained VGG16 and VGG19 to extract the features of our dataset and saved the feature vectors as hdf5 files. We removed the top layers of the network (including the fully connected layers) and used the output of the last pooling layer of the VGG network as the feature vector. The output of the last pooling layer is The results of three methods are shown in Table i  fine-tuning means that we froze the entire pretrained architecture except for the last i convolutional block and fully connected layers. B0 fine-tuning means that we kept all the convolutional blocks frozen and only trained the new fully connected layers while full fine-tuning means that we unfroze all the convolutional blocks and updated the weights on the whole network.In this part, we explored the effect of block-wise fine-tuning on the model’s performance and investigated the degree of fine-tuning required to achieve optimal performance. We tested six fine-tuning schemes. BSpecifically, we monitored the performance of each model when performing fine-tuning. We randomly spilt the training set and used 15% of the data as the validation set, and the remaining 85% is used for training. The validation set was separated from the training set and used to monitor and tune the hyper-parameters. Figure The block-wise fine-tuning results are shown in Fig. 17. Therefore, the optimal degree of fine-tuning should be balanced between shallow (too few training parameters) and deep fine-tuning (too many training parameters), which in our case is the B2 fine-tuning.It is also noted that the VGG16 and VGG19 networks pretrained with the fine-tuning strategy achieved the highest classification accuracy of 93.95% and 94.76%, and they improved the accuracy by up to 10.15% and 8.79%, respectively, compared to random initialization. During the block-wise fine-tuning experiment, the optimal classification result, which indicates the optimal number of fine-tuned convolutional blocks, is achieved by the B2 fine-tuning method. Considering that the network with shallow fine-tuning has few unfrozen convolutional layers and trainable parameters to learn the typical features on the new dataset. And the weights of the model still too much involved the features of the original dataset. On the other hand, the deep fine-tuning would unnecessarily retrain the weights associated with the generic features at the bottom layers, resulting in too many training parameters for the model and thus overfitting on our small dataset27. Generally, the red regions on the Grad-CAM map represent the areas where the network pays the most attention during the classification, while the blue regions receive the least attention. Several Grad-CAM maps of the three kinds of subjects are shown in Fig. To understand what features have been learned and which area in the input images have been activated by VGG network, we used gradient-weighted class activation mapping (Grad-CAM) to extract gradients from VGG19’s final convolutional layer and highlight the regions that are most responsible for the predictionAccording to aforementioned analytical results, the best classification performance was achieved by adopting the b2 fine-tuning method and the VGG19 pretrained network. The confusion matrices of the entire dataset under this setup and 10-fold cross-validation is shown in Fig. After the proposed CNN classifier predicts the category label of each image, our CAD system integrated the classified results of all images for each patient, counted the percentage of all three symptoms and gave a final result based on the majority rule. We then compared the results obtained by the majority rule-based voting method in 42 cases with the diagnostic results given by radiologists. The confusion matrix of the comparison results is shown in Table As mentioned above, a single image cannot not represent the overall characteristics of the patient. A series of CT images of two patients with PE are shown in Fig. To the best of our knowledge, this is the first time that a CNN-based CAD system was applied for automatic assessment of PE. Our results indicated that the proposed CNN model with the transfer learning and fine-tuning strategy was able to classify the severity of PE deformities with an average high accuracy of 94.76%. By combining the classified results of various parts of the patient’s chest, our CAD system provided the proportion of images belonging to each symptom and drew comprehensive diagnosis results based on the majority rule, providing second objective opinions to assist radiologists in interpreting and diagnosing PE images. Comparing the results obtained by the majority rule-based voting system in the 42 cases with the diagnostic results given by radiologists based on clinical experience, we concluded that the proposed computer-aided diagnosis process has high reliability.28 proposed an algorithm related to CT image processing to automate the initialization and facilitate the index calculation process. Glinkowski et al.29 developed an optical 3-D chest measurement system and automatic data processing algorithms to assess the severity of deformities in PE. However, all these methods still relied on the subjective and manually selected indexes, which cannot represent the whole information of chest and be applied as a universal diagnostic and assessment approach due to the individual differences. To the best of our knowledge, previous works have not considered the combination of deep learning and a CAD system to eliminate subjective and manual feature extraction processes. Our results indicated the feasibility of using a CNN-based CAD system to intelligently and automatically learn discriminative features and evaluate the severity of a PE deformity.For automatic diagnosis of PE, many researchers were committed to modifying index-based methods to improve the evaluation metric and reflect the severity of PE deformities. Kim et al.In medical domains, a large amount of labeled data is usually unavailable, which is not conducive to the training of CNN. To address the overfitting problem caused by limited data, we adopted transfer learning strategy to improve the classification performance of the CNN model on our dataset. Our experimental results suggested that fine-tuning achieved better classification performance on our dataset than the feature extractor method for the application of transfer learning. Implementing the block-wise fine-tuning strategy on the pretrained networks can find the optimal fine-tuning degree and obtain the highest classification accuracy for the experiment. We adopted the b2 fine-tuning method, which means all the pretrained network layers were frozen except for the last two convolutional blocks to achieve the highest classification accuracies of 93.95% and 94.76% using the pretrained VGG16 and VGG19 networks, respectively. A possible explanation for this is that among the transferred weights, it is the weights of the bottom layer that are truly valuable for the new dataset because they represent the universal features in various images, including natural and medical images. The weights of the top layers that are related to the unique features of respective datasets must be retrained to fit our own dataset. This explains why modest fine-tuning works best.Compared with the diagnosis based on a single image of a patient, our study considered the role of a single image classification in the overall diagnosis of a patient. Because the classification of a single image helped the overall diagnosis of the patient, we also focused on the comprehensive diagnosis of the patient as an individual. The proportion results given by our CAD system can well reflect the overall severity of PE deformities. We also followed the majority rule to draw a conclusion on the overall severity of patients, which is equivalent to assigning the same weight to each picture and then the category with the largest proportion can determine the overall characteristic of the patient. In fact, the same weight took into account the distribution of all image classification results for each patient on the basis of a certain rationality. We tended to think that based on what we have learned from our dataset, the same weight setting works well when using other new data.The present study had several limitations. First, although we addressed the problem of limited annotated data using a data augmentation technique and transfer learning strategy, a larger sample of patients with PE is needed to draw stronger conclusions. Second, all images from the same patient were assigned the same weight and then combined to make a comprehensive diagnosis decision in this study. Considering that the presence of an image representing a severe symptom indicates more warning meaning than a normal image, the former may be given a higher weight and future studies should investigate the optimization of the weights to obtain more reasonable final results. Finally, our study is a preliminary attempt to apply deep learning to PE assessment. An automatic and application-oriented CAD system remains to be developed in further work.This study had a number of strengths. It is the first time that a CNN-based CAD system was investigated for automatic diagnosis of PE. We exploited a CNN approach based on transfer learning and a fine-tuning strategy to classify the severity of PE deformities in various parts of each patient. We subsequently combined the classified results of various parts of the patient’s chest and derived the final diagnosis decision on the overall condition for each patient via majority rule. Our system eliminated subjective and manual processes and provided accurate and intelligent diagnostic advice.


The combination of biomechanics and inertial pedestrian navigation research provides a very promising approach for pedestrian positioning in environments where Global Positioning System (GPS) signal is unavailable. However, in practical applications such as fire rescue and indoor security, the inertial sensor-based pedestrian navigation system is facing various challenges, especially the step length estimation errors and heading drift in running and sprint. In this paper, a trinal-node, including two thigh-worn inertial measurement units (IMU) and one waist-worn IMU, based simultaneous localization and occupation grid mapping method is proposed. Specifically, the gait detection and segmentation are realized by the zero-crossing detection of the difference of thighs pitch angle. A piecewise function between the step length and the probability distribution of waist horizontal acceleration is established to achieve accurate step length estimation both in regular walking and drastic motions. In addition, the simultaneous localization and mapping method based on occupancy grids, which involves the historic trajectory to improve the pedestrian’s pose estimation is introduced. The experiments show that the proposed trinal-node pedestrian inertial odometer can identify and segment each gait cycle in the walking, running, and sprint. The average step length estimation error is no more than 3.58% of the total travel distance in the motion speed from 1.23 m/s to 3.92 m/s. In combination with the proposed simultaneous localization and mapping method based on the occupancy grid, the localization error is less than 5 m in a single-story building of 2643.2 m The inertial pedestrian navigation system has a wide range of applications in hotspots such as firefighting, indoor security, tunnel patrols, and other fields. These practical applications typically require pedestrians to repeatedly move vigorously in the same indoor environment, such as running and sprint, rather than just regular walking. The reliable navigation information can guarantee the smooth execution of tasks and the safety of personnel in these fields. Although great progress has been made in recent years, there remain various challenging problems such as unbounded accumulative heading error and the capability in dealing with the drastic motions.The zero-velocity update (ZUPT) algorithm has been widely applied in the field of inertial pedestrian navigation because it is suitable for the foot-mounted IMU case to compensate for traveled distance error during normal walking by introducing heuristic periodic zero velocity corrections . In addiThis mechanism considers the separation constraints both in horizontal and vertical directions, which is more effective to correct the pedestrian height error. Since these methods require a long enough period of stationary time for the foot to recognize the stationary state, correct for velocity and attitude errors, and apply the separation constraints. During running and sprint, however, this stationary time is very short or even non-existent. Therefore, these methods cannot achieve the same positioning accuracy in running and sprint as in normal walking.Another pedestrian inertial navigation method is the pedestrian dead reckoning method (PDR), which estimates the pedestrian’s step length and heading, and performs dead reckoning to track the pedestrian’s position. However, with the current state of the art of MEMS technology, the accuracy of accelerators and gyroscopes are not good enough for deriving step length and heading estimation over longer terms due to their large biases and scale factors.Several mathematical models for indirect stride/step length estimation in the PDR algorithm. Compared with ZUPT, the indirect stride/step length estimation could restrain the accumulated error caused by the measurement noise of inertial devices after integration.The inverted pendulum model is widely used to estimate step length in these PDR systems . The geoAs for head estimation error correction, combining both gyroscope and magnetometer inputs has yielded some success since the two sensors have complementary error characteristics—gyroscopes give poor long-term orientation, while magnetometers are subject to short-term ,8,9. NevMap matching could correct not only the heading estimation but also position estimation ,13,14. TThe occupancy grid map is an important representation model in the mobile robot environment mapping application, proposed by Moravec and Elfes . This moTo enhance the capability of the inertial pedestrian navigation system in dealing with the drastic motions and address the unbounded accumulated positioning error, this paper proposed a triple-node IMUs pedestrian inertial navigation system with an occupancy grid-based FastSLAM algorithm. The triple-node IMUs are composed of one waist-worn IMU and two thighs-worn IMUs. The thighs-worn IMUs are utilized to detect and segment the gait. To estimate the step length accurately for drastic motions, we introduce the horizontal acceleration probability distribution function into a step length estimation process to address the issue that the statistic feature fitting method cannot deal with drastic motion with a large speed range. Furthermore, we construct a virtual range sensor to precept the external environment and introduce it into an occupancy grid-based FastSLAM algorithm to correct the long-term position error of the inertial odometer.This paper is organized as follows: In This section presents the step detection, step length estimation, and heading estimation algorithms in the proposed trinal-IMUs pedestrian navigation system. In this work, three body locations are waist and thighs. We refer to one waist-worn IMU and the two thigh-worn IMUs. Each IMU is comprised of one 3D-accelerator and one gyroscope, as shown in Most of the human drastic movements are driven by thighs with the strong periodicity, especially for running and jumping. In order to describe the periodicity feature of thighs changes, a lower limb model is constructed, as shown in The thigh-worn IMUs are not parallel to the femurs due to the presence of muscle tissue in the thighs. To eliminate the deviation between thigh-worn IMU and thigh frame, a model alignment is required in addition to the initial alignment of the inertial navigation system.z-axis of n-frame, so that the attitude matrix can be obtained by the following equation:n is the local navigation frame (ENU), n-frame.The initial alignment process requires the pedestrian to stand still before the navigation begins. When INS is stationary with respect to the ground, the accelerometer measurements only include gravity and the gravity direction coincides with the b with reference to the thigh frame The purpose of model alignment is to solve the attitude deviation matrix of the body frame After initial alignment and model alignment, the inertial odometer utilizes the difference of pitch angle between thighs to segment steps. As can be seen, the zero-cross point of pitches difference correspond to step 6 of one running action breakdown, and the pitches difference zero-crossing point correspond to step 6 in l indicates the pedestrian leg length, For pedestrian step length estimation, the inverted pendulum model is widely used. In such model, the step length is calculated as follows:K is the coefficient. In addition, [A, B, C are the regressive parameters. On the one hand, the vertical acceleration included in methods [Another class of methods for step length estimation is the fitting of statistical features of acceleration, such as in  by fittiSL=K·AmaxSL=K·Amaxdition, [SL=A+B·SFThe acceleration probability distribution function not only contains the maximum and minimum acceleration information but also characterizes the degree of dispersion of the data. Moreover, it contains the components of different velocities. Hence, fitting the probability distribution function and step length can achieve more accurate step length estimate, and better adaptability to various velocities. To establish a step estimation method suitable for drastic motions with a large speed range, we proposed to fit step length through a segment function of horizontal acceleration norm probability distribution of the waist-worn IMU within one gait.Let Since we can record the acceleration values for each moment within one gait, and calculate its probability distribution function, M times in different speeds. For the m-th movement, according to Equation lm=s and the map In general, the goal of SLAM is to estimate the pose of pedestrian i-th landmark, and N is the number of environment features. The above decomposition shows that the a posterior probability distribution of SLAM can be decomposed into the product of the a posterior probability distribution of the pedestrian’s path and the posterior probability distribution of environment features given a known path.The FastSLAM approach attempts to estimate the full SLAM posterior:N environmental map features is estimated by a bundle of binary Bayesian estimator. The conditional probability of each feature is the probability of the grid being occupied in Occupancy Grids-based FastSLAM. As a result, each particle is composed of a pedestrian pose and a map; thus, the m-th particle at time t is defined asm-th particle where n is the total number of particles used in the particle filter, m-th particle at time t, and m-th particle.The FastSLAM algorithm uses a particle filter to estimate pedestrian trajectory distribution. For each particle, the individual path-based map features are independent of each other. Therefore, in the second part of Equation , the conThe FastSLAM algorithm consists of four basic steps: sampling new poses through the motion model, updating the environmental map, calculating the particle importance weights, and resampling based on particle weights.m-th particle:The first step of algorithm is to generate new pose for each particle by sampling from the probabilistic motion model of the pedestrian. For the This estimate is added to a temporary set of particles, along with the path Equation  are distOccupancy Grid-based FastSLAM decomposes the unknown environment into a finite set of cells where each cell holds the probability that the location in the environment enclosed by the cell is occupied by an object. A solution to estimate these probability is the binary Bayes filter. In binary filter, the posterior of grid is commonly implemented as a log odds ratio in order to avoid truncation problems that arise for probabilities close to 0 or 1. The odds of grid state is defined as the ratio of the probability of this grid being occupied is divided by the probability of its negate:t can be estimated by involving an inverse measurement model Thus, according to Bayes theory, the posterior of grid being occupied at time Here, Samples drawn from the proposal distribution are distributed according to Once the weight of particles is calculated, a new set of samples Equation .A key characteristic of the FastSLAM implementation is the sensing of the modality of the grid via an exteroceptive sensor. In order to minimize costs, decrease deployment effort, and remove the reliance on prior maps of the indoor surroundings, the basic idea of inertial pedestrian occupancy grid-based FastSLAM is constructing a virtual exteroceptive sensor, called ’visual range sensor’, using proprioceptive information, including the heading change and the step length estimation from the triple-node IMUs pedestrian navigation system, to track the pedestrian’s motion and build the layout of the environment.The major steps of the inertial pedestrian occupancy grid-based FastSLAM algorithm is illustrated in Once the pedestrian has moved one step, a new distribution of the pedestrian’s pose, the proposal distribution, can be calculated based on the state transition model and the pose distribution at the time u of inertial pedestrian occupancy grid-based FastSLAM. Then, the pedestrian’s pose can be achieved recursively as follows:t, t. That is, a new pose could be generated based on the state transition model of pedestrian Equation (m-th particle at time step The step length and the heading variation within one step constitute the control vector  follows:st=st−1+uEquation .In order to perceive the accessibility of grids in the environment, we construct a virtual range sensor using inertial information. This sensor can be seen as a laser range sensor with a very short sensing range. Meanwhile, compared to the laser sensor, which perceives the distance between the obstacle and the sensor, the virtual range sensor perceives the distance between the pedestrian and the accessible grid. The virtual distance sensor measures the distance from the grid where the pedestrian is currently located to the nearest accessible grid. Obviously, the grid which the pedestrian located is the accessible area. Thus, the sensor output is always zero.m-th particle goes through the i-th grid. i-th grid being passed through and is selected as 1 typically. i-th grid not being accessed and is selected equal to The occupancy grid mapping algorithm requires a marginalized inverse measurement model, Equation . This prIt can be seen from Equation  that p(θi-th grid is accessible for pedestrian and i-th grid is intransitable. The size of the grid is an important parameter in the occupancy grid mapping algorithm. In the laser range sensor-based mapping method, the laser range sensor pixel size is chosen as the grid edge length which allows the measurement of obstacles to be continuous and without overlap.The idea of the occupancy grid map is to divide the entire environmental space into a number of discrete grids In the proposed method, it is difficult to guarantee non-overlapping observations of the approachable grid due to the inconsistent step size of pedestrian movements. However, it is still possible to guarantee the continuity of the observation of the accessible grids by limiting the edge length of the grids to no less than the minimum step length. Excessively large grid edge lengths result in uncorrected errors in the position estimation of pedestrians moving within one grid. The grid edge length is therefore chosen as the maximum step length, which ensures continuity of observation of the approachable grid and allows the pedestrian positioning error to be corrected in a smaller number of steps.As opposed to the original occupancy grid update that estimates the probability distribution of obstacle position, the inertial occupancy grid update estimated the probability distribution of accessible space, such as the corridor and rooms in the indoor environment, shown as the blue grid in The inertial pedestrian occupancy grid-based FastSLAM implements the occupancy grid map update using Equation , where tm is updated using the formula for a FastSLAM weight update Equation (The weight associated with each particle Equation , which fD is too large.The particle set is resampled for importance only if To validate the application effect of the proposed inertial pedestrian occupancy grid-based FastSLAM, several different experiments were conducted on different testers. As illustrated in The first test was designed to verify whether the proposed step length estimation method was suitable enough for regular walking, running, and sprint motions. Four testers, including two males and females, were asked to walk, run, and sprint along a 55.8 m straight track at least 12 times for each motion. After calculating the step length estimation parameters for each tester, the cumulative distribution function of the absolute value of the distance estimation residual is shown in These results indicate that it is applicable to utilize the proposed method to estimate the step length both in walking, running, and sprint motions. Once the parameters are determined in the initialization procedure, for example, walking along a track in different motions, the step length can be estimated in real-time.In order to evaluate the occupancy grid-based FastSLAM for inertial pedestrian navigation algorithm, the indoor experiments were conducted in a rectangular area of the building in Nanjing University of Aeronautics and Astronautics. As shown in t is experienced, the expected position in the graph at time t will not change due to the subsequent optimization process. The reason we do this is that in most scenarios the user is most concerned with the accuracy of navigation at the moment. The expectation of posterior distribution of posture at all final moments after optimization has a smaller error than the above method, which may mislead the researchers’ evaluation of the navigation system.The tester was asked to go through some ground truths mentioned above; during the test, the tester recorded the number of steps taken before the moment through ground truths. The estimation of the position of these ground truths is shown in the black cross in the figure; meanwhile, the red circles indicate the true position of the ground truths. In order to eliminate the influence of the Monte Carlo particle sampling randomness to the location result, we have processed the above data 100 times, and the cumulative distribution of the RMS position error between the trajectory and the ground truths are shown in The second experimental scenario is a random walk in multiple rooms. The experimental route consisted of multiple turns, which posed a challenge for heading estimation. The tester was asked to walk around the multiple rooms randomly, and return to the starting point after every certain time. We used the starting point as ground truth to evaluate the positioning accuracy of the system. The parameter settings of the system remained the same as in the previous experiment.Similarly, in order to eliminate the influence of the randomness of Monte Carlo particle sampling on the localization results, we processed this data set 100 times, and the cumulative distribution function of the RMS position error is shown in The main purpose of this study is to develop a multi-IMU inertial pedestrian odometer integrating with the occupancy grid-base FastSLAM approach to improve the performance of the inertial pedestrian navigation system on drastic motions. The traditional PDR based inertial pedestrian navigation system has advantages including no additional infrastructure needed and no radio frequency interference but has disadvantages including the inaccurate step length estimation on drastic motions such as running and sprint, and unbounded error on heading estimation. We suggested a simple step detection and segment approach by detecting the zero-crossing of the pitch of thighs. Meanwhile, a step length estimation method for drastic motion based on the pdf of the waist-worn horizontal acceleration norm is proposed. These two methods together constitute an inertial pedestrian odometer adapted to drastic motions. The experiments on different testers on different speed motions indicate that the proposed inertial pedestrian odometer can provide the step length estimation with the same precision as the slow one when the pedestrian is moving fast. Moreover, we extended the occupancy grid-based FastSLAM algorithm to the inertial navigation system. By constructing a virtual range observer, the simultaneous positioning and mapping of pedestrians can be realized by relying on inertial odometer without any additional sensors. The experiment results indicate that the proposed inertial occupancy grid-based FastSLAM can accurately construct the map of the passable area, and the system can effectively restrain the course error divergence of pedestrian inertia odometer. This method can provide long-term and high-precision pedestrian positioning information without increasing the cost, which is of great significance for the practical application of pedestrian inertial navigation.The proposed method can improve the long-term accuracy of inertial pedestrian navigation system. However, there are still some limitations in this study. Firstly, the inertial occupancy grid-based FastSLAM utilizes the historic trajectory information to correct the positioning and heading error which means that the position and heading errors are corrected only when the pedestrian revisits the area they have already visited. Secondly, the determination of the initial position is not considered in this study. We focus on solving the issue of accumulative error rather than the initial position. Nevertheless, according to the basic principles of inertial navigation, the initial position error will be gradually coupled to the attitude estimation. On the one hand, it will lead to an increase of course estimation error; on the other hand, it will lead to inaccurate waist attitude estimation, which will affect the step length estimation, which limits the application of the system in larger areas.For the first limitation, we can use the historical trajectory of multiple people to correct map estimation and user location estimation by studying the method of multi-pedestrian collaborative SLAM. For the second limitation, the combination of GPS and the inertial system can be used to obtain the absolute position of pedestrians through the satellite navigation system before they enter the blocked area of the satellite.In this paper, we propose a triple-node IMU inertial pedestrian odometer which improves the accuracy of inertial pedestrian localization in dealing with drastic motions. Compared to the traditional PDR system, the proposed approach can provide the step length estimation with the same precision as normal walking when the pedestrian is running or sprinting. Moreover, we extended the occupancy grid-based FastSLAM, which is commonly used in a vision-based navigation system, to the inertial pedestrian navigation by constructing a virtual range sensor to precept the external environment. The combination of these two methods could estimate step length accurately and eliminate the accumulated positioning error of the inertial pedestrian odometer. Less than 5 m positioning accuracy in a single-story building of 2643.2


The early screening and diagnosis of tuberculosis plays an important role in the control and treatment of tuberculosis infections. In this paper, an integrated computer-aided system based on deep learning is proposed for the detection of multiple categories of tuberculosis lesions in chest radiographs. In this system, the fully convolutional neural network method is used to segment the lung area from the entire chest radiograph for pulmonary tuberculosis detection. Different from the previous analysis of the whole chest radiograph, we focus on the specific tuberculosis lesion areas for the analysis and propose the first multicategory tuberculosis lesion detection method. In it, a learning scalable pyramid structure is introduced into the Faster Region-based Convolutional Network (Faster RCNN), which effectively improves the detection of small-area lesions, mines indistinguishable samples during the training process, and uses reinforcement learning to reduce the detection of false-positive lesions. To compare our method with the current tuberculosis detection system, we propose a classification rule for whole chest X-rays using a multicategory tuberculosis lesion detection model and achieve good performance on two public datasets . Our proposed computer-aided system is superior to current systems that can be used to assist radiologists in diagnoses and public health providers in screening for tuberculosis in areas where tuberculosis is endemic. Tuberculosis is a communicable disease that is one of the top 10 causes of death worldwide. At least 10 million people were estimated to have contracted tuberculosis in 2018 , and spuThe most common method estimates whether CXRs contain tuberculosis by classifying the chest X-ray images. Binu Joykutty et al.  used adaThe next type of method detects and estimates whether a certain lesion in a CXR is tuberculosis. Shen et al.  adopted Recently, some studies based on deep learning models  have offThese studies are indispensable explorations for the establishment of X-ray-based tuberculosis computer-aided systems, but there is space for improvement in practical applications. We provided a tuberculosis dataset labelled by two professional radiologists, which covered almost all types of tuberculosis lesions, including exudation, nodules, calcification, miliary tuberculosis, encapsulated pleural effusion, and free pleural effusion. As shown in Therefore, this paper is different from the previous pulmonary tuberculosis analysis methods. Instead of using the whole image as a classification target, we focus on the specific tuberculosis area for analysis. We apply an effective lesion detection method to the image, which reduces the amount of calculations and also allows us to obtain more accurate analysis results. Meanwhile, in order to avoid the problem of manually extracting image features, a computer-aided system based on deep learning is proposed for the detection of multiple categories of tuberculosis lesions in radiographs. Our proposed system can effectively solve these challenges, and it includes a two-stage process for the accurate detection of tuberculosis.The first stage in the design of the chest radiographic computer-aided system is the correct segmentation of the lung region from a CXR . In compIn the second stage, this paper proposes a multicategory tuberculosis lesion detection scheme for the first time, which is different from the methods that classify the whole CXR image and the previous single-category symptom detection methods. Various object detection methods using deep learning have been reported in the computer vision literature –34. The Finally, in order to further illustrate the effectiveness of the proposed system, we propose a classification rule based on the tuberculosis lesion detection model. We compare the performance of the existing tuberculosis classification methods based on a convolutional neural network in a computer-aided system. We have achieved good performance on two public datasets , which eIn this study, we propose a computer-aided system based on deep learning for detecting multiple categories of tuberculosis lesions in radiographs. A schematic diagram of the proposed computer-aided system is illustrated in The JSRT dataset was gathered by the Japanese Society of Radiological Technology. It consists of 247 CXRs. 93 are normal and 154 are abnormal with diﬀerent TB manifestations. Each image in the dataset is stored in PNG format with 2048 × 2048 pixels and has a grey-scale depth of 12 bits. The lung masks of the images are provided by the same authors in another dataset named the SCR dataset . So, theThe Montgomery dataset was created by the U.S. National Library of Medicine in collaboration with the Department of Health and Human Services, Montgomery County, Maryland, USA. It contains 80 CXRs that have been classiﬁed as normal by doctors and 58 CXRs that have TB manifestations. The images are stored in PNG format with a pixel resolution of 4020 × 892 pixels with a grey-scale depth of 12 bits. The corresponding manual lung masks that were generated under the supervision of radiologists are also available. The Shenzhen dataset was collected in the Guangdong Hospital in Shenzhen, China. The dataset contains 662 frontal CRs. 336 of those have tuberculosis inflections, and 326 are normal. All image resolutions are approximately 3,000 × 3,000 pixels [The local dataset was collected from the First Affiliated Hospital of Xi'an Jiao Tong University (FAHXJU) in Shanxi, China. The FAHXJU dataset contained 2382 CXRs classified as normal and 2962 CXRs classified as containing tuberculosis by two professional physicians. The resolution of the images ranges from 858 × 1004 to 3480 × 4240 pixels. The local dataset was collected to detect multiple categories of tuberculosis signs. The most common seen signs of tuberculosis, including exudation, nodules, calcification, miliary tuberculosis, encapsulated pleural effusion, and free pleural effusion, were defined according to the glossary of terms for thoracic imaging of Fleischner Society . Since tLung segmentation is the primary task in the detection of multicategory tuberculosis lesions in computer-aided systems. In segmentation, it is necessary not only to classify pixels into foreground or background, but also to preserve their spatial information. The previous research has well solved the issues in the use of an encoder-decoder network. Therefore, we chose the fully convolutional neural network U-Net  for lungFinally, the trained segmentation network is used for lung segmentation on its own tuberculosis dataset. For the segmented data, there is a small amount of data containing some small holes, small unnecessary objects/points in the lung region or fused left and right lungs. We have postprocessed these data. In the entire computer-aided detection system, the main role of the segmentation stage is to obtain complete lung regions. Therefore, in image postprocessing, the method of image connected region detection is mainly used to detect whether the nonlung region is segmented in the segmented image. Because the nonlung regions in the segmented images are mostly other tissue images with small areas, we use the method of preserving the two largest areas obtained from the image connected area detection, and the small objects are removed by selecting the two foreground objects having the left and right lung regions. In this way, a segmented image of the final lung region is obtained, and then the original chest X-ray is cropped according to the size of the segmented lung region to acquire the data for tuberculosis detection analysis. This method reduces unnecessary tissue images in chest X-rays and avoids the problem of losing some context information due to segmentation. Therefore, the quality of the lung region image sent to the detection stage is not significantly degraded.This paper focuses on the specific tuberculosis area for analysis. Since the RCNN is computationally expensive and extremely slow, the Faster RCNN network is used as the main framework of the detection stage. The different lesion sizes pose a great challenge to the detection task, especially the detection of small lesions. It is difficult for the traditional detection framework to combine rich semantic information with detailed feature information. The current target detection network uses the feature pyramid network (FPN) structure to solve the multiscale problem, which is very helpful for detecting small objects. It adopts a backbone model that is usually used for image classification. Furthermore, it builds the feature pyramid by sequentially combining two adjacent layers in the backbone model, and the feature pyramid is built via top-down and lateral connections to produce high-resolution and semantically strong feature representations. The FPN is simple and effective, but it may not be the optimal architecture design. Therefore, a better learning scalable feature pyramid structure is used here for the detection of multicategory tuberculosis lesions in chest radiography.The new FPN architecture is customized using the Neural Architecture Search approach. First, the search space is constituted to cover all cross-scale connections to produce multiscale features. Multiple merging cells constitute the search space, where one merging cell fuses two feature connections from different feature layers to generate the feature output. The feature fusion method here uses nearest neighbour sampling or max pooling to adjust the input feature layer to the output resolution. The merged feature layer will always follow the ReLU, a 3 × 3 convolution, and a batch normalization layer. Then, it uses the reinforcement learning training controller to select the optimal FPN architecture in a given search space. Since the FPN cross-connection is not known, a recurrent neural network is used as a controller and trained using the Near-end Strategy Optimization algorithm to construct different connections. The final FPN structure is obtained via its controller's convergence, and its accuracy is the highest .As shown in The screening of tuberculosis diseases, especially large-scale screening, is different from daily clinical detection. To find more cases earlier, screening usually emphasizes that the screening technique is more sensitive; therefore, in the proposed computer-aided system, we adjusted the parameters to minimize the detection of false negative lesions, but this resulted in more false positives. This mainly occurs because ordinary lung infections are often similar to tuberculosis in imaging, which makes them difficult to distinguish and prone to misjudgements. Therefore, we added a false-positive tuberculosis lesion reduction module to mine the indistinguishable samples during the detection process. Based on the previous tuberculosis detection model, we send the normal chest X-ray collected by us to the best trained tuberculosis detection module and apply the detector to the normal chest radiograph. After the detection, a portion of the normal chest X-rays will be determined to contain labelled suspected lesion areas. These areas are similar to the appearance of tuberculosis in chest X-rays. Therefore, we apply normal labels for false positive samples. Then, we put them into the negative sample dataset and use reinforcement learning to train the detector again to reduce the detection of false positives.In this work, we use the JSRT  and the To train a better deep learning model, we enhance the data by improving the diversity of the training data. In the segmentation stage, we performed related experiments on the size of the input image. Therefore, we uniformly convert the images in the two public datasets , 35 to tFor the end-to-end training of the segmentation stage, the number of training epochs is set to 100, and the batch size is set to 8. We use the Adam optimizer to iteratively update the neural network weights based on the training data, the loss function selects the cross entropy, and a checkpoint is used in the training process to get the best model. In the detection stage, we used the stochastic gradient descent for back propagation and weight decay, and the number of training steps is set to 50k. Due to memory issues with the GPU, the batch size is set to 1. The learning rate is set to 0.00125 in the first 25k steps, 0.000125 from steps 25k to 40k, and it drops to 0.0000125 after 40k steps. The weight is set to 0.0001. The optimizer is the SGD, the MOMENTUM is set to 0.9, and Nesterov gradient acceleration is used. A mass is considered to be detected if the overlap ratio between the bounding box of the candidate region and ground truth is 0.3, which is similar to other works in the field –34. We uTo evaluate the performance of the segmentation method, we use the two common evaluation indicators of the Dice coefficient and Intersection over Union (IoU) to evaluate the medical image segmentation performance of the segmentation stage in our proposed system. The Dice indicator is the ratio of the intersecting area to the total area between the ground truth images (G) and the segmented images (O). It can be computed by using equation , and theIn the segmentation stage, considering the capacity of the GPU memory, we compared the segmentation experiments with different resolution images as input. As shown in In the segmentation stage, as shown in We use the Mean Average Precision (MAP) that is commonly used to assess multiobjective detection models to evaluate our model in this paper. First, we calculated the IoU of the prediction box and ground truth. We specified that the prediction box whose intersection ratio exceeded 0.3 was the correct prediction. Then, we calculate the number of correct detections for each category in each graph, divide that by the total number of targets of this category in the graph, and obtain the accuracy of this category in the graph, as shown in the following equation: Then, the accuracy of each category is calculated for all test images, and their average is calculated to obtain the average accuracy of this class, as shown in the following equation:Finally, we integrate all the classes in our dataset and average the average precision of each class to get the mean average precision, as shown in the following equation:We compared the detection results of the Faster RCNN model and the Faster RCNN model with the added original feature pyramid structure. The following To compare our method with existing tuberculosis computer-aided systems, a classification rule based on the tuberculosis detection model is proposed to verify the effectiveness of the proposed system. We tested the proposed strategy for classifying chest X-rays using the tuberculosis detection model on the local collected dataset and the two publicly available tuberculosis datasets , MontgomThe classification results are also evaluated using the receiver operating characteristic curve (ROC), and we also obtain the area under the ROC curve (AUC). In We select the detection model trained by the NAS-FPN network  as the bThis paper proposes a computer-aided system, which is based on deep learning, for the detection of multiple categories of tuberculosis lesions in radiographs. For tuberculosis, the proposed computer-aided system can automatically segment the lung area and then detect the multiple categories of tuberculosis lesions in the segmented lung X-rays to achieve the auxiliary analysis and diagnosis of tuberculosis. Therefore, it can play a vital role in forming the second opinion of a radiologist. In the segmentation stage, the system achieves higher segmentation performance on the two public datasets , 35 than