Supporting the elderly to maintain their independence, safety, and well-being through Active Assisted Living (AAL) technologies, is gaining increasing momentum. Recently, Non-intrusive Load Monitoring (NILM) approaches have become the focus of these technologies due to their non-intrusiveness and reduced price. Whilst some research has been carried out in this respect; it still is challenging to design systems considering the heterogeneity and complexity of daily routines. Furthermore, scholars gave little attention to evaluating recent deep NILM models in AAL applications. We suggest a new interactive framework for activity monitoring based on custom user-profiles and deep NILM models to address these gaps. During evaluation, we consider four different deep NILM models. The proposed contribution is further assessed on two households from the REFIT dataset for a period of one year, including the influence of NILM on activity monitoring. To the best of our knowledge, the current study is the first to quantify the error propagated by a NILM model on the performance of an AAL solution. The results achieved are promising, particularly when considering the UNET-NILM model, a multi-task convolutional neural network for load disaggregation, that revealed a deterioration of only 10% in the f1-measure of the framework’s overall performance. The growing rate of the elderly population has been a central topic in the last decade and will remain relevant for the coming decades; the lifespan of people has become longer than it has ever been. Such development will undoubtedly impact economic and societal systems, including healthcare services. In Europe, for example, the ratio between people of working age and those aged above 65 years was 4:1 in 2001 . ProfessA new set of technologies, titled Active and Assisted Living (AAL) technologies, have emerged in response to these challenges. They refer to all products and services that are designed to support individuals in retaining or enhancing their independence and well-being ,6. The mExisting AAL approaches rely on deploying several sensors inside the house to enable ageing-in-place. Nevertheless, adopting this approach appears intrusive and, in some cases, expensive for older adults. The intrusiveness is generally related to solutions that consider sensors intervening in the daily activities or violating privacy constraints. The subjects are thus less accepting toward these solutions ,7. On thA promising sub-field of AAL relies on the indirect monitoring of daily activities through energy data generated by the explicit usage of electric devices . These dA primary challenge in monitoring daily routines (and identifying significant deviations) is to design systems considering the heterogeneity and complexity of daily routines. Moreover, scholars gave little attention to evaluating state-of-the-art NILM models in AAL applications. We suggest a new framework for activity monitoring based on custom user-profiles and deep NILM models to address these gaps. Furthermore, we propose an interactive framework by exchanging feedback with external agents such as health care professionals or family members. The key objective of our work is to provide them with a tool that can take over basic monitoring means which currently require manual efforts . This is particularly relevant for the case of rural areas and older relatives living alone. When the technology works appropriately, phone calls and personal visits can focus on social and relationship aspects rather than daily activities monitoring. The contributions of the presented work fall in two main points: (1) The proposition of a new activity monitoring framework based on the usage time of hand-operated appliances inferred from energy data, and (2) the discussion of two case studies of houses occupied by older adults from the REFIT dataset where we evaluate the whole proposed pipeline including NILM approaches and their influence on activity monitoring.The remainder of the paper proceeds as follows: The set of habits and routines carried out by individuals for self-caring is formally referred to as Activities of Daily Living (ADLs) and Instrumental Activities of Daily Living (IADLs). ADLs include basic self-care tasks summarised in six categories: eating, bathing, dressing, toileting, mobility, and grooming . On the Kettle, for example, was identified as a good candidate in several works [The current manuscript focuses on a subset of IADLs highly correlated with the use of hand-operated electrical appliances  as illusal works  as it isal works . Moreoveal works . In summTwo viable strategies are recognised to quantify the power usage of appliances present inside a household. The first strategy, referred to as Intrusive Load Monitoring (ILM), requires the connection of each appliance to a metering point. It follows from this definition that this alternative is expensive and complex to maintain. Furthermore, subjects could consider it to be intrusive to their daily activities. The second strategy, referred to as Non-Intrusive Load Monitoring (NILM) or load disaggregation, follows the approach of decomposing the total power consumption of the household into the different contributions of each appliance  using onZhang et al.  obtainedAlthough previously presented contributions showed promising results, a main missing point is the availability of annotated data to evaluate these approaches. Generally, the authors propose to evaluate in labs, simulating real homes see . The onlActive and Assisted Living (formerly Ambient Assisted Living) (AAL) aims to help the ageing population maintain autonomy and well-being. Achieving this goal is generally linked to tracking and monitoring daily routines and activities that is approached using two different techniques: (1) Direct techniques based on Wireless Sensor Networks (WSNs) ranging from lightweight systems to more sophisticated systems incorporating cameras and motion sensors , and (2)The availability of smart meters at a large scale and considerable low cost makes NILM an appealing solution for diverse tasks, such as energy-saving , but alsModelling and tracking human behaviour is a complex process that includes several aspects. In the scope of our contribution, we consider three main aspects. First, individuals have different routines and lifestyles, which highlights the importance of defining a custom daily profile, for each user, based on a pre-recorded historical energy consumption. This profile is the base for scoring a new day and deciding its normality. Demonstrative examples are different sleeping, eating, and waking up times. Second, human routines are subject to change. The profile must thus be adaptable to this dynamic character of human behaviour. In this regard, we differentiate between variations and deviations in the behaviour . Variatihttps://www.openhab.org/, accessed on 30 December 2021) platform, for example, as it already contains a load disaggregation module developed in a previous work of the authors [The overall architecture of the proposed framework is illustrated in  authors . Moreove authors ), providThe load disaggregation module is responsible for identifying features relative to the activation of different appliances. As an input, it uses the aggregate power of the whole household obtained with a single metering point, the smart electrical meter. The consumption of individual appliances is inferred afterward using load disaggregation techniques on an hourly basis. For active appliances, a summary of the current activation is extracted containing both contextual information  and operational information . The operational information is related to the operational characteristics of the appliances. The goal from using these pieces of information is to confirm that the device used for monitoring an activity is not defective. For demonstration purposes, we only use the temporal data as contextual information in the remainder of the manuscript represented by: the start time of use, duration of use, the day of the week, and day of the month.The activity monitoring module relies on the features created by the load disaggregation module to generate recurrent reports and anomalies that are transferred to an external agent to intervene whenever required. In regard to the first considered aspect , we suggest a two-phase functional mode for this module. The first phase is an observation phase in which the framework only enables module one and uses its outputs to build a custom consumption profile of the occupant. After this period, the framework becomes fully operational and can detect anomalies automatically.The observation phase is based on passive monitoring where only daily reports are generated, and activity patterns are saved. For each day during this period, the daily routines are recorded for a set of pre-defined activities considering two levels. The first level is the activity level that is modeled using activity curves  taking iJSD) calculated as per Equation , the curve of each activity is calculated for the current day. At the end of the day, these curves are compared to the database of distributions recorded during the observation phase. The comparison measure used is the Jensen–Shannon Divergence  to decide about the normality of the current day. The distance between each observed day and the rest of the data is calculated to estimate the distribution of the distance in the case of the normal days. The IQR is calculated as the difference between the third Appliance anomaly detection is performed only in days that were classified as anomalous see  to proviAnomalous days result in notifying an external agent for further decision-making. The communication with the external agent is performed in an asynchronous mode. The notifications are sent to the external agent who has the choice to provide feedback or not to the framework. The framework continues thus to be functional even for days where the feedback was not received. In this case, it considers that the reports provided are correct and updates its historical data accordingly. This update allows the framework to consider recent patterns when evaluating a new day in order to take into consideration seasonal change in the routines. In the opposite case , if the agent finds the anomaly irrelevant, they can forward their decision to the framework. For interpretable feedback, we propose four classes of notifications: (1) Abnormal usage time which is related to an abnormal activation of an appliance during the day, (2) usage during the night, and (3) absence of activity during a day. Providing interpretable information about the anomalies would help to guide the external agent in establishing diagnosis.i operates on two levels. First, the self-similarity matrices are updated by replacing the line i and row i with distance measures related to the current day. Second, the activity curve of the day i is tagged as having feedback and being replaced by the current day distribution.The feedback provided by the external agent is integrated in the framework through an update of the database of historical consumption. The new activations are added to the existing database of historical events using a forgetting (vanishing) factor that favours the replacement of old and unlabelled entries. The replacement procedure is straightforward. It takes into consideration two main metrics: (1) how old the information is, (2) the distance to the rest of the historical data, and (3) external agent feedback. If all the days included in the observation database have feedback, the framework will choose the oldest most divergent day to be replaced. The replacement of a day Aiming to evaluate the proposed framework in real scenarios, we present two case studies of the kettle’s usage relative to two different houses from a publicly available dataset. The two presented cases exhibited different usage patterns and were chosen to evaluate the proposed framework considering different routines. The kettle is used as a representative appliance for the cooking activity. The choice of this appliance is motivated by related literature arguing that its frequent daily usage is a good indicator of habits’ deviations ,45,46. ITo evaluate our contribution, we suggest an evaluation procedure as described per First, we evaluate several disaggregation algorithms to identify the best-performing model in the case of the kettle. For this purpose, we consider six different disaggregation approaches: Combinatorial Optimisation (CO), Factorial Hidden Markov Models (FHMM), and four different deep models. The four deep models consist of two well-known NILM baselines, the Seq2Seq and Seq2Point models, and two recent models, the temporal-pooling model  as well MAE), the f1-measure, the recall, and the precision, as demonstrated by the formulas below where TP represent the correctly identified ON states, FN represent undetected ON states, and FP represent the overestimated ON states:During training, the data is split into 85% for training and 15% for validation. During testing, we provide special attention to classification performance metrics as ON events and usage time are more relevant in the case of activity monitoring. To construct the confusion matrix and derive the operational states from the power consumption, a threshold of 500 watts was used. All models were implemented using the PyTorch framework in a compatible format with NILMtk API . We consThe first phase of the evaluation generates a set of pre-trained models that can be directly used to identify the kettle’s power consumption. As our purpose is to simulate a real scenario, the second phase of evaluation, which consists of evaluating the activity monitoring module, relies on the predictions generated by the best performing model during the first phase. Furthermore, due to the non-availability of annotated data, the authors suggest pre-processing the data extracted from both houses to generate an extended version of the data for both buildings, including labels relative to abnormal usage of the kettle. Three classes of the anomalous days are considered: (1) Divergent usage that corresponds to activations occurring at a non-usual time for the householder, (2) usage during the night, and (3) absence of usage during a day. When not available in the data, synthetic daily consumption of a class is inserted with a rate of 10% of the considered period’s length. Divergent usage patterns are generated by selecting a random day from the real dataset and altering the usage hour. On the other hand, the usage during the night is generated by inserting a random number (<3) of activations at random hours (0–4 a.m.) during the night.The aggregate power of the house is adjusted accordingly to the new generated consumption of the kettle resulting in an energy dataset annotated with abnormal usage. This new data is used as input to evaluate the activity monitoring module. From the first evaluation phase, the pre-trained model is used to identify the consumption of the kettle during the monitoring period. Before evaluating the activity monitoring, the authors argue that it is necessary to re-evaluate the disaggregation error to assess if the abnormal activations cause a deterioration in the disaggregation error. The evaluation of the activity monitoring module is then performed using both the actual consumption of the kettle and the predicted consumption obtained with a NILM model. The goal from using both data sources is: (1) To evaluate the performance of the monitoring module, and (2) to assess the influence of the disaggregation error on the monitoring module.https://reshare.ukdataservice.ac.uk/852367/11/Documentation.zip, accessed on 30 December 2021), illustrated in All reported results during the experimental setup conducted in the current study were obtained using the REFIT  dataset.To provide a more concrete understanding of the daily routines in house 4, On the other hand, As seen from In the case of the unseen scenario (house 11), a deterioration in the performance can be noticed for all models, mainly caused by an increased number of undetected ON events as recorded per the recall. A possible justification can be related to the different operational characteristics of the kettle in house 11. Remarkably, the Temp-Pooling model demonstrated a significant draw in the performance with an MAE multiplied by a factor of 2.7 and a difference of 23% in the f1-measure. It could be the case that this model over-fitted the data from house 4 and cannot generalise to data from unseen houses. The three remaining deep models yielded acceptable results with a minimum f1-measure of 71%. The UNET model also demonstrated the best trade-off considering both the MAE and f1-measure.In summary, NILM approaches provided acceptable results. Deep models demonstrated outstanding improvements in comparison with classical approaches. In particular, the UNET model yielded good performance even on the unseen scenario where it yielded the lowest deterioration in the f1-measure. Thus, we argue that this model is the most suitable for activity monitoring. Therefore, only the UNET model will be considered during the rest of the experimental setup, where the pre-trained model will directly be used to generate the predictions for the activity monitoring module.The current section presents the results of the monitoring module considering the data from houses 4 and 11 as described in The results obtained from the activity monitoring module relative to the identification of anomalous days, considering both the actual consumption and the consumption generated by the UNET model, are illustrated in In the case of house 4, both input sources, the UNET predictions, and the real consumption yielded acceptable results with a minimum f1-measure of 67%. Both data sources provided precision values higher than the recall values. The previous observations imply that in both cases, the number of ordinary days identified as anomalous (FP) is greater than the number of undetected anomalous days (FN). A viable justification for this finding can be that the dataset already contained anomalous patterns that were not considered during pre-processing. When comparing the results generated by both inputs, we record a difference of 10% in the f1-measure. A closer look at the precision and recall values reveals that both data sources have approximate precision values but a difference of 15% in the recall values. The previous observation is directly linked to the results obtained in As expected, the activity monitoring module could not detect the days labeled as anomalous (mainly days with usage during the night) during the pre-processing, where it yielded a very low f1-measure with both input types. These results are due to the extensive usage of the kettle that was labelled during pre-processing as anomalous but considered normal by the proposed framework as it was part of the observed data. The case of this house can be viewed from two different perspectives. The first is the case where the night usage is considered anomalous. In this case, if the observed data is not validated, the proposed framework will fail to identify what is hypothesised to be abnormal . This finding highlights a central failing point for our contribution where it shows its sensitivity to the data recorded during the observation phase that needs to be established in a very controlled manner and under the validation of professionals. The second perspective would be the case where the kettle (or other hand-operated appliances) usage during the night is just part of the subject’s routines and is far from being an indicator of a sleep disorder. This would mean that what is anomalous for one individual could be part of a routine for another individual. This finding stresses the importance of the observation phase to define custom anomalies. In this case, the definition of abnormal behavior remains relative.The manuscript at hand proposed a new activity monitoring framework using NILM and suggested an evaluation protocol that considers different modules of the framework. Accordingly, the evaluation results, on two case studies from a publicly available dataset, were presented to simulate a realistic scenario.The evaluation of the first module relative to the load disaggregation using both classical and deep approaches revealed that NILM models provide good results in identifying the kettle’s activations and power consumption. Notably, deep models provided the best performance for this module. Compared to classical models, deep baselines demonstrated their competitiveness, confirming findings from recent literature . Even wiThe evaluation of the second component relative to the activity monitoring also yielded acceptable performance. The activity curves demonstrated good representativeness of the overall activity patterns. Nevertheless, we argue that they are more suitable for activities that rely on frequently used appliances during a single day. We stress that it could be the case that they are less effective in activities inferred based on less frequently used appliances, such as the washing machine. A more elaborated version of the proposed framework would rely on hierarchical activity curves, including other temporal dimensions such as the day of the week. Adapting established methods for activity monitoring or combining them demonstrated good results but remains subject to future improvements. The activity monitoring of older adults remains a sensitive health care service that requires more robust and elaborated models to help mitigate the number of errors. Moreover, using two case studies demonstrated that the definition of anomalous behaviour remains difficult and dependent on the context.On the other hand, the evaluation of the activity monitoring module considering both real data and NILM predictions allowed to assess the propagated error that was estimated to be around 10% for the f1-measure. This difference shows encouraging results for using deep NILM models in activity monitoring and demonstrates that more elaborated models (both NILM and monitoring models) would help to mitigate this error. The presented case studies also highlighted the sensitivity of this module to the data recorded during the observation phase.The proposed framework showed potential for benefiting from the new electrical grid in active and assisted living. To the best of the authors’ knowledge, this is the first application of deep NILM models in activity monitoring with an explicit assessment of their effect on the activity monitoring algorithms. Nevertheless, a source of limitation in the proposed solution is its high dependency on the data recorded during the observation phase that must represent the average occupant’s routine. Moreover, the scope of this study was limited in terms of the set of monitored activities. The authors intend to consider a more extensive set of activities with more appliances in future work. Furthermore, we argue that gaining a deeper understanding of the challenges imposed by the problem can only be enabled with more established datasets. More particularly, a significant obstacle in evaluating the applicability of NILM in activity monitoring is the non-availability of annotated data. The authors overcame this problem by introducing abnormal synthetic usage and annotating available suspicious usage time. The generated data allowed the evaluation of the proposed framework on data from two different houses. However, this aspect remains the main limitation for the evaluation protocol adopted in the experimental setup and a major obstacle towards the generalizability of these results. We urge the need to establish more elaborated datasets to develop more complex scenarios and more extensive evaluations of activity monitoring approaches based on NILM.Adopting Non-intrusive Load Monitoring for daily activity monitoring is a promising approach. The current manuscript proposed a new framework for activity monitoring based on this paradigm. The evaluation of the whole modules of the framework revealed that deep NILM models are a good fit for the problem with an influence of –10% in the f1-measure of the activity monitoring algorithm. Nonetheless, the generalisability of these results is subject to certain limitations. For instance, the data used during the evaluation remains limited regarding the number of houses and appliances. Further research may focus on providing larger energy datasets of older adults, allowing deeper evaluations. Despite its limitations, the current study is the first to suggest using deep NILM models for activity monitoring and to measure the error propagated from the NILM algorithm on activity monitoring. In this regard, the presented work can be considered as a proof of concept that the basic algorithms are appropriate. The plan for future activities is to transfer the method from artificial environments into real-world settings, similar to the ones described in , which i


In recent years, entity relation extraction has been a critical technique to help people analyze complex structured text data. However, there is no advanced research in food health and safety to help people analyze the complex concepts between food and human health and their relationships. This paper proposes an entity relation extraction method FHER for the few-shot learning in the food health and safety domain. For few-shot learning in the food health and safety domain, we propose three methods that effectively improve the performance of entity relationship extraction. The three methods are applied to the self-built data sets FH and MHD. The experimental results show that the method can effectively extract domain-related entities and their relations in a small sample size environment. Food is inextricably linked to human health, particularly the nutritional components that can significantly improve health. For instance, increased spermidine intake protects against cancer, metabolic disease, heart disease, and neurodegeneration . A higheWith the proliferation of available human health data, humans use predictive or classification models to extract useful information, which helps people enjoy better health protection. Typical of this available human health data is the electronic health record (HER), the largest source of medical text data. One of the critical points in analyzing these unstructured textual data is to extract vital medical concepts, so many named entity recognition methods and entity relation extraction methods have emerged. Wan et al. proposed an ElMo-ET-CRF model approach for extracting medical named entities from Chinese electronic medical records (CEMR) using dynamic context-dependent ElMo character embedding to merge more lexical, syntactic, and semantic information that alleviate the long context-dependency problem . Luo et It is possible to help explicitly store the relationships between complex medical concepts for use in subsequent tasks by extracting them. Similarly, for the food health and safety domain, food regulators sample food products on the market and examine the ingredients and form structured data. However, in addition to structured data, more unstructured textual data goes beyond the act of food sampling and inspection to include announcements of food safety events, news reports, and knowledge about food and human health. Cenikj et al. proposed a method to detect the relationship between food and disease entities from text. They explored the feasibility of migration learning using a pretrained model based on BERT and achieved good results with few-shot learning . PopovskHowever, the food health and safety field does not have many available resources like the biomedical field, so few-shot learning becomes an effective method to improve information extraction. Qu et al. proposed a Bayesian meta-learning method for relation extraction in few-shot learning environment, applying graphical neural networks to global relation graphs to improve accuracy . Sainz eThe essential technique for extracting valid information in the food health and safety domain is entity relation extraction. This task requires the model to identify entities and relations in a sentence correctly and combine them correctly. The complexity of the task and the small sample size together contribute to the difficulty of the entity relation extraction task with few-shot learning. For supervised deep learning models, small sample size can lead to an underfitting of the model. In other words, there is not enough information to train a valid model. In addition, the model tends to converge at a local optimum, leading to poorer practical results of the model.https://en.wikipedia.org/wiki/Radical_(Chinese_characters)) and construct semantic input at the radical level. As pictographs, radicals are part of the structure of Chinese characters. The radicals have specific meanings; in other words, they represent part of the semantic information of the constituent characters. Therefore, constructing semantic input at the level of radicals can enrich the semantic information of the input text very well [To solve the underfitting problem, we propose two methods to enrich the semantic features of the input text. The first is to disassemble Chinese characters into radicals .To reduce the influence of noise in the text, we propose a text denoising method for domain entity relation extraction to convert the text denoising task into a sequence prediction task, which effectively reduces the correct range of entity relation extraction.To address the lack of semantic information due to the small sample size under the few-shot learning, we propose constructing the input at the part-head level to enrich the semantic information.We propose the fused character and word method to improve the ambiguity problem caused by errors in Chinese word separation boundaries.Our contribution can be summarized as follows:There are two approaches in current research . TraditiDue to the scarcity of textual data in the food health and safety domain, few-shot learning can effectively improve the effectiveness of entity relation extraction. Most of the current research on small-sample learning has focused on relationship extraction tasks. For relation extraction in a few-shot learning environment, Qu et al. propose a Bayesian meta-learning method that applies graphical neural networks to global relation graphs to improve accuracy . Sainz eFor the entity relation extraction task in food health and safety, we propose a joint entity relation extraction method FHER with noise removal and feature enhancement, which mainly addresses how to perform effective entity relation extraction in a few-shot learning environment with high noise and low data volume. Our model adopts a cascading pointer annotation approach  to mitigThe FHER method divides the whole entity relation extraction task into five parts: the first is a text noise removal model, the second is the construction of a radical level input, the third is the fusion of character and word features, the fourth is the prediction of possible subjects in a sentence, and the last is the prediction of relations and corresponding objects.Hc. The prediction sequence Ps is obtained after passing through the text noise removal model. The position vector E can be obtained through the binary function. Then, the input Inr of sentence In at the radical level is constructed, and the vector Hr is obtained after passing through the BERT encoder layer. The vectors Hc and Hr and the position vector E are fed into the fusion function to obtain the vector Hcw after fusing the characters and words. Using the position vector S and the fusion vector Hcw, we can obtain the possible object positions corresponding to each relation and thus the combination of entity relations present in the sentence.x, all possible triples T{s, r, o} are extracted, with s representing the subject in sentence x, o representing the object in sentence x, and r representing the relationship between the subject and the object. For sentence x, the probability of all possible triples T that it contains is as follows:We define the entity relation extraction task in the food health and safety domain: unstructured input text, output extracted human health-related concepts and their relationships, and further form food- and health-related knowledge triad. Formally, for a given sentence s ∈ T denotes the subject appearing in the triplet T, T|s denotes the triplet T containing the subject,  ∈ T|s denotes the combination of objects and relations appearing in the triplet containing the subject, and R is the set of relations. For a given training set D, the probability that all sentences may contain the triplet T is as follows:where, according to the chain rule, s, r, o) in the data set D.The goal of the entity relation extraction task in the food health and safety domain is to find all possible triples . The BERT model consists of a multilayer bidirectional transformer consisting of an encoder that learns the valid information in the context very well . We use We summarized the possible relationships between concepts related to food health and safety into 12 kinds by reading many food health and safety information. We transformed the text noise removal task into a sequence prediction model. To capture the sentence context information, we used the BiLSTM layer for further feature extraction. Then, considering the dependencies between the labels, we use the undirected probabilistic statistical graphical model CRF layer to handle the exact sequence labelling problem. Each character is assigned a BIO tag . At the same time, we modified this tagging method to predict the possible types of entities see .Hc of the sentence, which is then fed to the BiLSTM layer to obtain the intermediate vector representation HBi as follows:With the BERT encoder, we obtain the vector representation Y, the model is trained to obtain the prediction sequence result Ps as follows:For the input sequence In and the corresponding predicted label sequence As the smallest semantic unit, radicals themselves have particular semantic meanings. Chinese characters are usually composed of smaller primary radicals, which are the most basic units that constitute the meaning of Chinese characters. In essence, this radical semantic information helps make characters with similar radical sequences (writing order) close to each other in the vector space, so it can be used to enrich the semantic information of the word vector and enhance the model effect. As shown in r is generated based on the radical decomposition, and for each character wri in In, the radical is added to Inr if there is a radical, or directly to Inr if it is a unique word. Finally, we get the input Inr at the radical level as follows:Formally, for the input sentence In, the corresponding radical input InHr corresponding to the input Inr can be obtained through the BERT encoder layer as follows:Similarly, the vector representation Ps obtained from the text noise removal model is converted into a position vector E by a binary function I(·). I(·) will mark the food health-related tag positions as 1 and the irrelevant tag positions as 0.The sequence prediction result Hc and Hr and the position vector E are input to the fusion function G(·). The function G(·) multiplies the position vectors E with the input vectors and concatenates them. At last, the vector Hcr is calculated as follows:The input vectors Hw. The difference is that the BERT pretraining model used in the previous section is to segment the text into Chinese characters before training, while the BERT pretraining model used here is first to segment the text into Chinese words before training. The length of the input sentence after Chinese word separation is N.As in Hc and the word-based vector representation Hw.Because of the difference between Chinese and English word construction methods, different characters will be combined into words in Chinese word construction. Then the generated words will generate new meanings. So, in the food health and safety domain, especially in few-shot learning, we need to use as much prior knowledge as possible to help us improve the recognition, so we fuse the character-based vector representation E obtained from the text noise removal model with Hw to obtain the vector Hwe. Then we concatenate Hcr to Hw according to the corresponding position to obtain the fused vector Hcw (see equation (K(·) function expands the vector Hcr to the same length as Hw. For example, the vector for the word “食物” is A, and the vectors for the characters “食” and “物” are B and C, respectively. Then, concatenate vectors A and B, and concatenate vectors A and C.We multiply the position vector equation ), where s, any relation related to s (relations in T) corresponds to the corresponding object o in the sentence, while all other relations necessarily have no corresponding object in the sentence, that is, the set of corresponding objects is the empty set. So we first predict the possible positions of the subject in the sentence.After combining characters and words, we can predict the location in the sentence where the subject may be. Because the entity relation extraction task itself suffers from the problem of overlapping entity relations, according to the view proposed by Wei et al. : for a gFor a given input, we predict the location of potential subjects. Two binary classifiers form a subject marker, which marks potential subjects' starting and end positions. The subject marker is expressed as follows:Piheadsub denotes the probability that each position in the input vector may be the start position of the subject and Pitailsub in denotes the probability that each position in the input vector may be the end position of the subject. If this probability is greater than the threshold τ set in advance, it is for the position is a candidate for the start or end of the subject, marked as 1, and the rest is marked as 0. Khead and Ktail are learnable parameters, and bhead and btail are deviation parameters.where We obtain all possible sets of entity relations in the input sentence by object identification of each subject to all relations. After the subject tagger, we obtain the candidate subject position information, incorporated into the fusion vector as follows:Similarly, the object tagger predicts the potential object location, consisting of two binary classifiers. The object tagger is expressed as follows:Piheadobj in equation . Twelve types of food-health-related relationships are defined, and then “OTHER” relationship types are defined to represent less relevant relationships to food health in the sample data. The total number of valid annotated sentences in the FH data set is 1,420. This paper divides the FH data set into a training set and a test set. http://39.96.33.199:9009/FFH_dataset.zip) used for entity relation extraction experiments.To improve the performance of the model, we introduced more data samples from the public domain into the data set FH to form the incremental data set FH++, which is expanded from 1,420 sentences to 11,327 sentences to get better results in the text noise removal model. For the entity relation extraction task, we remove the “Other” relationships from the FH data set to form the FFH  data set (P), recall (R), and F1 value, to measure the experimental results from different aspects. Their calculation criteria are as follows: in the formula, TPi indicates that the model correctly predicts the number of relationships present in given sentence Ini, FPi indicates that the model incorrectly predicts the number of relations that do not exist in given sentence Ini, and FNi denotes the number of relations that the model failed to predict for given sentence Ini contained in the sentence.For the evaluation metrics used in the experiments, we used three commonly used evaluation metrics, precision  published by Cui et al.  with itsThe parameters used for the text noise removal model, construction of features at the part-head level, the fusion of character and word features, and entity relation extraction experiments are listed in F1 values for the text noise removal experiments on FH and the incremental data set FH++. The model can obtain better convergence and accuracy when training with the incremental data set. The positive and negative values in the experimental results shown in this paper are confidence intervals calculated by combining multiple results at a confidence level of 95%.The following are the experimental results of text noise removal, and the model is trained iteratively on the data set FH++. F1 values. In F1 for a given sentence containing one to five relationships. The experimental results show that the extraction effectiveness of the model decreases as the number of relations in the input sentences increases, which is related to the imbalance of the data itself, with most of the samples in the data set having less than three relations. However, the overall recognition effect is still good.For the entity relation extraction task, F1 value on the FFH data set and achieves good extraction results in few-shot learning.The FHER method achieves 91.32% accuracy, 82.21% recall, and 86.52% N represents the number of relations in the sentence, and the value in the table is the precision values for the entity relation extraction task. The experimental results show a significant decrease in recognition precision after the denoising model for a sentence containing multiple relations because the model identifies group or entity relations that we do not want. Therefore, denoising the input samples in few-shot learning is an effective method.We did ablation experiments on the proposed denoising model to verify the effectiveness of few-shot learning for denoising. In Similarly, we conducted ablation experiments to add prefix features to enhance and fuse character and word features. At last, we test the generalization performance of the method on the data set MDH. The experimental results show that constructing the input at the radical level and fusing characters and words can achieve good results even with smaller data set volumes. http://www.gov.cn/gzdt/att/att/site1/20130819/tygfhzb.pdf). These 3,500 characters can be combined in vast amounts of meaningful words, making the vector space generated after representation learning of Chinese through language models more complex. In contrast, the radicals, which are the main constituents of Chinese characters, can be grouped into 201 radicals for 20,902 Chinese characters (http://www.moe.gov.cn/downloadvideo/yuxinsi/15hanzibushou.zip). Therefore, radicals reduce the complexity of the Chinese vector space.We discuss the degree of semantic information in the text after its transformation into a radical. There are as many as 3,500 commonly used Chinese characters (For the entity groups “olive oil” and “unsaturated fatty acid” and their radical representations, we used the Chinese pretrained language model BERT-wmm to calculate their similarity based on the distance see . We collThis paper proposes a method for entity relation extraction in food health and safety. Three methods are proposed for entity relation extraction under small sample learning to improve the entity relation extraction effect. Firstly, we propose removing text noise according to domain specialization. The experimental results show that removing entity relations in the text unrelated to the domain can significantly improve the effect of entity relation extraction. Then, we construct text features at the radical level by disassembling individual Chinese characters from the characters' structure. The experimental results show that the input at the radical level has low complexity and contains rich semantic features. Finally, we fuse the input text's character vectors and word vectors from the Chinese character constructions. The experimental results show that fusing characters and words can effectively enrich the semantic information in the input vectors. The above method has a good extraction effect in few-shot learning. In the method FHER, the effect of the entity relationship extraction task is subject to a threshold, which leads to an unstable performance of the model. In the future, we will further explore the influence of noise in the text on the degree of deep learning models, the semantic relationships, and differences between the three different levels of Chinese radicals, characters, and words. Moreover, we attempt to transfer the proposed method to other application fields such as agricultural image recognition, greenhouse environmental time-series prediction, and food safety risk assessment –31.


Functional Near-Infrared Spectroscopy (fNIRS) is a wearable optical spectroscopy system originally developed for continuous and non-invasive monitoring of brain function by measuring blood oxygen concentration. Recent advancements in brain–computer interfacing allow us to control the neuron function of the brain by combining it with fNIRS to regulate cognitive function. In this review manuscript, we provide information regarding current advancement in fNIRS and how it provides advantages in developing brain–computer interfacing to enable neuron function. We also briefly discuss about how we can use this technology for further applications. A brain–computer interface (BCI) is a system that acquires signals from the brain, translates the signals, and outputs to devices in order to enact a desired action . A BCI sFunctional Near-Infrared Spectroscopy (fNIRS) is an optical imaging technique in which the emitted light in the brain undergoes attenuation due to absorption and scattering. It utilizes the general transparency property of the bones and skin to gain access to the tissues that are being monitored. While the absorbed light contributes to the inside of the medium where it is absorbed, the detector measures the non-absorbed component of the scattered light . As a reThe concept of fNIRS relies on the fact that certain wavelengths of light can penetrate the skin and skull  3]. In . In 3]. https://www.artinis.com/theory-of-nirs (accessed on 10 July 2021)) [The development of fNIRS started in 1992, and the hardware has only been improved since then. Here, we present a brief history of the important developments of fNIRS hardware over the years . The useThe original equipment was designed by Jöbsis as a single-channel continuous wave (CW) system and had low spatial resolution and was utilized for qualitative purposes, and later versions were developed with enhanced resolution . The useThe focus of this manuscript is to discuss the role of fNIRS-BCI design and application in the regulation of neuron function ,9. By usThe general steps in implementing BCI comprise: (1) acquiring the brain signals, (2) pre-processing the acquired data, (3) feature extraction relevant to the nature of the acquired brain signals, (4) feature selection, and (5) classification. Relationships between each of these steps are outlined in Correlations between these five steps have been discussed earlier in the text and shown . The manThe acquisition of the hemodynamic response from the brain is the first step in the process of obtaining the necessary data. The underlying principle of fNIRS relies on neurovascular coupling and the Blood Oxygen Dependent (BOLD) response as a result of a passive or active stimulus. In a typical fNIRS experimental setup, a predetermined number of optodes, i.e., light sources/emitters and detectors, are placed on the subject’s head, covering a specific region or the whole brain, called montage. The choice of the montage is determined by the nature of the experiment and based on the prior knowledge of the activation region in the brain for a given stimulation . For, e.The optimal separation between the emitter–detector pairs is about 3 cm and the penetration depth is half the separation of the source–detector distance . As the The relative changes in the hemoglobin (Hb) concentration are thus calculated by the differences in the absorption spectrum of oxyHb and deoxyHb. The quantification of oxyHb and deoxyHb depends on the type of the source used for light illumination. In general, there are three techniques used for light illumination, viz., (1) a constant light illumination through the continuous wave (CW) technique, which gives the measure of the light attenuation, (2) the frequency domain technique in which the source delivers the intensity-modulated light, which gives the measure of the light attenuation and the phase delay, and (3) the time domain method in which the light is illuminated as short pulses and the detector detects the shape of the scattering pulse. Among the above methods, fNIRS systems based only on the time and frequency domain techniques can be used to measure the absolute concentration changes in oxyHb and deoxyHb. The collected fNIRS signals consist of the raw optical density (OD) data. By using the modified Beer–Lambert law, the optical density signals are converted into changes in the oxyHb and deoxyHb concentrations .Pre-processing and feature extraction are followed by signal acquisition, in which the acquired fNIRS data are cleaned by the removal of various artifacts ,16,17. TGenerally, feature extraction utilizes heuristics and uses common features to see the similarities ,22,23. TFor the BCI to generate the right output, the features of the hemodynamic response must be interpreted correctly.Once the raw data are acquired, relevant features are filtered and extracted, making use of time domain, frequency domain and time–frequency domain methods. The most prominent of these is the use of the time–frequency domain for feature extraction purposes, which includes Wavelet Transform and Hilbert Transform. We discuss these two feature extraction processes in this paper.Wavelet-based pre-processing techniques are employed for the estimation of relevant features where multiple aspects of signal quality are taken into account. Scaling functions are used for reconstruction, whereas the order of the wavelets is used for decomposition purposes . PrimariHilbert transform performs orthogonal phase-shifting operations on a function, notwithstanding its frequency. This is most useful in modulating signals with single-sided Fourier transforms. However, it has high latency for low-frequency modulations given its long delay and implies a computational overhead for maintaining historical data of the signal . NeverthOnce the mandatory steps of data extraction and feature selection are completed, filtered clean data and unwanted artifacts are obtained that can be used for further analytics. Post feature extraction involving normalization and noise reduction, the clean data are passed through various classification approaches. These classifiers can be linear, Bayesian, nearest neighbors, discriminant analysis (LDA), or Artificial Neural Networks (ANNs). The linear model is the simplest, with a weighted approach having signal parameters attached to each other in a linear fashion, whereas the Bayesian model is a probabilistic model where variables are proportionated via a directed acyclic graph. LDA is applied via an n-k support hyperplane with a penalty for miscalculations. ANNs have a multi-layered structure and employ likelihood algorithms with multivariate generative distribution to their weights. Here, we discuss the various ANNs in detail.Given the holistic nature of the decision-making process of an ANN that is based on the predictability of input features, a multilayer feed forward perceptron can be trained as a non-linear classifier using the generalized backpropagation (BP) algorithm, in which momentum is chosen as a standard training principle to speed up convergence while maintaining generalization ,37. BipoA series of discrimination algorithms may be utilized to aid in computing and translating the data . Some ofThis approach has been in use within the framework for pattern recognitions in various waveforms. It has been derived from the radial basis function (RBF) network, which, in turn, is a bell-shaped function following the parameter in a non-linear manner. The most important advantage of PNNs is their speed, which happens to be many times faster than BP networks and is able to match Bayes optimal results fairly easily with much better performance. The relative speed is achieved on account of the core framework of matrix multiplications that makes the process inherently fast . In thisSupport Vector Machine (SVM) is a classification algorithm that utilizes the principle of structural risk minimization that utilizes a high-dimensional feature space wherein waves are charted using non-linear mapping fundamentals and subsequent linear regressions are conducted on each planar space. To obtain a sense of linearity, a hyperplane is traced using maximizing projection between the means and classes that bisects the latter and optimizes the margin of separation. This provides generalization ability to the learning algorithm .Both LDA and SVM are linear classifiers. Artificial neural networks (ANNs) are systems that mimic human and animal brains in discriminating points of data by taking environmental factors into account. They work via many artificial neuron pathways and look for groupings to classify the data. The Hidden Markov Model utilizes the probability of seeing distinctions to classify the data. Both ANN and HMM are non-linear classifiers. Such discrimination systems are employed in data extraction for convenient utilization.The most important goal that SVM regression chases is to minimize the error parameterized by the hyperplane that maximizes the margin. However, the idea is to make sure that the error is always kept within tolerance levels. This means that SVM excels in both Gaussian and radial datasets and performs quite well for non-linear data without requirement of any form of guesswork about their functional form. Since data segregation is performed with the maximum possible margin, the resulting model has better stability and can deal with inconsistencies such as noise or training bias .There are many advantages in the use of BCI based on fNIRS. One of the major advantages is that it is portable, while imaging techniques such as MRI and MEG require a large room with protective shields to minimize external interference. In comparison to EEG-BCI, fNIRS does not require any application of gel on the subject’s scalp . An fNIRApart from these general technical challenges, researchers have to also deal with session and subject-wise differences in classifying the working memory-based varying workloads. These problems are classified as domain adaptation in the machine learning step where data from separate subjects as well as sessions are classified as belonging to different domains ,33,34,35While there are many advantages, there are also disadvantages to fNIRS-BCI. fNIRS is based on a relatively slow hemodynamic response, and due to the limited number of sensors used, the spatial resolution is low. The non-uniform nature of the skull remains a challenge in positioning the sensors, which introduces some error in the measurement . MultiplThese challenges can be overcome with a combined imaging system. An EEG-BCI uses the post-synaptic potential that occurs in the brain to monitor and detect areas in the activated region . In termMachine learning is a set of computation algorithms that allows for better classifying and sorting the data. With machine learning, it is possible to streamline and refine the feature extraction process as well as combine different modalities together to obtain better precision. A machine learning algorithm learns from prior data and then the experimental data are fed in for the analysis and classification. There are two types of classifiers: linear and non-linear. Linear classifiers mark the data in pre-set categories, while non-linear classifiers work better when there is not a clear indication of separate categories or features to extract.During the feature estimation step, a wavelet-based pre-processing technique is employed, where there are different parameters of signal quality for the estimation of task-relevant signals and the elimination of artifact interference that causes undue noise. Hence, for the detection of useful signals, the input is passed through filters for noise reduction, decomposed using candidate wavelet functions, and then soft thresholding is performed using MiniMax, hybrid, or SURE methods ,54. WavefNIRS-BCI systems have a multitude of applications, especially in the clinical domain. BCI finds its application in areas such as communication, motor function, neuroergonomics, neurorehabilitation, environmental control, and entertainment . In the In the application related to motor functions, disabled patients can benefit from BCI by having prosthetics controlled by the brain. Currently, there is development on enhancing the accuracy and speed of these BCI-controlled prosthetics and wheelchairs by improving the speed of data acquisition, processing, and transporting to execute the function. There have been many improvements in the area of wearables as well.Further, the application of fNIRS-BCI in neuroergonomics promises that mental workload and conditions can be performed in real time. Three different levels of workload have been identified in air traffic control tasks , attentiIn this section, we will discuss some of the software that supports fNIRS-BCI applications.Homer2 is the most popular software and the first to be developed for the visualization of measurements and image construction of fNIRS data, and it currently supports the calculation of the optical forward model (sensitivity matrix) from a homogeneous, semi-infinite slab geometry . The funOpenVibe is an open-source software that allows for connections to virtual reality (VR) . It is mAnother software is BCILAB. BCILAB is more focused on experimental BCIs and neuroscience . It utilNICA is a novel toolbox for fNIRS calculations and analyses based on MATLAB that enables the processing and visualization of fNIRS data, including different signal processing methods for physiological artifact correction coming from noise generated by systemic influences and physiological artifacts . This toThe current modalities are not completely portable and functional for purposes of BCI technology such as neuro-prosthetics . In contHowever, the technology must also be improved in the means of reliability. Currently, the machines are not reliable enough to be used for real applications. Current research is looking at dual fNIRS and EEG in data acquisition for BCI . It has This strategy is not just limited to imaging Hb; it can also be extended for imaging other targets of the brain by designing fluorescent probes that are pre-delivered before adopting NIR imaging, which can be combined with the existing functional imaging as a multiplex approach to collect more information regarding the health status of the brain, especially in the case of Alzheimer’s, Parkinson’s, and other brain diseases.There are many neuroimaging tools such as EEG, MEG, ECoG (Electrocorticography), and fMRI. While EEG and MEG offer higher temporal resolution, ECoG is invasive. A multimodal approach in which combinations of these techniques with fNIRS may allow for a higher accuracy in data acquisition thereby permits a higher rate of success in feature extraction while keeping them portable. However, one of the advantages of fNIRS is that it can be complementarily combined with other modalities where its instruments do not create any fresh artifact noise in the readings relevant to other modalities in question and vice versa . The mosIn this paper, we have reviewed the use of functional near-infrared spectroscopy in brain–computer interfaces. We discussed that fNIRS is the process of observing the hemodynamic response via near-infrared light-based imaging. We discussed how BCIs work; in summary, after receiving data from the fNIRS, BCIs are programmed to filter out extraneous noise before the data can be extracted for useful features via filtering, feature selection, and post-processing. Through such feature extraction methodologies, an appropriate response is executed. Additionally, we discussed the development of hardware over time, chronologically from the discovery of NIRS, to observe hemoglobin levels up to the development of small, wireless fNIRS devices. The application of BCI was briefly discussed, and we learned about the various uses for such devices. We learned that the devices can be used for communication by having the patient answer binary questions with the thought of mental arithmetic. They can also be used to help amputees regain motor functions through prosthetics that are controlled by BCI systems. Finally, fNIRS-BCI can be used in neurorehabilitation by giving the patient feedback. We also looked at the advantages and disadvantages of fNIRS-BCI systems. The system was found to be useful in that it is extremely portable. The system was also relatively cheaper than most other imaging systems. fNIRS is also non-invasive and thus can be used easily. The disadvantages we found were speed limitation due to delay in the hemodynamic response and the fact that the system is not completely accurate and not ready yet for real applications. Finally, we looked at future developments that are being worked on. Future developments include making the system more user-friendly. For non-scientific or medical use of this system, there needs to be an improvement in making it easier to set up and use. There is also research looking to make the system more reliable. Currently, there are developments on making a hybrid fNIRS–EEG system for BCI.


This paper presents a novel dense optical-flow algorithm to solve the monocular simultaneous localisation and mapping (SLAM) problem for ground or aerial robots. Dense optical flow can effectively provide the ego-motion of the vehicle while enabling collision avoidance with the potential obstacles. Existing research has not fully utilised the uncertainty of the optical flow—at most, an isotropic Gaussian density model has been used. We estimate the full uncertainty of the optical flow and propose a new eight-point algorithm based on the statistical Mahalanobis distance. Combined with the pose-graph optimisation, the proposed method demonstrates enhanced robustness and accuracy for the public autonomous car dataset (KITTI) and aerial monocular dataset. Uncrewed aerial vehicles (UAVs) have drawn significant attention from the research community and industry in the last few decades. The primary advantage of a UAV is the ability to access places that are hazardous and hard to reach, such as for inspection of infrastructure ,2, preciMost monocular visual odometry methods use sparse feature points matched between images to compute the inter-frame motion ,10,11. TRecently, dense optical flow for monocular visual odometry has received significant attention, and the current state-of-the-art methods are capable of producing, on average, more than 85% of dense correspondences, having less than 3 pixels error ,14,15. Arobust monocular simultaneous localisation and mapping in a principled probabilistic framework. This is accomplished by using dense optical flow with estimated uncertainty as to the input to our visual odometry pipeline. Utilizing the existing robust back-end pose-graph SLAM )State-of-the-art monocular visual odometry methods suitable for a forward-looking camera in an unstructured environment are primarily designed for ground vehicles ,11,22. TDense optical flow methods have been investigated to address the difficulty of feature matching in low or repetitive textured scenes with dynamic objects. Some notable work includes the use of full discrete cost volume to compute the dense optical flow ,23 direcThe dense optical flow correspondences are treated like sparse feature matches, which are used to estimate the so-called fundamental matrix. The well-known normalised eight-point algorithm  can be uA new optical flow method is developed based on DCFlow . The accDense optical flow is a method to estimate the motion of each pixel between two input images. The algorithm typically optimises a cost function consisting of a data cost that penalises visually dissimilar pixels and a regularisation cost that encourages spatially smooth variation of the optical flow field. For each pixel in the first image of The epipolar constraint is added to the cost function before regularisation is applied. DCFlow  and mostFor a general two-dimensional Gaussian distribution, we know that the negative logarithm of the likelihood function is half of the squared Mahalanobis distance. The squared Mahalanobis distance, puted as :(1)dM2, and their uncertainty are used to estimate the camera motion, called visual odometry. The dense correspondences are used similar to conventional sparse feature matches. In contrast, the optical flow uncertainty is used for the random sampling and consensus and the Mahalanobis eight-point algorithm.From a pair of input images, we can find a set of matching pixels n pairs of matching image features, the linear constraints can be concatenated into a matrix form asEach matching pixel provides a linear constraint on the elements of The solution is then computed by finding the null space of matrix However, solving the null space of Equation  only mindistance ,27, whicf A from . The ranIn this work, we propose to use the uncertainty of the optical flow to accurately weigh each equation  and not too far to the side  of the camera. Plane fitting provides us with a plane equation satisfying h, is known , the scale of the inter-frame translation s can be computed as Two methods are proposed to minimise the scale drift: fitting the ground plane and using the computed depth map. First, the scale is determined by fitting a plane through the 3D reconstructed points roughly parallel to the camera axis’s We can combine the scales estimated from the ground height and depth map for ground-based vehicles using a simple average. For aerial vehicles, we cannot ensure that the height from the ground is fixed, and thus, we estimate the scale using ground height for the first frame and rely on the scale from the depth map for subsequent frames. The camera’s height is continually updated using the reconstructed 3D points of the scene, which is only used to reinitialise the translational scale when not enough (<5%) 3D points from the previous estimate overlaps the current triangulated points. The estimated scale is then multiplied by the estimated camera translation, 3D reconstructed points, depth map, and uncertainty. The information matrix of the reconstructed 3D points is divided by the squared scale.Given the dense optical flow, there are two methods to estimate the inter-frame poses depending on the magnitude of the pixel motion . If there exists enough parallax, the Mahalanobis eight-point algorithm  along wilgorithm  instead.The use of scaled-down images  for dense optical flow estimation cannot guarantee the accuracy of the matches when the pixel translation between two images is too small. This case occurs when the vehicle moves very slowly or hovers, causing the motion estimation to be error-prone. Small translational motion estimation is a common problem in most monocular visual odometry, as the small parallax between two images leads to difficulty in estimating both motion and structure accurately. Two conditions determine the sufficient parallax. First, the Shi–Tomasi corner matches have a median displacement magnitude of at least Loop closure (or revisiting the previous locations) detection is accomplished by using the robust linear pose-graph optimisation method from . SimilarWe evaluated our proposed SLAM framework using the public KITTI dataset  and our image_2 folder). In the odometry experiment, we assume the camera is The dataset we used to verify the performance of our proposed algorithm for ground-based vehicles is taken from the KITTI benchmark. For optical flow evaluation, we use the flow 2015 dataset , while fEpicFlow  interpolThe uncertainty estimate for the dense optical flow is visually inspected, where it was observed that occluded, out-of-bound, or textureless regions of the image have high uncertainty value. For ground-based vehicle’s visual odometry result, we compare our performance with existing methods, specifically VISO2-M , MLM-SFMSince our visual odometry method does not rely on the restrictive motion model of the vehicle, we can easily apply our proposed method with slight modification to aerial vehicles . The difference with a ground-based vehicle is that the camera height is not assumed constant but is updated for each motion since the aerial vehicle can change its height arbitrarily. Another challenge of aerial visual odometry is that it can rotate its yaw with no translation, making pose estimation and 3D scene reconstruction highly under-constraint and error-prone. We also incorporated such motion in the video sequences we used in our experiment.We captured 500 frames of video from a quadcopter flying among some trees near a road, where the scene has highly repetitive, unstructured, and dynamic objects . Due to the lack of ground truth, unlike the KITTI dataset, we evaluate the scale drift by reversing the frames and appending them to the end of the video, where the last frame coincides with the first frame. Remark 1.We do not compute the loop closure constraint for this video sequence. The result shown in From the third plot of For the next experiment with UAV video, we captured 563 frames of a UAV flying at high speed with significant variation in height. We have also marked some trees with yellow tapes (1 m apart) to calibrate the first translational scale and obtain a measure of scale drift after the UAV returns to the same spot. The error in the estimated position can also be visually observed by comparing the location of the reconstructed scene points. We plot 3D scene points with a depth standard deviation less than 0.2 m for the first and last frames. This paper presented a new Mahalanobis eight-point algorithm using the dense optical flow. The full uncertainty of the optical flow was estimated in a principled manner by using the negative logarithm of a bivariate Gaussian distribution and fitting it to the matching cost. The weighted eight-point algorithm optimised the Mahalanobis distance of each pixel correspondence to obtain a robust inter-frame motion estimate. With the SLAM pipeline of the front-end and back-end modules, the performance of the proposed method was evaluated using real datasets, demonstrating improved performance on a UAV platform compared to other state-of-the-art methods regarding accuracy and robustness. Future work includes the use of high-speed dense optical flow methods proposed as in  for real


Public health is threatened by climate change and extreme temperature events worldwide. Differences in health predispositions, access to cooling infrastructure and occupation raises an issue of heat-related health inequality in those vulnerable and disadvantaged demographic groups. To address these issues, a comprehensive understanding of the effect of elevated body temperatures on human biological systems and overall health is urgently needed. In this paper we look at the inner workings of the human innate immunity under exposure to heat stress induced through exposure to environment and physical exertion. We couple two experimentally validated computational models: the innate immune system and thermal regulation of the human body. We first study the dynamics of critical indicators of innate immunity as a function of human core temperature. Next, we identify environmental and physical activity regimes that lead to core temperature levels that can potentially compromise the performance of the human innate immunity. Finally, to take into account the response of innate immunity to various intensities of physical activities, we utilise the dynamic core temperatures generated by a thermal regulation model. We compare the dynamics of all key players of the innate immunity for a variety of stresses like running a marathon, doing construction work, and leisure walking at speed of 4 km/h, all in the setting of a hot and humid tropical climate such as present in Singapore. We find that exposure to moderate heat stress leading to core temperatures within the mild febrile range (37, 38] Increasing incidence, magnitude, and duration of heatwaves4 not only have a direct impact on excess mortality rates worldwide6 but also induce physiological stress and undermine labour productivity7. In conjunction with population growth projected to occur primarily in hot tropical and sub-tropical regions8, more and more people will be exposed to heat threats that put public health at great risk due to heat-related illnesses. Inequality in terms of access to infrastructure (e.g. air conditioning), thermoregulatory capacity (e.g. infants and elderly10), pre-existing health conditions (e.g. cardiovascular and respiratory disorders12), and occupation (e.g. outdoor workers) raises an issue of health equity and requires urgent research efforts to comprehensively understand the negative effect of short- and long-term heat exposure and physical exertion on health. This knowledge will inform public health workers and policies to minimise risks of heat-related illnesses, particularly in vulnerable and disadvantaged populations.Heat puts a significant burden on global public health13, respiratory14, endocrine15, urinary16 and reproductive17 systems has been studied extensively, the interaction of the core temperature with the human innate immune system (HIIS) is not well studied in works on public and occupational heat-related health, safety and productivity18. Recent studies show that the core temperatures observed in humans have significantly fallen over the last two centuries, most probably due to advances in health care leading to less amount of chronic inflammation in human body20. On the other hand, the global processes of climate change and urbanisation put more and more people at risk against a threatening exposure to urban heat21, which can lead to fever-like levels of core temperatures, perturbing the HIIS. Therefore, there is a critical need for comprehensive understanding of the interactions of climate, exposure, physiology and human innate immune system to assess the associated benefits and risks and to suggest best mitigation strategies on an individual level and devise policies on a population level.While the effect of heat on cardiovascular22, up to a threshold temperature, which varies according to cell type. Beyond this threshold, their syntheses is constrained and exponential cell death follows24. The threshold, at which thermal damage occurs in the immune system, was detected in individuals suffering from heat stress or heat stroke25.In this work, we take a closer look, from physical activities down to the cellular level, on how heat affects the inner workings of the human innate immune system response, trace this response and relate it to the body’s reaction during heat-inducing physical activities. On a molecular level, heat stress increases the synthesis of HSP 70, intracellular proteins shown to possess the capacity of inducing lasting protective immune responses27. An increase in granulocytes’ bactericidal capacity was observed at 4029. Thermal stress increases the recruitment of neutrophils to the sites of infection and in distant tissues31. It also increases the number of circulating neutrophils34 in the body.Fever-range temperatures heighten the respiratory burst that is often linked with neutrophil activation and increasing neutrophil’s bacteriolytic activity36. Koch et al., have shown that thermal treatment induces the release of cytokines, such as TNF37. Macrophages lining the synovial tissue of rheumatoid arthritis joints produce cytokines such as IL-1b, IL-6, and TNF-a in response to increase in body temperature43. Humans and rats that are exposed to heat stress were found to have elevated plasma concentrations of pro-inflammatory cytokines44. In the event of a heat stroke, both human- and animal models experience an increase in the levels of pro- and anti-inflammatory cytokines45. A loss of intestinal barrier integrity increases its permeability to ITMs,as was observed in cows, which implies gut leakiness activity attributed to alkaline phosphatase concentration changes37.Heat is known to improve the phagocytic capability of macrophages by heightening their responsiveness to inflammation triggering moieties (ITMs)46. By employing mechanisms such as vaso-dilation and constriction, sweating and shivering, the system ensures that the body’s core temperature is maintained at the levels of approximately 36.8 48.For a healthy individual, where inflammatory processes are at a bare minimum, the core temperature is maintained by a complex physiological system of thermal regulation49 such that it can predict the dynamics of its key players as a factor of core temperature. This then allows us to identify the core temperature regimes, which either benefit from, or impede, the efficient response of the innate immunity. We then couple this model of innate immunity with a model of thermal regulation of the human body50 to investigate scenarios of heat exposure and human activity typical for the hot and humid tropical climate of Singapore.We trace the effect of heat starting from the inner workings of the innate immune response all the way to identifying its effects on the human physiology by coupling two validated computational models: a human innate immune system model and a model of thermo-regulatory response and core temperature dynamics. To do this, we first extend a previously developed model of HIIS51, which has to be addressed in public health policies52.We show that even in hot climate, the human physiology is capable of maintaining a healthy state by adapting to temperatures in the mild febrile range  is to maintain the body core temperature by eventually achieving the neutral heat storage (S) expressed by the following heat balance equation:In the body temperature model, core and skin components are considered as stocks of energy. The energy is exchanged between the stocks as well as with the environment through evaporation 50 for a full mathematical specification of the model, its experimental validation, and a demonstration of its predictive performance in dynamic thermal conditions. As is shown in Fig. The complete stocks-and-flow system dynamics representation of the model is provided in Fig. In the following subsections we show the dynamics of ITMs and cytokines in response to change in core temperatures. Dynamics of alkaline phosphatase see Fig.  and neutcase 1: one with very high initial ITM concentration patterned after patients experiencing severe inflammation like that of cardiac surgery , and case 3: one with intermediate initial ITMs pertaining to those individuals belonging to disadvantaged groups  that have higher heat-related health risks compared to healthy individuals.Inflammation Triggering Moieties (ITMs) may refer to bacterial lipopolysaccharides and extracellular nucleotides serving as pro-inflammatory signals that trigger local and systemic inflammatory responses in HIIS. We look at three cases: 60. This shift in the dynamics of ITMs from the mild febrile range to higher core temperatures is observed in all three cases: high, intermediate, and low initial ITMs.Our results show that ITM concentrations are lower for temperatures 37 entclass1pt{minimaCytokines are “messenger” proteins that orchestrate the complex mechanisms of the innate immune response. Cytokines can either be pro-inflammatory or anti-inflammatory. Pro-inflammatory cytokines are produced by immune cells called macrophages during the inflammation process and migrate to the endothelial barrier, opening it up, and thus allowing the entrance of neutrophils to the site of inflammation. Anti-inflammatory cytokines, also produced by macrophages, are immuno-regulatory molecules that control the production of pro-inflammatory cytokines.The shift in dynamics of anti-inflammatory cytokines at temperatures in the mild febrile range is also seen in all three cases . See Fig. 60. Increasing further this ITMs concentration will induce the production of more pro-inflammatory cytokines. The intermediate initial ITMs case provides a proof of concept on how the innate immune system functions for those individuals belonging in disadvantaged groups who may experience recurring complications that are not fatal to the system.Pro-inflammatory cytokines, on the other hand, are generated in low concentrations for temperatures within the mild febrile range. This apparent divide is seen in Fig. In the previous sections, we have identified that there are two different levels of elevated core temperature, which either have a beneficial or detrimental effect on HIIS. While the immune response is improved for core temperatures rising up to 38 61. For all the activities we derive the efficiency of mechanical work M from the expression proposed by Fiala et al.62:64.Here we provide the results of simulations of the core temperature dynamics over a period of 2 h and 46 min for a broad range of human activities varying from light to vigorous, while exposed to typical outdoor conditions of equatorial Singapore. In these scenarios we assume a constant level of metabolic rate production due to physical activity and initial state of the thermo-physiological system in steady state  when the pace of work or activity is forced (e.g. in marathon).Considering the upper boundary of intensity of occupational construction work of 8 METIn the previous two sections, we have (1) identified the values of core temperatures that either induce benefits to the innate immune response or undermine its functions, and (2) pinpointed the regimes of physical activities and environmental conditions that these core temperature correspond to. In these simulations, core temperatures did not change over time. Those results provide a good understanding of 2 key points: how temperature affects the dynamics of the immune system and how HIIS battles the inflammation over a period of 36 hours. It however does not represent the dynamically changing core temperatures and their effects in real scenarios of physical activities and heat exposure.In this section we investigate how HIIS responds to three physical activities: a marathon of 2 h 46 min, construction work of 9 h with an hour lunch break, and leisure walking at speed of 4 km/h for 6 h in Singapore setting. We then simulated HIIS’ reaction over a period of 36 h. This period covers the entire course of the activity and the recovery that follows. The activity schedule and conditions for each of the considered scenarios are provided in Table 64. The core temperature of construction workers stays just below 38.0 Figure The dynamics of ITM concentrations Fig. B in consThe generated core temperatures for the three physical activities are slightly above the normal temperature of 36.8 Although marathon induces the highest level of ITMs, the ITM concentration still decreases over time. Construction work, on the other hand, is done on a daily basis. Therefore, in the long run, construction work induces a greater impact on the innate immunity through the continuous production of ITMs at a longer period of time.Figure 49.A marathon induces the highest level of pro-inflammatory cytokines among the three physical activities. Recall that pro-inflammatory cytokines are messenger proteins responsible for opening up the endothelial barrier, recruiting more of the circulating neutrophils into the site of inflammation, which then aggravates the ongoing inflammation. Pro-inflammatory cytokines are at low levels for construction work and walking. Anti-inflammatory cytokines are also produced, but the decay for marathon is slower than the two activities (see snippet in Fig. We emphasise that the combined model is not yet calibrated to real data and thus the rates at which anti-inflammatory cytokines as well as pro-inflammatory cytokines decrease have yet to be refined to realistically model HIIS in the context of a physical activity. We note that we have kept the calibrated parameters of the modified HIIS model to only three values. As such, this work is a proof of concept of how the core temperatures affect the performance of HIIS in different physical activities. Part of our future work is to collect data on concentrations of pro- and anti-inflammatory cytokines and to calibrate the combined model.Although the human body has evolved to adapt to changes in ambient temperature, the imminent threat of climate change, which promises more heatwaves, will inevitably cause the rise of heat-related health problems. Despite the urgency of knowing the risks associated with a rising core temperature to the innate immune response, there is an apparent knowledge gap between understanding the underpinning cellular mechanisms and the associated processes of the innate immune system and its implications for the human body, and consequently the types of healthy physical activities humans are constrained to do in given thermal environment.We bridged this gap by coupling two validated and established computational models: the human innate immune system model and a model of thermo-regulatory response and core temperature dynamics.In order to do so, we first needed to modify the existing model of the human innate immune system in such a way that it takes temperature changes into consideration. We identified the parameters in a previously developed model of the innate immune response that are directly impacted by temperature. Since appropriate data to calibrate the modified HIIS model with respect to temperature is currently unavailable, we chose those parameters that best describe the qualitative dynamics of HIIS based on known behaviours documented in literature. We found that a simple linear response for majority of the parameters coupled with an exponential increase in the rate for induced ITMs with respect to increasing core temperature, capture the shift of the dynamics of key components in HIIS from the beneficial regime of mild fever to a detrimental effect above 38.0 66. Thus, these activities should be limited in duration or other measures such as active cooling should be put in place to protect people from hazardous heat stress.Combining the innate immunity model with a thermo-regulatory model, we identified the regimes of climate, activity intensity and duration which lead to body core temperature exceeding the threshold level of 38.0 To the best of our knowledge this is the first time that core temperature is modeled in conjunction with the human innate immune response. This allows for a better understanding of the underlying mechanisms of the human innate immune system in response to heat, and for us to be able to probe its consequent beneficial or detrimental effects on the human physiology. Our study identified the lack of empirical data reporting the effect of heat on human biological systems, which constitutes an important direction for future work. Empirical research in human-heat interaction will enable greater predictive performance of computational models, such as those demonstrated in our work.67. It is therefore important to understand that the effect of heat and physical exertion in combination with these factors can have even more dramatic effect on HIIS especially in vulnerable categories of population such as the elderly and overweight. To account for these parameters, our model can be easily extended and coupled with other computational models (e.g. of obesity68 to simulate more focused scenarios, which fall outside the scope of our investigation. Other simplifications such as static environmental conditions and metabolic rates were made to enable easier comparison of scenarios. Our models readily provide for the simulation of more dynamic microclimates and physical activities to study more specific scenarios at hand.Our study focused on the general case scenarios, where the “average” person was modelled by the model of physiological system. It is known that individual parameters of people such as body mass composition, physical fitness, acclimation, water intake, gender, and age influence the performance of the thermophysiological systemOur work contributes to the existing knowledge on how changes in human thermoregulation affect innate immunity—not only on the cellular level, but more importantly, its implications on individuals and subsequently on public health. Heat-related health inequality elucidated in our study by an example of risks of outdoor occupation for the innate immune system performance poses an additional challenge for public health and should be specifically addressed by future research and public health policies.