Individuals were more willing to view  and share  COVID-19 videos with high emotional consistency with virus variant topics. (4) Conclusions: The results of empirical analysis showed that content relevance and emotional consistency between videos and virus variant topics significantly positively impacted video views and shares. The trained SVM classifier can support public health departments in monitoring and assessing the COVID-19 pandemic. Our study provides management advice while helping individuals reduce harm and inform next-step decisions.(1) Background: The coronavirus variants have posed serious challenges for the prevention and control of the COVID-19 pandemic. Individuals selectively watch and forward videos that help them reduce the damage caused by the virus. Therefore, the factors influencing video viewing and sharing in the context of the COVID-19 pandemic caused by virus variation must be explored. (2) Method: Based on a combination of uncertainty reduction theory and functional emotion theory, this paper designed hypotheses regarding how content relevance and emotional consistency affect video views and shares. We used the support vector machine (SVM) classification algorithm to measure the content relevance between videos and virus variant topics. We performed sentiment analysis of video text to evaluate the emotional consistency between videos and virus variant topics. Then, we used empirical analysis to build the model. (3) Results: The trained SVM classifier was effective in judging whether the video text was related to virus variant topics (F = 88.95%). The content relevance between COVID-19 videos and virus variant topics was generally low. The results showed that the higher the content relevance, the more views (IRR = 1.005,  Since the COVID-19 pandemic spread around the world, the damage and changes to the global economy and trade, as well as people’s lifestyles and health, have been immeasurable ,3. The CCOVID-19 belongs to the coronavirus family and has a single plus-stranded RNA genome. The unstable single-stranded structure means COVID-19 is prone to variation . VariantCrisis refers to unexpected, sudden, and uncommon  events tInternet platforms provide the easiest access to information, and more than 70% of adults use them when their health is at risk . InformaAny form of interaction between individuals on Internet platforms during a crisis is meaningful  as interIn the context of the COVID-19 pandemic situation caused by the emergence of virus variants, in this study, we aimed to explore the impact of content relevance and emotional consistency between videos and virus variant topics on the number of views and shares of videos. We used the support vector machine (SVM) classification algorithm to calculate the content correlation between novel coronavirus videos and virus variant topics and then used sentiment analysis to calculate the emotional consistency between COVID-19 videos and virus variant topics. In this study, we included the calculated content relevance and emotional consistency scores in the independent variable index, and we conducted empirical to test the study hypothesis.Many researchers have focused on the COVID-19 pandemic. The information factors that have been mined include information content , informaThe studies on emotion during the COVID-19 pandemic have mainly been conducted within the field of public opinion and have rarely considered the number of views and shares of videos. Researchers have focused on understanding public attitudes by analyzing public responses and interactive texts with the public health sector and news media. Huyen (2021) explored the public’s concern about the COVID-19 pandemic in Vietnam on different Internet platforms and found that the public’s attitude toward COVID-19 pandemic information changed from negative to positive as the COVID-19 pandemic progressed . In addiSocial psychology scholars think that only when people feel sure about their environment can they feel confident in their following actions and have reasonable expectations for the future . UncertaUncertainty is a primary state experienced by humans; as such, uncertainty management is a fundamental activity . URT, deCrises create a high degree of uncertainty. Because crises are unpredictable and irregular events, uncertainty exists at all stages of the life cycle of a crisis . URT staDue to aversion to uncertainty, also known as ambiguity aversion, individuals reduce uncertainty in the environment by collecting accurate details and sharing them with others . AlthougEmotion is an individual’s internal psychological state as different responses to events, agents, or objects . FET staBased on the above theories, scholars have described the process of FET as follows: an individual perceives the environment or an event related to them, the individual produces a unique emotional experience, and specific emotions manage the individual to produce a specific state of readiness for action or action-oriented state; in the end, the specific emotions affect individual cognitive or action, with the purpose of behavior being consistent with the emotional target . In a stIn a crisis, information can help individuals to reduce ambiguity , distingCrises are constantly changing, and unexpected events are uncontrollable . When acHypothesis 1 (H1):When the COVID-19 coronavirus variation causes a crisis, the more relevant the video content to the virus variant topics, the more likely the video to be viewed by more people.Hypothesis 2 (H2):When the COVID-19 coronavirus variation causes a crisis, the more relevant the video content to the virus variant topics, the more people were likely to forward the video.Emotions are considered distinctive, transient, and targeted, triggering different adaptive behaviors in response to the event causing the emotion . In the Hypothesis 3 (H3):When the COVID-19 coronavirus variation causes a crisis, the higher the emotional consistency between videos and the virus variant topics, the more likely the video will be viewed by more people.Hypothesis 4 (H4):When the COVID-19 coronavirus variation causes a crisis, the higher the emotional consistency between videos and virus variant topics, the more likely the video will be forwarded.On video platforms, to motivate creators, platforms will hold various events from time to time, providing creators with video exposure and creation incentives. Videos posted by creators participating in activities will gain additional exposure, and individuals may choose to watch and forward the video to engage in the activity. To push high-quality content to users, video platforms have established some leaderboards, which attract traffic to the videos on the leaderboard in the top positions. The content accuracy and video information quality released by public health departments and news media are higher than those of ordinary video producers. Hence, individuals trust them more. Individuals may view and forward their videos without trust in public health departments and news media. Therefore, we hypothesized that whether the video is an activity, whether it is included on a leaderboard, and whether the source is public health departments and news media would interfere with our study. We considered these variables as control variables in the model.The independent variables in this study included content relevance and emotional consistency. Content relevance referred to the proportion of sentences referring to virus variant topics in the effective sentences in the video. We considered emotional consistency as the difference between the video’s emotional score and virus variant topics. The dependent variables included the numbers of video views and shares, and the control variables included activity videos, being listed on a leaderboard, and video source. The definitions of the variables are provided in During data preprocessing, we used the SVM algorithm to evaluate the content relevance between videos and virus variant topics, and we used sentiment analysis to evaluate the emotional consistency between videos and virus variant topics. For hypothesis testing, we used empirical analysis to test the impact of content relevance and emotional consistency between videos and virus variant topics on the number of video views and shares. The research workflow is shown in Step 1: Audio to text: We used mature audio-to-text technology to convert video audio into text.Step 2: Sentence segmentation: We segmented the video audio that was processed into text, and we segmented the video audio text into several sentences.Step 3: Data cleaning: We eliminated valid data should when the audio was converted to text and sentences were segmented.Step 4: Conversion to text and sentence segmentation: With stratified random sampling and labeling, we randomly selected 4 sentences from each video as the labeling set. We invite researchers to label the sentences as related to virus variant topics or not, which formed the labeled data set.Step 5: SVM classification: We used the annotated data set as the training set to train the SVM classifier with the linear kernel function, and all valid sentences were classified. The output of two data sets included a sentence set related to and a sentence set unrelated to virus variant topics. Then, according to the content relevance calculation formula, we calculated the content relevance between video and virus variant topics.Step 6: Emotion analysis: We used the mixed emotion dictionary to analyze the emotion score of the two data sets, and we obtained the emotion score of each sentence and the emotion consistency score of each video.Step 7: Negative binomial regression: We used the extracted independent variables (content relevance and emotion consistency) and dependent variables to perform negative binomial regression and hypothesis testing.Step 8: Robustness test: We used the replacement variable measurement method to test the robustness of the model.The research data in this paper comes from a well-known Chinese video platform, which is characterized by attention to originality and strong social functions. Originality ensures that videos on the platform are produced and published by the video owner. Videos on the platform rarely have different people publishing precisely the same content. Intense sociability indicates that users not only use the platform to obtain information but also are willing to socialize on the platform, such as by forwarding information. The platform and data characteristics met our study needs.This paper used Chinese words related to virus variants and virus variant names such as “omicron” as search keywords to obtain video data. Our results of data analysis showed that May to August 2021 was the platform’s hottest time for virus variant topics. The number of videos published was above 500 per month, higher than the average of 238 in other months. In addition, the monthly growth rate in May 2021 was 25.37%, decreasing in August to 34.81%. Therefore, this article selected video data from May to August 2021, for a total of 4049 videos. The video data included the video screen and audio, video duration, views, shares, uploader certification type, activity video identification, leaderboard ranking, and other tags.Platform’s search mechanism finds keywords that appear in the video’s title, tag, description, or the publisher’s nickname, and then displays the search results. Therefore, our search results were not necessarily related to the virus variant crisis. Most videos were 2–16 min in length, and videos less than 2 min contained limited information. In a crisis, users’ negative emotions make needs for information acquisition or sharing urgent, so videos longer than 16 min do not conform to user habit of fragmentary information acquisition. Therefore, we eliminated videos that were entirely irrelevant to the COVID-19 topic through manual reading, and we removed videos that were shorter than 2 min and longer than 16 min, for a total of 2501 remaining videos.This study used mature speech-to-text technology to convert the data of the 2501 videos from audio to text. The conversion accuracy was more than 99%, which met the study requirements. Through text processing, we found that some videos did not use human voice dubbing, so the audio-to-text result was empty. We removed the videos for which the result of audio-to-text was empty, the language was not Chinese, or the language was utterly illogical and rough. We finally obtained 2037 videos that met the study requirements.A video 2–16 min in length contains multiple sentences. This article uses the Chinese ending words as the segmentation basis and divided the 2037 videos into several sentences. The video text contained the uploader’s opening and closing remarks, transitions between topics, and other content unrelated to the video content, which needed to be eliminated. The final total was 39,703 sentences.SVM is a machine learning algorithm based on statistics. It has substantial advantages in binary classification problems, is widely used in natural language processing, so was appropriate for our study . This paThis paper used the “Jieba” library for word segmentation, processing the words after eliminating useless sentences. We used the disabled dictionary to remove meaningless words, imported the custom dictionary to increase the accuracy of the segmentation result, and replaced synonyms for words with the same meaning. Due to the limited information expressed by a single character, we retained only phrases with a character length greater than 2 in word segmentation. Then, we used TF-IDF to process the word vectors of the segmented data. The SVM using a linear kernel learned whether sentences were classifiers for virus variant topics. The accuracy of the SVM classifier was 89.37%, and the F1 value was 88.95%, indicating that the training was effective.We obtained the data used in training the classifier from 39,703 sentences after segmentation, and each video had data included in the classifier. Therefore, our classifier could be appropriately applied in our experiments. Next, we used the trained classifier to analyze all 39,703 data items to determine whether they were related to virus variant topics.The training classifier determined whether each sentence was related to virus variant topics, so that we could judge the correlation between the video and virus variant topics. The number of sentences related to virus variants indicated how many sentences related to virus variant topics were in a video. The total number of sentences indicatedSentiment analysis is a text analysis method used to explain emotion intensity . We adopWe performed the empirical analysis in this study with Stata 16.0. The descriptive statistical analysis results of the 2037 video data are shown in This paper used canonical correlation analysis to test whether the independent variables were strongly correlated, leading to multicollinearity. The results of the canonical correlation analysis  showed tIn addition, this paper evaluated the variance inflation factor of the independent variables. The results showed  that theIn this study, we incorporated the independent variables  and control variables  into the regression model. The dependent variables were the numbers of views and shares. The hypothesis test results of negative binomial regression are shown in p = 0.017). So, H1 was supported. The results of the test of H2 proved that video shares  increased by 8% when the content relevance increased by one unit, and the effect was significant, supporting H2.H1 stated that the more relevant the video content to virus variant topics, the more views the video would receive. Model 1 in the table shows that when the content relevance score increased by one unit, the number of views increased by 0.5%, which was statistically significant (p < 0.001), so H3 was supported as well.Model 1 also showed that when the video emotional consistency score increased by one unit, the number of views significantly increased by 52.5% , which supported H4.We found from the test results of Model 2 regarding emotional consistency that with an increase in emotional consistency, video shares significantly increased (IRR = 1.761, This paper used punctuation marks as the standard when segmenting the sentences in the videos, where each segmented sentence expressed complete semantics. Therefore, we chose the number of sentences as the standard for calculating content relevance. Although sentences with shorter character lengths were eliminated, the character lengths of the sentences widely differed. In terms of video length, we found large differences in the proportions of each sentence. Therefore, this paper changed the method of calculating content relevance and used the character length of the sentence as the calculation standard to test the regression results and the robustness of the original model. We also proved that the model that replaced the method of calculating content relevance had no multicollinearity. The experimental results of negative binomial regression are shown in The model results after changing the method of calculating content relevance showed that the emotional consistency and content relevance between videos and virus variant topics significantly impacted video views and shares. The robustness test results supported the original model’s regression results.From the theoretical combination URT and FET, this paper used emotion analysis, the SVM algorithm, and empirical analysis to explore the factors influencing the number of video views and shares when COVID-19 variation caused tension among people.First, functional emotion theory well explained the impact of emotional consistency on the number of views and shares in this study. This paper used the combination of URT and FET to explain the complexity of individual acquisition of information and sharing behaviors. FET could explain the impact of emotion on the number of video views and shares during the COVID-19 crisis. URT is a theory commonly used in the study of public health emergencies, but uses of FET to explain individual information collection and sharing behaviors in crises have been relatively rare. FET has often been used to understand the motivational effect of emotional perception on behavior. Previous studies focused on the influence of negative emotions on adopting protective behaviors. Nabi proposed discrete emotions as a psychological framework. In drunk driving scenarios, emotions stimulate relevant behavioral tendencies, which promote selective allocation of attention and information-seeking behaviors . HoweverSecond, this paper trained an SVM classifier to automatically divide the audio text into a set of sentences related to virus variant topics and a set of unrelated sentences. The trained SVM was effective, with an accuracy of 89.37% and an F1 of 88.95%. Previous researchers mostly used content analysis to classify video content , which rThird, the results of empirical analysis showed that the content relevance between videos and virus variant topics positively correlated with the number of video views and shares. Compared with historical pandemic video information, viewers are more likely to watch and forward videos related to COVID-19 . BecauseFourth, our results showed that the higher the emotional consistency, the more a COVID-19 video is viewed and shared. Emotion, as the external manifestation of an individual’s feelings, is reflected in individual language and writing, which reflect the individual’s attitude toward events. In studies on emotion in crisis, researchers determined individuals’ emotional expression from individual language and writing to measure individual feedback on public crisis management . HoweverCOVID-19 prevention and control have always had to face the challenges produced by new crises, such as the emergence of new virus variants. From our findings, we provide several recommendations for public health departments, news media, and individuals concerned with the COVID-19 crisis.First, the SVM algorithm classifier could identify whether the text in COVID-19 videos was related to virus variant topics. The coronavirus variations have led to crises during the COVID-19 pandemic, which have posed constant challenges to the prevention and control of the pandemic. The classifier could monitor the relevance between videos and virus variant topics and evaluate the change in viewer attention to virus variant topics. It can also provide technical support for popularizing highly relevant videos when a new COVID-19 variant emerges.Second, this paper revealed that when the COVID-19 virus variant, the number of video views and shares is related to content relevance and the emotional consistency between the videos and virus variant topics. Specifically, the public is more willing to watch and forward videos highly related to virus variant topics and that have similar emotional expressions. Public health departments and news media use the Internet to provide information to the public and support for the public to use the Internet to collect and share information. This finding suggests that public health departments and news media should consider both the relevance of videos and virus variant topics, as well as whether the video’s emotion is close to the crisis emotion when providing video information to people in areas where the COVID-19 pandemic is serious.Finally, for individuals in crisis, our findings explain what kind of video can help them understand the crisis and is suitable for sharing with others. In the case of unpredictable danger and the timing of crises, our findings can help individuals quickly find the information they need when a crisis occurs. Individuals can also reduce their uncertainty by watching and forwarding videos and feel sure about the next steps in a crisis.This study has many limitations. First, the sectional data we used could not reflect the changes over time in the number of video views and shares. The situations caused by the pandemic are variable, resulting in sudden tension, followed by relaxation as waves caused by the variants pass. The given situation considerably and quickly changes individual information-seeking and -sharing. Therefore, the time when a video is uploaded during a crisis is critical. In future studies, panel data of the number of video views and shares on specified days can be considered. The classification of time can also be more detailed. For example, when the information demand is high, we can use the classification of the video release time based on the search volume on Internet engines as the classification standard. Second, due to the limitation of the study methods and data, we did not consider the entertainment experience when individuals watched the videos. In addition to information and emotional needs, watching videos is a form of entertainment. Factors affecting individual viewing and immersion perception, such as the camera position of the video producer when recording the video, will also impact the number of video views and shares .From the perspective of video content relevance and emotional consistency between videos and virus variant topics, in this study, we explored the factors that affected the number of video views and shares. The study leveraged the SVM classification algorithm to assess video content relevance and used sentiment analysis to provide support for scoring the sentiment coherence of the videos. The results showed that when the emergence of virus variants caused sudden tension during the COVID-19 pandemic, individuals tended to watch and forward videos whose content and emotional expression were similar to virus variant topics. Our findings can help public health departments better monitor the development of the COVID-19 pandemic and cooperate with news media to push information that more in line with the needs of the public. Our study can also help with providing timely, rapid, and accurate advice for individuals to respond to a new crisis, helping them confidently make decisions. The limitations in data and variable extraction in this study need to be addressed. In future studies, we will use panel data to improve consideration of time and other experimental methods to measure individual entertainment needs.


Caenorhabditis elegans and seek to investigate the possibility of generating lower complexity models that capture the system’s dynamics with low error using only measured or simulated input-output information. Such models are often termed black-box models. We show how the nervous system of C. elegans can be modelled and simulated with data-driven models using different neural network architectures. Specifically, we target the use of state-of-the-art recurrent neural network architectures such as Long Short-Term Memory and Gated Recurrent Units and compare these architectures in terms of their properties and their accuracy (Root Mean Square Error), as well as the complexity of the resulting models. We show that Gated Recurrent Unit models with a hidden layer size of 4 are able to accurately reproduce the system response to very different stimuli. We furthermore explore the relative importance of their inputs as well as scalability to more scenarios.Given the inherent complexity of the human nervous system, insight into the dynamics of brain activity can be gained from studying smaller and simpler organisms. While some of the potential target organisms are simple enough that their behavioural and structural biology might be well-known and understood, others might still lead to computationally intractable models that require extensive resources to simulate. Since such organisms are frequently only acting as proxies to further our understanding of underlying phenomena or functionality, often one is not interested in the detailed evolution of every single neuron in the system. Instead, it is sufficient to observe the subset of neurons that capture the effect that the profound nonlinearities of the neuronal system have in response to different stimuli. In this paper, we consider the well-known nematode  Recent developments in experimental neuroscience have considerably increased the availability of novel recordings and reconstructions shedding further light into the structure and function of the brain as well as many other systems. But understanding the complexities behind the relations between structure and function as well as the behaviour of such systems across multiple scales in these neuronal collections is constrained by the methods available to study them. This challenge has raised interest in many related fields, such as electrophysiological analysis, imaging techniques, brain-related medicine, computational modelling, simulation, and model reduction. Many of these efforts, while not directly providing specific information regarding structural or functional dynamics, do supply large volumes of recordings, measurements or simulations of observable input-output behaviour. The availability of these large datasets raises the question of whether low complexity, data-driven, black-box models can be used to model such input-output relations with low error, avoiding the reliance on detailed inner structures that may not be known or available.To determine whether such an approach can be used for large, complex systems, one research direction is the study of smaller and simpler nervous systems, for which the underlying principles of network organization and information processing are easier to postulate. These organisms can become useful models to gain insight into the fundaments of neuronal dynamics and whole brain organization, validate hypotheses and develop and test modelling methods, simulation instruments and model reduction techniques. The hope is that the knowledge gained from these analyses and the techniques developed for these simpler organisms can later be used to model more complex systems.Caenorhabditis elegans (C. elegans) belongs to this category of organisms and is quickly becoming one of the benchmarks in whole brain organization studies. C. elegans is a nematode (roundworm) of about 1 mm in length with a compact nervous system consisting of less than 1000 cells across all sexes and around 15,000 connections1. This rather small nervous system allows the worm to solve basic problems such as feeding, predator avoidance and mate-finding. Moreover, at least the cell-lineage and the anatomy of C. elegans are invariant, in the sense that every individual possesses the same number of neurons and they occupy fixed positions in the organism; the invariance of the synaptic connections is still under debate2.C. elegans allowed for its almost complete description from different perspectives and scales, from its genetics and genomics to the molecular biology, structural anatomy, neuronal function, circuits and behaviour. This information is available in comprehensive databases of genetics and genomics3, electron micrographs and associated data, online books and atlases of the neurobiology, structural and behavioural anatomy4. Creating a realistic model that encapsulates all this information is not a trivial task. Open-source databases of digitally reconstructed neurons5, computational models6 and collaborative solutions7 are opening the door for more flexible, multi-scale and multi-algorithm simulation environments for C. elegans and other complex biological systems.The relative simplicity of C. elegans contains 302 neurons for the adult hermaphrodite8 and 385 neurons for the male1, but for the latter, the respective 3D reconstructions are not yet published. Digital reconstructions for the male are only available for the posterior nervous system of 144 neurons9.The underlying models are based on the connectome, the map of the neuronal connections in the brain. Usually described as a neuronal network, the connectome is a graph where the nodes are the neurons and the edges represent the synapses. The complete connectome of The more complex the organism, the more complicated the resulting model, needing more computationally demanding and potentially intractable simulations of its dynamic behaviour. This increased complexity stems from the detailed modelling of the internal structure. However, in many cases, especially of highly complex systems, this detail is not available since the internal mechanisms may not be well known or mapped or it may be simply impossible to examine and record. Notwithstanding, frequently one is really only interested in the peripheral, or input-output behaviour, which can be checked against recorded or measured data. This motivates our efforts not only to place the focus more on observable input-output data, as well as to try and generate reduced models that avoid extraneous detail not necessary to explain these peripheral relations.C. elegans as a proxy for our study.In this work we propose a methodology for generating a reduced order model of the neuronal behaviour of organisms using only peripheral information. We use C. elegans, which take into account spatial distribution and biophysical properties of neuronal compartments have been reported in the literature10. We start with a similar model created in-house11. Our model comprises the complete connectome of the adult hermaphrodite of C. elegans, with 302 multi-compartmental neurons and 6702 synapses8. The model is described in Python and implemented in NEURON12, one of the traditional neural simulators that has support for biologically realistic multicompartmental models of neurons. A 3D reproduction is extracted from NEURON in Fig. 11 against four behavioural scenarios described in related literature13: Forward Crawling Motion (FCM) for the full network, Ablation of AVB interneurons + FCM, Ablation of AVA interneurons + FCM and the Nictation behaviour. We first reproduce here the FCM scenario, and for the purposes of this study we identified four input neurons  and four output neurons  known to be strongly associated with forward movement13. Other dynamics can be validated similarly, and we can obtain single models that allow reproduction of multiple scenarios simultaneously14—see Experiment 5.Realistic models of C. elegans connectome under the described scenarios, using neural networks. The ultimate goal, however, is to show that our methodology is able to produce reduced-order, compressed models, that can be efficiently used in simulation to test and validate hypotheses regarding the behaviour and functionality of the neuronal systems of complex organisms. An illustration of our methodology is presented in Fig. Using this full 302-neuron connectome model with all its synapses, which we designate as the high-fidelity model, and the NEURON simulator, we obtain a collection of synthetic datasets representing the system’s response to different input signals. Next, assuming no prior knowledge of the original system’s structure and equations, we create a completely equation-free data-driven model using neural networks trained on these datasets. The immediate goal is to generate a reduced model to replace the original, detailed one. This reduced model should be able to reproduce with reasonably low error (the error metric used here being RMSE - Root Mean Squared Error) the behaviour of the realistic model while having fewer degrees of freedom. In this work we focus on the issue of reduced RMSE, which we equate to fidelity in reproducing the system dynamics, showing that we can produce sufficiently accurate models for analyzing the behavioural response of the The connectome-based models mentioned above are often termed white-box models, as they are based on direct knowledge and access to the internal structure and parameters’ values of the modelled system. These are distributed models, where each neuron has a 3D description and position in space and the synapses are associated with neuronal sections. Such models enable highly accurate simulation of the dynamics of the systems but easily become extremely complex as they incorporate detailed structural and functional information of the system.15. This knowledge, together with the ever-present need to avoid potential numerical intractability in large-scale networks with many degrees of freedom, has generated renewed interest in applying model reduction, often also referred to as model compression, to these neuronal networks, including techniques such as Dynamic Mode Decomposition (DMD)16, Proper Orthogonal Decomposition (POD)17 and Discrete Empirical Interpolation (DEIM)18. Depending on the level of morphological accuracy of the underlying models, reduction techniques can have any shade of grey from white-box to black-box, the latter assuming no preliminary knowledge of the system structure and building the model solely out of knowledge of its input-output behaviour.While the white-box approach ensures access to and evaluation of inner parameters during simulation, it has been shown that the activity of complex networks of neurons can often be described by relatively few distinct patterns, which evolve on low-dimensional manifolds19 and even to extract the governing equations of the estimated model20. It is therefore quite natural to consider using state of the art learning methods for developing reduced models of neuronal behaviour using data obtained from available recordings or even simulations obtained with more complex models.Black-box approaches are often built upon data-driven models, sometimes learning-based, which have the ability to grasp more naturally and more efficiently the complexity induced by the profound nonlinearities in the neuronal transmission of information. Machine-learning techniques are used to extract data-driven reduced order models for systems arising from differential equations describing the intrinsic dynamics24. Bidirectional LSTMs were used to model genome data by25, whereas a combination of CNNs and LSTMs generates a model for epileptic seizure recognition using EEG signal analysis in26. An attempt to model the human brain activity based on fMRI using RNNs (LSTMs and GRUs) is reported in27. In recent years, deep network approaches were used to model realistic neural activity data30. Few studies examined the behavioural output of network models of C. elegans using machine-learning techniques. RNNs are generated in a grey-box manner to study the chemotaxis behaviour31 or to predict the synaptic polarities32 of C. elegans, yet these models only include a subset of the connectome.Especially designed to capture temporal dynamic behaviour, Recurrent Neural Networks (RNNs), in their various architectures such as Long Short-Term Memory (LSTMs) and Gated Recurrent Units (GRUs), have been extensively and successfully used for forecasting or detecting anomalies in multivariate time series dataGiven that the starting point is in fact represented by time series data obtained from simulations of the realistic connectome-based model, the modelling task is akin to a sequence to sequence conversion for which the most suitable neural network models are the recurrent ones.35. The second model used for the recurrent layer is the LSTM unit37, and finally we analyze its sibling, the GRU38.In this work we analyze the suitability of three recurrent neural networks architectures. We start with the least complex unit, the simple RNN, originally proposed in the 1980’s to model sequence data35 are a family of neural networks used for processing sequential data, particularly adept to processing a sequence of values RNNs39, seem to be a good model for sequential tasks, they are known to suffer from various issues, mainly vanishing and exploding gradients40. Exploding gradients refer to a large increase in the norm of the gradient during training, which appears due to the explosion of long term components that can grow exponentially faster than short term ones. This is the less common of the two problems and there are known solutions to handle it, such as the clipping gradient technique41. A harder to solve issue is the vanishing gradient40, which refers to when long term components go exponentially fast to zero, making it impossible for the model to learn the correlation between temporally distant events.Although simple RNNs Fig. -left, whIn our case, for a faithful reproduction of the dynamics, the simulations require the use of fine time steps, leading to long sequences in the datasets. This in turn implies that the response at a given time will depend on values which are far back in the sequence. This situation, however unavoidable, may lead the RNN to experience difficulties in learning our data resulting in a model with unacceptable RMSE.36 appeared as a solution to the vanishing gradient problem, later improved with the inclusion of the forget gate to adaptively release internal resources when necessary37.The Long Short-Term Memory unitA LSTM unit consists of three main gates, the input gate 37, but these units are complex structures composed of three gates. For that reason, in 2014 a new type of unit, the GRU38, was suggested, described by:The use of LSTM units in recurrent neural networks already produced models able to learn very distant dependenciesThe GRU Fig. -right isC. elegans, with 302 multi-compartmental neurons and 6702 synapses. The neurons are described by 3D geometrical information extracted from NeuroML and LEMS files5 for C. elegans. We added membrane biophysical properties and connectivity data  from10 for the complete connectome. We term this as a high-fidelity model, since due to the level of detail taken into account we assume it reproduces with fidelity the output of physical models of C. elegans neurons. The simulations reproduce the Forward Crawling Motion scenario, by applying varying input currents to two sensory neurons  and two interneurons  and record the responses of four neurons known to have strong activity during forward locomotion . The resulting system is described in Python and simulated in NEURON12. The Python code invokes NEURON to generate the neuronal network, simulate its behaviour with respect to certain input signals (currents) and save the responses in time of the four output neurons (voltages).The starting model is based on the complete connectome of the adult hermaphrodite of We simulate the full high-fidelity model for 500 ms with two time steps—0.5 ms and 0.1 ms—and 40 different shapes for the input currents. The input-output waveforms are extracted into two datasets of 40 snapshot files each, which are further fed to the learning framework. These datasets are available in the online repository.To train and tune the hyperparameters, learning rate and batch size, the data was divided into three sets: training, validation and test. The separation of data is done as follows: the training set uses 42, using the libraries Keras43 and TensorFlow44. Details on the code and dependencies to run the experiments are listed in a Readme file available together with the code in the online repository.The machine learning models are developed in PythonAll architectures consist of one recurrent layer described in “45 and the loss function to the root mean squared error. The other two hyperparameters, the learning rate and the batch size, as well as the activation function were tuned separately for the three architectures, by keeping one fixed and varying the others, for a fixed hidden size of 16 units. Each model was trained for 1000 epochs, with the final model chosen as the best iteration on the validation set. We then fixed the hyperparameters at the optimal values yielded from this search: the batch size at 32 for all three, the activation functions kept to the default and the learning rate at 0.001 for the RNN and 0.05 for the LSTM and the GRU14. Note that these represent the upper limits of the learning rates, since the Adam optimizer computes individual adaptive learning rates for each parameter.For a consistent comparison, we fixed the optimizer to AdamIn Experiment 1 “” we compIn this experiment we compare the performances of the three types of units on the dataset corresponding to the coarser time step (0.5 ms).In the interest of fairness we use layers with comparable number of parameters, e.g. a RNN with Figure The GRU, due to its low RMSE and relative simplicity, therefore emerged as the prime candidate unit for our modelling purposes. However, we now want to determine how small the models can be without compromising the overall error. The focus of this second experiment is therefore to test different sizes of the recurrent layer and determine the smallest size that is still able to generate a model with sufficiently low RMSE.We test both the LSTM and GRU units using the dataset with the coarser time step of (0.5 ms). Since the LSTM does not produce noticeable improvement over the GRU with a similar number of parameters, we only report here the results obtained with the GRU for six sizes of the recurrent layer: entclass1pt{minimaentclass1pt{minimaWe now explore the models’ behaviour on data sampled with different time steps as in the same simulation interval this leads to sequences of different lengths. From a methodology standpoint this is important since even though time-wise the dynamics do not change, the temporal dependencies that the model has to learn are farther back in the sequence, which increases the difficulty of the learning process. We run the model for two datasets, one with the coarser (0.5 ms) time step and one with a finer (0.1 ms) one. The experiment is done only for the GRU, with Even though the model takes more time to converge for the finer time step, it ends up stabilizing with a loss of the same order in both cases and the model fits the test data well, as shown in Fig. Throughout this work we guaranteed the models are tested against unseen input signals, by deliberately placing in the test set data with inputs with unique and varied shapes. We now want to investigate the relative importance of certain inputs with respect to the outputs. Hence, in this experiment we train the models on the same simulation datasets as before, except we discard the data for two input neurons during the learning process, and see how the resulting models predict the outputs. We do this in two separate settings. First we train the models with recurrent layers of 8, 16 and 32, without the input information for the AVB neurons (No-AVB case). Next we repeat the process without the PLM data (No-PLM case).The models perform similarly from an accuracy perspective and the size does not influence the overall RMSE. We notice that in each case, the models are able to accurately replicate the voltage of two output neurons but do a poor job at predicting the other two outputs. In the No-AVB case, the two neurons well reproduced are LUAL and PVR, as shown in Fig. This result indicates that the AVB neurons are important for accurate prediction of the behaviour of LUAL and PVR, while the PLM inputs have more influence on the DB1 and VB1 neurons. This result is interesting from an interpretability standpoint, as it shows that some inputs are more relevant than others for specific outputs and behavioural scenarios.46. This new dataset therefore has a total of 80 examples, We create a dataset composed of data for two completely different behaviours, FCM and Nictation. For the FCM scenario the data is the same as described in “mentclasspt{minimaOur analysis shows that a GRU with as low as 4 hidden units is consistently able to reproduce the outputs of the original model with a reasonably low RMSE and it adapts well to longer sequences, resulting from data sampled with both coarser and finer time steps. There is no noticeable difference in the errors between the two datasets in Experiment 3, the model consistently showing a RMSE below 1.3e-02 for the test set. What is interesting to note from Experiment 2 (see the RMSEs for Test in Table We are interested in further understanding to what degree we are able to make some assumptions on the structure of the (reduced) model and extract a description, perhaps mathematical, for it. The fact that we are able to replace a complex biophysical model with a simpler recurrent neural network with few neurons also means the interpretability improves . Furthermore, in Experiment 4 we show that the models are able to accurately predict only certain outputs when deprived of certain inputs during training. This already suggests specific input-output dependencies, which a systematic study on input importance can further reveal and quantify. For a deeper analysis of what the RNNs learn, future work will focus on interpretability techniques, both model-agnostic and model-aware. Attention mechanisms and saliency maps can show the importance of inputs and features in the final prediction, which can further facilitate solving the inverse problem of extracting a small network from the RNN model for NEURON.Generalizability indicates a (low-order) model’s ability to predict the original system response beyond the data used for modelling. From a machine learning perspective, this can be understood as prediction for a test set with input-output data unseen during training. While this is an expected merit of the neural networks in general and also demonstrated here, this is not the only implication of our modelling efforts. We are modelling a system having biophysical descriptions with well-established accuracy. However we are treating it as a black-box, using only peripheral input-output information from the initial high-fidelity simulations. Even though for many other more complex neural circuits the internal details are less known or completely unknown, this peripheral information can still potentially be obtained, hence the effectiveness of our models extends to these systems as well. This could especially be useful for specific cases , where the system dynamics can be learned and predicted so that the number of real experiments further needed would be reduced.C. elegans nervous system with three different recurrent neural networks architectures: simple RNNs, LSTMs and GRUs. The objective is to further generate a low-order description to replace the original, detailed model in the NEURON simulator. To achieve this goal we seek a model as simple as possible and therefore the ideal unit would appear to be the simple RNN. However, this unit does not perform sufficiently well compared to the other two architectures. The LSTM and GRU give comparable results in terms of overall fidelity, measured through RMSE, for different sizes of the recurrent layer. Due to its simplicity, GRU is preferable, and with a hidden size of 4 units, is able to reproduce with high fidelity, i.e. low RMSE, the original model’s responses to different types of stimuli. Furthermore, from a computational standpoint, explicitly inferring the response of the GRU model to such stimuli will vastly outperform the cost associated with simulating the high-fidelity model within NEURON, which has to solve the set of nonlinear equations implicit in the connectome network. Quantifying the potential advantage would require solving an inverse problem and performing an identical simulation of the extracted low-order model in NEURON, which is not the subject of this paper.In this paper we create reduced order models for the Further work will concentrate on improving the automation in choosing appropriate stimuli for the training, validation and test sets as well as optimal parameter selection. This will require a systematic analysis of compression possibilities of the learning-based models with error control. The novel concept of physics-informed machine learning will also help improving not only the predictions, but also the interpretability and generalizability of the neural nets. It implies adding structural or context information like physical constraints  to the training process. The resulting models will be more reliable, as they are guaranteed to satisfy physical laws, and they will potentially need less training data, making them even more suitable for experimental neuroscience, where there is a limited amount of labelled datasets available and in some cases it is not viable, for financial or ethical reasons, to procure additional data.These results nonetheless show that it is feasible to develop recurrent neural network models able to infer input-output behaviours of realistic models of biological systems, enabling researchers to advance their understanding of these systems even in the absence of detailed level of connectivity.


Recent advances in additive manufacturing have provided more freedom in the design of metal parts; hence, the prototyping of fluid machines featuring extremely complex geometries has been investigated extensively. The fabrication of fluid machines via additive manufacturing requires significant attention to part stability; however, studies that predict regions with a high risk of collapse are few. Therefore, a novel algorithm that can detect collapse regions precisely is proposed herein. The algorithm reflects the support span over the faceted surface via propagation and invalidates overestimated collapse regions based on the overhang angle. A heat exchanger model with an extremely complex internal space is adopted to validate the algorithm. Three samples from the model are extracted and their prototypes are fabricated via laser powder bed fusion. The results yielded by the fabricated samples and algorithm with respect to the sample domain are compared. Regions of visible collapse identified on the surface of the fabricated samples are predicted precisely by the algorithm. Thus, the supporting span reflected by the algorithm provides an extremely precise prediction of collapse. Additive manufacturing provides considerable freedom in the fabrication materials with diverse geometries. In particular, the fabrication of geometries that include complex internal spaces, e.g., fluid machines such as heat exchangers, has been attempted using additive manufacturing technology . In thisThis indicates that an overhanging part in the vicinity of the support is partially supported by its geometry, although the overhang angle of the part is much smaller than the critical angle. Target geometries for additive manufacturing must be verified in advance to minimize material and time losses. Although the precise prediction of collapse regions based on the overhang angle is critical, it has been addressed in only a few studies. Thus, an algorithm is developed in this study to estimate the actual probability of a surface requiring reinforcement by support structures. The algorithm identifies the actual collapse region by reflecting the geometric attributes of the support and overhanging part by the invalidation of the overestimated collapse region. i and i. The estimated orientation angle provides a broad spectrum of collapse region detection that includes overestimated collapse regions. The orientation angle-based collapse region detection assumes any surface oriented with an orientation angle below a critical overhang angle is a collapse region. The critical overhang angle is generally defined based on experimental results while considering the surface roughness and part disruption [To investigate the support effects, the orientation angle over the part surface should first be estimated. Hence, a geometrical analysis was performed using STL (standard tessellation language) formats. The STL format includes three-point facet surface data and their normal vectors. The geometry of the parts is described by a set of facets and normal vectors toward the external directions of the facets. sruption . The crisruption  investigThe algorithm reflects the support span on a set of facet data, and the geometric attributes of the overhanging part are recognized as connections among facets by their edges. As the initial procedure of the algorithm, edge searching over facets is performed to determine the supporting boundary edge as a set of supporting edges for each facet. Any facet part of a solid in STL format is connected with three adjacent facets. The supporting edges are evaluated within the facet connections based on the decision process. To detect supporting edges in a pair of stable facets and a facet collapse region, the orientation angles of three adjacent facets are first addressed. Subsequently, qualified edges are determined only between a facet with a smaller orientation angle than the critical overhang angle and a facet with an orientation angle larger than the critical angle. If any of the part facets with a larger critical overhang angle are constructed earlier than the supporting facet, then the support effect will not be transferred to the unstable facet because additive manufacturing is a sequential procedure along the build direction. Therefore, the previously qualified edges are evaluated via the build sequence constriction. The build orders of a facet and its adjacent facet are classified via a comparison of their points.z-coordinate of the free point of each facet. If the z-coordinate of the free point of the adjacent facet  of a facet with index i, respectively. If a facet consists of a supporting edge, the initial value To reflect the spanwise supporting area along the detected supporting edges, the initial value of the supporting span should be decided based on the facet collapse region, which includes the detected edges. The initial value along the supporting edges is defined as follows:Spanwise supporting is addressed over the surface with facets via an iterative propagation from the initial values on the facets along the supporting boundary edge. However, the facet to be supported by adjacent facets must be identified. In this regard, a decision procedure to categorize edges is proposed. The support effect is propagated from one facet to another; thus, the edge category is determined by considering the interconnection of a pair of facets. Although the facets share an identical edge, the edge type can be evaluated differently depending on the facet from which the propagation originated. i, and j are the facet index and index of edges in the facet, respectively. The initial value of propagation is assigned a negative value; therefore, the effect is propagated only when a facet is assigned a negative value. The auxiliary variables The first step of the propagation process begins from the initial facets assigned with initial values. The values of the transferring effect are defined by an assigned value, the edge category, and length of the edges of the origin facet. The transferring value is calculated as follows:j is an NPE, the transferred value, j to the total length of all PEs. i, i.e., the origin facet of propagation, the propagation of i is shared by a facet on the support structure and a facet in the collapse region; thus, the third edge is categorized as an NPE, which always transfers a zero value. Meanwhile, the first and second edges are shared by adjacent facets on the parallel collapse region surface, and the adjacent facets are not calculated in the previous iterations. Hence, the two edges satisfy all conditions of a PE, which transfers values across the edge.When the edge i; therefore, the number indexes of the edges are not identical with the number indexes of the origin facet in the first step of propagation, although the affected facet and origin facet share an identical edge. Thus, the transferred value, j with a facet i, not as its own index. This second step is the final sequence of the single iteration for propagation. The first and second steps are iterated. However, the negative value of In the second step of the propagation process of the algorithm, the newest value A calculation example is presented in this subsection based on specific cases for the first and second steps. 1.Based on Equation (3);2.Based on Equation (4);3.Based on Equation (5);4.In the second iteration, the value assigned to the affected facet 8 with a dashed outline is defined by three 1.Based on Equations (2) and (6);2.3.The procedures above are repeated over the faceted surface until the origin facets in the first step constantly transfer values of zero to their adjacent facets in subsequent iterations. This allows the affected facets to be determined by the supporting span over the surface. Consequently, the latest propagation procedure returns the value i, the critical overhang angle, and the orientation angle defined by a normal vector of the facet i, respectively. Changing the orientation angle of a facet to the critical overhang angle implies that the risk region is a stable region.The support effect based on the span area is reflected by the propagation procedure explained in earlier sections. Based on the result of the propagation, any facet assigned with a negative The algorithm presented herein was developed to precisely detect collapse regions by predicting the over-melting area. Thus, the algorithm was applied to a target model with an extremely complex enclosed internal space to predict surface collapse in the enclosed space. Several sample sections were extracted based on high-risk areas evaluated via critical overhang angle-based risk estimation. Domains of the extracted samples were fabricated using AlSi7Mg metal powder. The surface disruption of the fabricated samples was compared with the result yielded by the proposed algorithm for verification.The algorithm was applied to a heat exchanger model (the target model) that featured a complicated enclosed space. Moreover, a collapsed internal surface, or one with unsatisfactory quality, induces severe problems, e.g., detached metal fragments, which can flow into the fluid circulation system. Thus, collapse areas must be predicted in advance for applications pertaining to heat exchangers. Based on the three selected sectors, the corresponding samples were extracted from the target model. Printing the entire target part for the manufacturability test is time-consuming and incurs a high material cost. Thus, fabricating representative samples for testing is preferred . Figure Prototypes of the extracted samples were fabricated via LPBF for further experimental validation. However, the process parameters of additive manufacturing directly affect the quality of the fabricated part. Thus, the conditions of the additive manufacturing process are specified herein. The material is one of the major parameters affecting additive manufacturing. Hence, the properties of the material were first analyzed. AlSi7Mg metal powder was adopted as the material of the samples. The characteristics of the metal powder grain must be specified; thus, microscopic images of the powder were captured via scanning electron microscopy (SEM). ® software, https://ww2.mathworks.cn/products/matlab.html (accessed on 21 December 2022) was adopted to execute all the protocols of the algorithm using a computing interface comprising an Intel® Zeon® E5-2697 v3 (14 cores) dual processor. The correct representation of the complex geometry requires numerous fine facets, which increases the calculation time and thus the processing time of the entire computation process. The number of facet elements to describe each sample and the number of iterations for estimation are listed along with the processing time in The collapse detection algorithm was applied to the extracted sample parts. MATLABIn sample 1, no visible and intolerable collapse regions were observed. To provide a more detailed comparison, four features of the sample were analyzed, as shown in Sample 2 was fabricated to verify the manufacturability. In particular, sample 2 was segmented into two pieces to observe the collapse regions over the surface of the internal space. Collapse was clearly detected on the surface of sample 2, as shown in Based on the close-up images of the collapsed regions in sample 2, the algorithm-based estimation is more suitable for predicting collapse on the surface. To validate the results of the algorithm, sample 3 was fabricated. Collapse regions must be predicted for applications of fluid machines that involve complex internal surfaces. An algorithm for the enhanced detection of collapse regions was presented herein. The algorithm reflects the supporting span over a faceted STL geometry to specify the actual collapse region over the surface of a part fabricated by additive manufacturing. The algorithm involves edge classification and a propagation process, where the supporting effect is transferred over the geometry surface. The proposed algorithm enables the detection of collapse regions over complicated nonlinear surfaces. In this study, the proposed algorithm was experimentally validated using three different samples. Results showed that the proposed algorithm provided a more comprehensive description of the collapse region profile compared with the critical overhang angle-based estimation. Furthermore, the results indicate that the proposed algorithm is suitable for extremely complex geometries. Thus, the proposed algorithm can be applied to predict the surface quality of parts fabricated via additive manufacturing  as well


Fully autonomous vehicles (FAVs) lack monitoring inside the cabin. Therefore, an in-cabin monitoring system (IMS) is required for surveilling people causing irregular or abnormal situations. However, monitoring in the public domain allows disclosure of an individual's face, which goes against privacy preservation. Furthermore, there is a contrary demand for privacy in the IMS of AVs. Therefore, an intelligent IMS must simultaneously satisfy the contrary requirements of personal privacy protection and person identification during abnormal situations. In this study, we proposed a privacy-preserved IMS, which can reidentify anonymized virtual individual faces in an abnormal situation. This IMS includes a step for extracting facial features, which is accomplished by the edge device (onboard unit) of the AV. This device anonymizes an individual's facial identity before transmitting the video frames to a data server. We created different abnormal scenarios in the vehicle cabin. Further, we reidentified the involved person by using the anonymized virtual face and the reserved feature vectors extracted from the suspected individual. Overall, the proposed approach preserves personal privacy while maintaining security in surveillance systems, such as for in-cabin monitoring of FAVs. Protection of facial privacy.Evidence of the accused in abnormal situations.Intelligent monitoring and surveillance systems are widely used to ensure safety and security. Popular applications of monitoring in public are video surveillance cameras (closed-circuit television); monitoring in intelligent transportation systems, including in-cabin monitoring and road traffic monitoring; and video monitoring for data generation and navigational tasks around city centers, airports, and public roads . DrivingTherefore, a robust solution is required to provide privacy-preserved monitoring in public . MoreoveAs illustrated in the above figure, an anonymous face protects personal information during in-cabin monitoring of an FAV. However, in certain irregular situations, personal identity is required to identify the accused person. An example of an abnormal incident or irregular situation can be an occupant of the FAV acting violently or attempting vandalism against the other occupants or toward the FAV itself. In such cases, it is important to identify the concerned person. Furthermore, this is an abnormal situation; however, in-cabin monitoring with real faces is not a solution to this problem. The breach of facial information leads to multiple consequences, such as misuse of facial data and banking and financial fraud , 13, 14.In-cabin monitoring is important in level 4 and beyond AVs . It provReal-time monitoring is essential in multiple monitoring applications. However, privacy in the public domain is an important concern in real-time monitoring tasks –30. FaciThe concept of having an appropriate source face for each target face enhances puppeteering and reenactment of facial emotion and behavior. It helps in event and behavior detection in intelligent monitoring and surveillance systems in the public domain.The involvement of the two-dimensional (2D) landmark position in the reenactment generator and separate segmentations of face and hair in the segmentation generator with inpainting and blending generators enhances the facial anonymization and reenactment operations.ID is stored. Therefore, there is no threat of privacy breach, even though the IDs are stored in the cloud.The 128D identity feature is a key marker for accurate facial identification in an anonymized domain. The concept of storing a pair of IDs  leads to reidentification without any privacy threat. It is not possible to know the original face with only 128D identity features. For reidentification, both the original visual input and the ID are required. In the cloud, the anonymized visual image with the original Therefore, the proposed approach augments the facial identity feature information to locate the involved person in any abnormal situation without any personal privacy breach.In this study, we propose an intelligent IMS. It is an efficient approach for identifying a person, even with an anonymized face. This method resolves both privacy and security issues. Accordingly, we can identify the person who causes an irregular situation, even with their anonymized face. In this approach, we preserved the key facial information of the occupants and stored these identity features on the cloud. These key features help in recognition of the person involved in the irregular situation. The highlights of this study are as follows:This approach pioneers a newer method of monitoring and surveillance to avoid any legal or ethical issues. Therefore, a monitoring database can be created in the anonymized domain, thereby facilitating further research on events and behavior monitoring in the public domain.IDs) was generated for each face before and after anonymization (IDR and IDA). Further, the anonymized video along with the IDs was fed to the cloud. The pairs of IDs were kept in the cloud for person reidentification when required. The anonymized video frames were sent to the data center for further processing (monitoring and surveillance). In stage 3, the IDs were matched to search the accused (person involved in an irregular situation (IDA_AS)). During the investigation, the similarity between IDs ensured the identification of the concerned person (ID). Further, during the investigation, this approach was verified by matching the IDs of the suspect face (IDinv) at the time of investigation with the accused person's ID.Personal privacy with identification is a challenge as well as a demand in real-time monitoring applications . In thisThe dilemma between monitoring requirements and legal and ethical issues is also resolved through this approach. The details of the proposed approach are discussed thoroughly in Many research works have been published on personal privacy and person identification considering these two issues as separate research problems. In this study, we briefly surveyed the related works and developments on both face anonymization and person identification.Face deidentification preserves privacy-sensitive information. It alters the original face to hide privacy-sensitive information. Anonymization of faces is an easier and more robust solution to personal privacy-related threats in the digital domain . BlurrinDeepFace [DeepFace have been demonstrated in multiple studies [Facial recognition has multipurpose objectives, such as recognition, classification, and discrimination. Urbanization and smart cities demand widespread applications for face recognition –52. TherDeepFace . Further studies –59. Adja studies . Kortli  studies . They exResNet) model was suggested in [However, an application involving person identification must address important privacy concerns . In partested in  for visuested in . They deested in . It is aested in , 68. AR/IDR and IDA) using the ResNet-based model. These IDs are stored in the cloud, and the anonymized video frames of in-cabin monitoring are transferred to the data center via the cloud for further processing. The third task of the proposed approach is to identify the accused by identity feature matching. Similarity matching of the ID of the accused obtained at the data center with the IDs of the occupants stored in the cloud ensures the identification of the concerned person (IDA). However, it is the ID of the anonymized face of the accused. The Euclidean distance metric was used for similarity matching. Similarly, using the stored pairs of IDs (IDR and IDA), we can obtain the real face identity feature of the accused (IDR). Finally, in the fourth task, the evidence of the accused is obtained by matching the similarities between the IDs of the suspects with the ID of the accused during an investigation. Further details of the proposed method are provided in the following sections.In this study, we proposed a representation learning-based approach to generate the identity signature of occupants. This signature is capable of deidentifying a person concerned with an irregular situation in the cabin of level 4 and beyond AVs. We proposed facial anonymization and reidentification system to provide countermeasures in case of an irregular situation. Therefore, this method provides personal information security with traces of the concerned person in case of any abnormality. The proposed method includes four main tasks. First, face anonymization with reenrollment. This is performed by using the face agnostic face swapping technique. It uses a set of GANs. These GANs are used for three purposes: facial reenactment and segmentation, facial inpainting, and facial blending. After accomplishing face anonymization, the second task is to extract the facial identity features of the occupant's faces in pairs  from the faces. Additionally, 128D is optimal embedding, which results in appropriate features required for reidentification or measuring the similarity between two faces. It has already been validated in the “FaceNet” architecture that fewer than 128D identity features deteriorate the identification performance; however, increasing the dimension only unnecessarily increases the number of parameters. This is the main reason for adopting the 128D identity features for recognizing faces.The facial identity feature is -dimensional encoding of a facial image. It contains the encoded landmarks of the face using the ResNet-based architecture consisting of 29 convolutional layers for this purpose. The ResNet architecture facilitates the dipper layer accessibility. Additionally, they have an inherent tendency to minimize the training error loss by increasing the number of layers. The triplet loss function is used to estimate the error in the reidentification of the concerned person. It performs similarity matching on the 128D identity features. For the anonymized anchor image ID (IA), positive anonymized image ID (IP), and negative anonymized image ID (IN), the triplet loss is estimated by the following equation:2) denotes the “Euclidean distance” between pairs {x, y} in the triplet loss function. A factor margin is included in equation (The anonymized anchor image ID (IA) represents the 128D ID of the person figured out in an irregular situation. The positive anonymized image (IA) is the stored image 128D ID of the same person on the cloud, and the negative anonymized image ID (IA) is the 128D ID of another occupant. Here,  represents the person. Image (C) is the stored image (copy) of the same person on the cloud, and image (D) is an image of another occupant. Further, ƒ(x) represents the 128D encoding of the image f(x). The similarity (S) in the vector space is measured by the following equation:Furthermore, a distance-based classifier compares the 128D features to identify the person involved in an irregular situation. It represents the difference between two feature vectors in Euclidean space. Suppose that image (R) and (C) are of the same occupant and are different from image (D), which is the image of another occupant.It guarantees that images  of faces (target faces) and matches the similarity of the target faces with the set of virtual human faces (source faces) to find the most appropriate source face. This similarity matching is in the vector space .Facial anonymization requires exactitude in the anonymized faces to mitigate errors in further processing. Therefore, swapping should be performed efficiently to provide unaltered expressions and emotions over the anonymized face. We used the concept of FSGAN for facial anonymization to provide personal privacy during in-cabin monitoring of irregular situations. This requires perfection in the following three tasks:(i) Facial Reenactment and Segmentation. To obtain proper facial swapping, we must estimate the proper reenacted face. This is performed by the proper segmentation of the face and hair segments of the target image. Proper facial reenactment requires separate face and hair segmentations with the mapping of 2D facial landmark positions. Therefore, the stepwise loss function is considered as the objective function for implementing facial reenactment. For ith layer feature map (Fi ∈ ℝCi×Hi×Wi), the perceptual loss (ℒperc) between pairs of images  is expressed as follows:rec) between a pair of images  is expressed as follows:λ” is the corresponding hyperparameter  and λreenactment is linearly increased from 0 to 1 during training. Pixelwise loss (ℒpixel) between a pair of images  is calculated as =‖x − y‖). We have used the multiscale discriminator adversarial loss objective function to improve the realism of the generated images. The adversarial loss (ℒadv) between the generator and discriminator  is expressed as follows:Ex, y) is given by the following equation:The reconstruction loss (ℒCE) is defined as (for truth label “ti” and the “SoftMax” probability “Pi” for ith class)The perpetual loss is used to estimate the errors in capturing fine facial details, and the reconstruction loss is used to evaluate pixelwise color inaccuracy. Adversarial loss improves the generated images and provides a realistic look. The standard cross-entropy loss (ℒSG) is obtained by the following equation:Further, segmentation generator loss (ℒ(ii) Facial Inpainting. This method estimates the missing portions of the reenacted face based on the face and hair segmentation of the target image. The inpainting generator loss (ℒIP) was calculated using the following equation:(iii)Facial Blending. It blends the completely reenacted face such that the swapped face matches the background environment like the original target face. The loss function (ℒB) for facial blending is obtained using the following equation:The identity signature is generated corresponding to each occupant  in the FAV. After facial anonymization, the video frames are transmitted to the cloud along with a pair of identity signatures of the occupants.ID) of the person involved in an abnormal situation.The proposed IMS facilitates the reidentification of the person involved in an abnormal situation. In our algorithm, in-cabin facial anonymization for preserving identity before transmitting the video frames to the cloud was achieved through the following pseudocode. The identity signature is generated corresponding to each occupant in the FAV. It is a vector of size 1 × 128. Therefore, for each occupant, we have a pair of identity signatures corresponding to the original and anonymized faces. Each pair is stored in the cloud. In any irregular situation, the concerned person is back-traced by matching the identity signature and anonymized face. The following is Pseudocode 1 of our proposed approach for obtaining the identity features ) in Pseudocode 1 refers to the crucial identity parameter of the person involved in an abnormal situation. Matching the identity feature at the time of investigation with the obtained ID ) confirms the person involved in an abnormal situation. Therefore, this approach easily locates the person involved in an irregular situation without any breach of others' identities.In the case of proof or evidence, our method determines who is the concerned person. The returned identity feature  is calculated by the Euclidean distance (ED) metric that is expressed as follows:i) and ƒ(j) represent the 128D encoding of images i and j, respectively. Therefore, the similarity measure identifies the distance (Euclidean distance) between two pairs of IDs (128D encoding). The lesser the distance is, the closer the faces are.In our experiment, we first anonymized the occupants of the FAV to secure their privacy in the public domain. Further, we applied the concept of vector space similarity to match the representation learning-based identity features for face recognition to locate the person involved in an irregular situation. The augmentation of the representation-learning-based identity feature introduces a new domain in reidentification. The proposed system was introduced to maintain personal privacy during the monitoring. We examined our proposed system for the in-cabin monitoring task of the FAV. We captured our database for in-cabin monitoring in abnormal situations. The similarity measure , an appropriate source face is obtained by matching their similarity in the vector space. We considered various scenarios to assess the efficacy of our proposed approach, including single and multiple faces in the input image frame. These faces are nonreal virtual human faces. Generated Photos provides GAN-generated faces, which are human faces of nonreal humans. This has the benefit of further augmentation in anonymization. We considered various scenarios in our experiments. Examples include images with a single face only , multiple faces for males only and females only, and multiple faces for both males and females. These are in-cabin images obtained from the public domain (through an image search on the web) and are shown in These values follow the facial similarities of the source and target faces. These values measure the distance between the identity features of the source and target faces. The lower the values are, the more similar the faces are. The values in the green boxes represent the minimum Euclidean distances. These minimum values indicate appropriate source faces for anonymization. We can observe that the male target faces have lesser distances for male source faces than for female source faces. Interestingly, the distance values follow the similarity in looks as well. The eastern looks target faces have a lesser distance for eastern source faces than for the western source face, and vice versa. Female source faces have a lesser distance than the identity features of children's target faces.Facial anonymization is performed after deciding the appropriate source face using FSGAN-based face swapping and reenactment. IDs) of each occupant were calculated for normal and irregular situations. It is clearly observed that O3 (in the green box) is responsible for the irregular situation .It is evident from this result that perfect reenactments are achieved even in the anonymized domain. Thus, it discerns the preservation of personal privacy during monitoring and surveillance operations. Furthermore, this appropriate reenactment supports the detection of abnormal or irregular situations in real time. To examine abnormality detection in the anonymized domain, we have experimented by considering vandalism as an irregular situation inside the vehicle cabin. We created our database for a similar situation. Snippets of the vandalism inside the vehicle are shown in IDA) with the anonymized facial identity feature of occupant #3 (IDA_IS). Here, IDA_IS is the anonymized identity feature of the occupant who is involved in an irregular situation calculated at the data center, and IDA is the anonymized identity feature of the occupant stored in the cloud.IDs indicate the involved person. The original ID of this person is stored in the cloud. Therefore, by mapping the ID, we can easily identify the real person. Reidentification was performed by backtracking the ID obtained from the cloud and pictures of the occupants taken during the investigation. The ID of the person involved in an abnormal situation from the cloud (IDR) needs to be matched with the IDs of the occupants inside the vehicle for facial identification of the person. This approach provides proof or evidence confirming the identity of the concerned person. For assurance of the person involved in the abnormal situation, we took pictures of the occupants (during an investigation). The images are shown in ID (IDR) (as per Pseudocode 2). First, we compared the similarity between the faces of the occupants inside the vehicle with those of the other faces captured during the investigation. This is required to ensure that the occupants are the same.The values in the green boxes represent the minimum Euclidean distances. These minimum differences between the IDs extracted during an investigation and their IDs extracted from the in-cabin images.ID of the person involved in an abnormal situation (stored in cloud IDR). IDs extracted from the in-cabin image with the obtained ID of the person involved in an abnormal situation (stored in cloud IDR).The minimum Euclidean distances are represented by the green boxes. Here, minima indicate that the occupants O and O′' are the same. Thereafter, assurance of the involved person is performed by matching the identity feature of the occupants extracted from the in-cabin image of the vehicle with the The zero value in the green box indicates that the occupant (O3) is the person involved in an abnormal situation. Overall, this approach focuses on in-cabin monitoring with personal privacy preservation to avoid abnormal situations. Personal privacy preservation is achieved by using the concept of event and behavior monitoring in an anonymized domain. The person's reidentification is only for providing evidence in cases where the involved person is denying it.Identity feature augmentation in anonymization is a potential solution for providing privacy in public domain monitoring. Identification of the involved person is crucial, especially in abnormal situations. The proposed intelligent IMS augments the security features with privacy. This method is suitable for creating a monitoring database without any restrictions or legalities. We performed various scenarios to assess the efficacy of the proposed system. It provided an efficient algorithm to perform monitoring tasks in the public domain without any threat to the personal identity of a person. This helped in reidentification, even with an anonymized face. In the future, this algorithm can be implemented on various public domain monitoring platforms, such as transportation systems, shopping centers, theaters, hospitals, highways, fuel refilling stations, smart city applications, and toll plazas.


Q-network with prioritized experience replay. A multiobjective reinforcement learning framework based on ensemble learning is further designed for policy integration in the main scene. Finally, by selecting the strategy from subtarget scenes in the designed framework, an optimized action selection strategy is trained and used for the action decision of the agent in the main scene. Compared with traditional value-based reinforcement learning methods, the proposed method achieves a 93% success rate in path planning in simulation scenes. Furthermore, the average length of the paths planned by the proposed method is 3.28% and 1.97% shorter than that of PER-DDQN and dueling DQN, respectively.It is challenging to perform path planning tasks in complex marine environments as the unmanned surface vessel approaches the goal while avoiding obstacles. However, the conflict between the two subtarget tasks of obstacle avoidance and goal approaching makes the path planning difficult. Thus, a path planning method for unmanned surface vessel based on multiobjective reinforcement learning is proposed under the complex environment with high randomness and multiple dynamic obstacles. Firstly, the path planning scene is set as the main scene, and the two subtarget scenes including obstacle avoidance and goal approaching are divided from it. The action selection strategy in each subtarget scene is trained through the double deep  In ocean exploration, the competition among countries to protect marine territorial sovereignty and develop marine resources has become increasingly fierce. The unmanned surface vessel (USV), as a kind of vessel with high autonomy, has broad application prospects in the field of ocean exploration. As one of the current research hotspots, the path planning of USV faces many challenges, including unknown environment, perceptual uncertainty, and dynamic obstacles –3. The UTraditional path planning methods perform well in simple known static environments and reach a destination while avoiding obstacles –9. But tHowever, the path length inevitably increases in the obstacle avoidance process, which conflicts with the requirement of destination reaching for the goal approaching subtarget task. Therefore, it is difficult for a single optimization strategy to simultaneously achieve these subtarget tasks. Recently, intelligence computing algorithms have been widely used in related fields –19. A moQ-network with prioritized experience replay (PER-DDQN) is applied to the action decision of USV in two scenes, respectively.Based on the main scene of path planning considering random goals and multiple dynamic obstacles, the dynamic obstacle avoidance subtarget scene and the goal approaching subtarget scene are constructed. The double deep A multiobjective reinforcement learning architecture based on ensemble learning is designed, optimizing the multiobjective policy integration method in the USV path planning task.A USV path planning algorithm based on multiobjective reinforcement learning is proposed, improving the success rate of USV path planning tasks and shortening the planned path length in the complex environment.Main contributions in this paper can be summarized as follows:The rest of this paper is organized as follows. The theoretical background of the PER-DDQN and the multiobjective reinforcement learning method is introduced in Q-learning algorithm is a value-based reinforcement learning algorithm  at each iteration i, with target yiDQN=r+γ maxa′ Q. Here, θi− are the parameters of a target network that is frozen for a number of iterations while updating the online network Q by gradient descent. The action a is chosen from Q by an action selector, which typically implements an ε-greedy policy that selects the action that maximizes the Q-value with a probability of 1-ε and chooses randomly with a probability of ε.To address the curse of dimensionality in high-dimensional state spaces, Mnih et al. used a neural network with  Q . DQNs arOnline reinforcement learning (RL) agents incrementally update their parameters  while they observe a stream of experience . BecauseQ-learning, respectively, to update the value function or action value function. Many scholars use DRL methods based on value functions, including DQN algorithm and some improved variant algorithms, to motivate robots or other agents to obtain optimal paths , .(2) Goal Approaching Subtarget Scene. The goal is a rectangle with a length of 5 m and a width of 1 m. The initial abscissa of the target is randomly selected within the range of . The initial abscissa and ordinate of the agent are chosen randomly from the range , .(3) Main Scene. The initial condition of the goal is consistent with the goal in the goal approaching subtarget scene, and the motion parameters of the dynamic obstacle are consistent with dynamic obstacles in the dynamic obstacle avoidance subtarget scene. The agent's initial abscissa is chosen randomly from the range , , and the initial ordinate is set to −8 m.The reinforcement learning settings for the agent in each scene, such as the action space, state, and rewards, are set as follows.(1) Action Space Setting. The agent's action space is set to 5 directions divided evenly into in the main scene and each subscene, as shown in (2) State Settings. The states of the agent  are set as equations   (2) Statst is set as equation  Reward Setting. The rewards of the agent  are set as equations  in t in t(3) After 800 times of training in the dynamic obstacle avoidance subtarget scene, the two samples are shown in After 800 times of training in the goal approaching subtarget scene, the two samples are shown in After 800 times of training in the main scene, the two samples are shown in The experimental results show that by dynamically selecting the strategy of subagents, the main agent can avoid obstacles and approach the goal in various scenes to accomplish the path planning task well. Therefore, the effectiveness of the proposed method has been verified.A∗ + ORCA and the path planning algorithm based on single-objective reinforcement learning to demonstrate the advantages of the proposed method in path planning tasks.To verify the effectiveness of the proposed framework on strategy integration, a comparison is made between reinforcement learning methods that use integration methods such as linear voting method and rank voting method and our method. At the same time, the proposed method is compared with Q-value of each action in the main scene is the normalized sum of the Q-values in the corresponding states of each subscene. In the rank voting method, the rank of each action in the main scene is the sum of the ranks of the corresponding states of each subscene. In these methods, the subagents and their strategies are consistent with the proposed method. The performance indicators of these methods in the results of 100 random experiments are shown in In the linear voting method, the The rank voting method has the worst integration effect and the lowest success rate. Compared with the rank voting method, the linear voting method considerably enhances the success rate by keeping the path length from increasing. The path length of the proposed method is slightly longer than that of the other two methods, but the success rate of goal approaching and dynamic obstacles avoidance are higher. The proposed method has the best overall performance.Three random samples of the path planning results of the three methods in the same environment are shown in The training times of PER-DDQN are 2400 times, and other hyperparameters are consistent with the method's parameter setting in the main scene. The performance indicators of these methods in the results of 100 random experiments are shown in A∗ + ORCA is low. DDQN algorithm solved the problem of overestimation of action value function in Q-learning. On this basis, PER-DDQN uses priority sampling to accelerate the convergence speed of the algorithm, and dueling DQN uses the competitive architecture to estimate the value function more precisely. They perform well in the constructed scenes. Our approach combines the strengths of reinforcement learning with ensemble learning. The experimental results show that the method proposed in this paper has the best overall performance when considering path length and success rate.The assumptions of the ORCA in the decision-making process are inconsistent with the requirements of dynamic obstacle avoidance in practical applications. Therefore, the success rate of agent using the A∗ + ORCA method are not sufficient for the agent to always avoid obstacles. The policies provided by dueling DQN are conservative, and there may be detours. The policies provided by the PER-DDQN are not mature enough in dealing with conflicts between subtarget tasks. Many problems still exist such as long planning path, failure to avoid obstacles, and approaching the goal. The experimental results show that the method proposed in this paper is generally safe and performs well in various environments.Four random samples of the path planning results of the four methods in the same environment are shown in In this paper, a path planning algorithm for USVs in complex marine environments based on multiobjective reinforcement learning is proposed. To simulate complex ocean environment, a complex scene including dynamic obstacles and random goal is built. On this basis, two subtarget scenes with goal approaching and dynamic obstacle avoidance are established, respectively. The PER-DDQN algorithm is used to train the action decision of the agent in the two subtarget scenes. A multiobjective reinforcement learning architecture is designed to optimize the agent's policy integration method in path planning. The simulation results show that the proposed method achieves a higher path planning success rate and a shorter path length than the traditional path planning methods.Although the proposed method realizes the decision making of the agents in the constructed scenes, the complexity of the scene is still insufficient. The computational efficiency and path planning success rate of the algorithm will be reduced in complex environments. Modelling more actual scenes and building more realistic training scenes can effectively improve the adaptability of the algorithm. In addition, the action space in the established model is discrete, which is somewhat different from the real world. Agents cannot output continuous action decisions in the scenario of discrete action strategy only. The assumption that the next time step after the action can reach the target position is also idealized, and inertial factors need to be taken into account to optimize the model. In future work, hostile ships with tracking capabilities will be added to the scene to train the model better. The dimension of the action space will be increased to enhance the USV's mobility. In addition, changing the scene from 2-dimensional space to 3-dimensional space is our follow-up research direction.